Classification by worked example
Ten customers, one question — who is about to cancel? We fill in a confusion matrix by hand, read four scores off it, and see exactly when plain accuracy lies to you.
The question: predict a category
Classification is what you use when the answer is a label, not a number: churn or stay, approve or reject, A/B/C. Our question: which subscribers will cancel next month?
The data: ten test customers
A model trained on past subscribers now faces ten customers it has never seen. For each we know what really happened (the truth) and what the model said:
| # | Months subscribed | Support tickets | Really churned? | Model said | Outcome |
|---|---|---|---|---|---|
| 1 | 2 | 5 | Yes | Yes | TP |
| 2 | 3 | 4 | Yes | Yes | TP |
| 3 | 1 | 6 | Yes | Yes | TP |
| 4 | 8 | 1 | Yes | No | FN |
| 5 | 4 | 3 | No | Yes | FP |
| 6 | 24 | 0 | No | No | TN |
| 7 | 18 | 1 | No | No | TN |
| 8 | 36 | 0 | No | No | TN |
| 9 | 12 | 2 | No | No | TN |
| 10 | 30 | 1 | No | No | TN |
The last column names each outcome: TP = true positive (churner caught), FP = false positive (false alarm), FN = false negative (churner missed), TN = true negative (correctly left alone). Customer 4 is the painful one — a long-time subscriber the model did not see coming. Customer 5 is the false alarm.
The confusion matrix, cell by cell
Count the four outcomes into a 2×2 table and you have a confusion matrix — the single most information-dense view of a classifier:
| Model: "will churn" | Model: "will stay" | |
|---|---|---|
| Really churned (4) | TP = 3caught | FN = 1missed |
| Really stayed (6) | FP = 1false alarm | TN = 5left alone |
Four scores from one table
(TP + TN) ÷ all = (3 + 5) ÷ 10 = 0.80
8 of 10 calls were right. Intuitive — but keep reading before you trust it.
TP ÷ (TP + FP) = 3 ÷ 4 = 0.75
Of the 4 customers the model flagged, 3 really left. High precision = few false alarms — what you want when each alarm triggers a costly retention offer.
TP ÷ (TP + FN) = 3 ÷ 4 = 0.75
Of the 4 who really churned, the model caught 3. High recall = few missed cases — what you want when missing one is expensive.
2 × (0.75 × 0.75) ÷ (0.75 + 0.75) = 0.75
One number balancing precision and recall. If either collapses, F1 collapses with it — which is exactly the point.
When accuracy lies: the imbalance trap
Now imagine 100 customers where only 5 churn — a case of class imbalance. A lazy model that always says "stays" gets 95 of 100 right: 95% accuracy, sounds excellent. But its recall on churners is 0 ÷ 5 = 0. It caught nobody. The one thing you built the model for, it cannot do.
This is why balanced accuracy exists: it averages the recall of each class, so the rare class counts as much as the common one — the lazy model scores 0.5, exposed as a coin flip. Bayanii also trains with balanced class weights on imbalanced data so the rare class is not ignored in the first place.
One more idea: scores, not just labels
Under the hood the model does not just say yes/no — it produces a probability ("72% likely to churn") that gets cut at a threshold. ROC AUC judges those raw probabilities directly: the chance that a random churner is ranked above a random stayer. It tells you how good the model's ordering is before any threshold is chosen.
This lesson describes Bayanii's real pipeline in simplified terms. It is updated by hand when the pipeline changes, so wording may occasionally lag the code.