Lesson 5

Understanding your metrics

One reference page for every score a model card can show — using the same houses from lesson 2 and the same customers from lesson 3, so each number has a story you already know.

For predicting numbers (regression & forecasting)

R²

Closer to 1 is better · unitless

How much of the target's variation the model explains versus always guessing the average. Lesson 2's house model: R² ≈ 0.994 — it explains 99.4% of the price differences. 1.0 is perfect, 0 is no better than the average, negative is worse.

MAE

Smaller is better · same units as the target

The average miss, in the target's own units. The house model's errors (0, 20,000, 0, 0, 20,000 SAR) average to MAE = 8,000 SAR: "typically about 8,000 off".

RMSE

Smaller is better · same units as the target

Like MAE but errors are squared first, so big misses hurt extra. Same five errors → RMSE ≈ 12,650 SAR. When RMSE sits far above MAE, the model is usually fine but occasionally badly wrong.

MSE

Smaller is better · squared units

RMSE before the square root: the same five errors give 160,000,000 SAR². Ranks models identically to RMSE — just harder to read, being in squared units.

MAPE

Smaller is better · percentage

The average miss as a percentage of the true value. Fresh mini-example: actuals 100, 200, 50 vs predictions 110, 180, 60 → misses of 10%, 10%, 20% → MAPE = 13.3%. Beware: when a true value is near zero, its percentage explodes and can dominate the score.

sMAPE

Smaller is better · percentage

MAPE's steadier cousin: each miss is divided by the average of actual and predicted rather than the actual alone, so near-zero actuals cannot blow it up. Same three rows, same idea — a bounded percentage.

For predicting categories (classification)

All from lesson 3's ten customers and their confusion matrix: TP = 3, FP = 1, FN = 1, TN = 5.

Accuracy

Closer to 1 is better

The share of calls that were right: (3 + 5) ÷ 10 = 0.80. Intuitive, but on imbalanced data it flatters lazy models — see balanced accuracy below.

Precision

Closer to 1 is better

Of the rows flagged positive, how many really were: 3 ÷ 4 = 0.75. Watch it when every alarm costs money — a retention offer, a manual review.

Recall

Closer to 1 is better

Of the truly positive rows, how many were caught: 3 ÷ 4 = 0.75. Watch it when a miss is expensive — a churned customer, an undetected fault.

F1 score

Closer to 1 is better

The harmonic mean of precision and recall: with both at 0.75, F1 = 0.75. If either collapses, F1 collapses — one number that refuses to let you hide a weakness.

Balanced accuracy

Closer to 1 is better · 0.5 = coin flip

The average of each class's recall: churners 3/4 = 0.75, stayers 5/6 ≈ 0.83 → (0.75 + 0.83) ÷ 2 ≈ 0.79. On lesson 3's 95/5 thought experiment, the lazy "always stays" model scores 0.5 here — exposed as a coin flip despite its 95% plain accuracy.

ROC AUC

Closer to 1 is better · 0.5 = random

Judges the model's raw probability scores, before any yes/no threshold: the chance a randomly picked positive is ranked above a randomly picked negative. 0.90 means the churner outranks the stayer 90% of the time; 0.5 is random ordering.

So which one should I look at?

Your situation	Lead with	Keep an eye on
Predicting a number, want a plain-language error	MAE	RMSE — a big gap above MAE means occasional bad misses
Predicting a number, big misses are costly	RMSE	R² — is the model better than guessing the average at all?
Forecasting a series	MAPE	sMAPE when values dip near zero
Categories, classes roughly balanced	Accuracy	F1 — a quick weakness detector
Categories, one class is rare	Balanced accuracy	Recall on the rare class + precision on the alarms
Comparing models before choosing a threshold	ROC AUC	Precision/recall at the threshold you will actually use

These are the same metric names your model card shows — and every score there was measured on held-back data with cross-validation, exactly as lesson 1 describes.

This lesson describes Bayanii's real pipeline in simplified terms. It is updated by hand when the pipeline changes, so wording may occasionally lag the code.