Model Evaluation: Accuracy, Precision, Recall, F1

Objectives

After this lesson, you will be able to:

• Recognize why accuracy is deceptive on imbalanced datasets, and always compare it against a majority-class baseline.
• Read a confusion matrix and compute precision, recall, and F1 from the four quadrants (TP, FP, FN, TN). Derive why F1 relies on the harmonic mean.
• Understand the precision–recall trade-off when tuning decision thresholds, identifying when to prioritize precision versus recall, and selecting thresholds on a validation set.
• Interpret precision–recall curves, ROC curves with AUC, and evaluate macro avg versus weighted avg in scikit-learn’s classification_report.

Prerequisites

• Logistic Regression, in Machine Learning (Learned)Machine Learning

Quick Reminder

This lesson evaluates binary classification models: the target label $y$ takes either 1 (positive class, typically the condition or anomaly we seek) or 0 (negative class). Models such as logistic regression do not output hard labels directly, but rather a probability $p$ that a sample belongs to the positive class. As discussed in Introduction to Machine Learning, all hyperparameter selections must take place on the validation set, reserving the test set strictly for final reporting.

Intuition

A clinic evaluates a diagnostic screening model on 20 patients with confirmed laboratory test results. Exactly 6 of them have the disease. For each patient, the model outputs an estimated probability $p$ of having the disease. The table below lists patients ranked in descending order of $p$ (synthetic data generated for this lesson).

Patient	Probability $p$	True Label $y$
1	0.93	1
2	0.86	1
3	0.78	0
4	0.71	1
5	0.63	0
6	0.58	1
7	0.54	0
8	0.46	1
9	0.42	0
10	0.36	0
11	0.31	1
12	0.27	0
13	0.22	0
14	0.18	0
15	0.14	0
16	0.12	0
17	0.09	0
18	0.07	0
19	0.04	0
20	0.02	0

A Trivial Model Can Still Score 70%

The most obvious evaluation approach is counting correct predictions. By standard convention, anyone with $p \ge 0.5$ is predicted as “diseased”. Patients 1 through 7 are predicted positive; the remaining 13 are predicted negative. Comparing with column $y$, the model is correct for 15 out of 20 patients, yielding 75% accuracy.

That seems reasonable. Now consider a “dummy” model that learns nothing and predicts healthy every time. It correctly identifies all 14 healthy individuals, securing $14/20 = 70\%$ accuracy. The trained model is merely 5 percentage points better.

Worse yet, the trivial “always healthy” model detects not a single sick patient. It is entirely useless for medical screening, yet boasts a 70% score. If the disease were rarer—say, 1 in 100 people—a dummy model would trivially reach 99% accuracy.

Not All Errors Are Created Equal

A single 75% accuracy figure conceals a critical nuance: what types of mistakes the model makes. The 5 misclassified patients fall into two distinct categories:

• Patients 3, 5, and 7 are healthy but flagged as sick. They undergo unnecessary confirmatory testing, incurring financial cost and anxiety.
• Patients 8 and 11 have the disease but are told they are healthy. They return home, missing timely intervention.

In medical screening, the second failure mode is vastly more dangerous. Accuracy treats both errors as completely equal.

Furthermore, the 0.5 threshold is an arbitrary design choice. Lowering it to 0.3 catches patients 8 and 11, but also misdiagnoses patients 9 and 10. Each threshold produces a different trade-off between error types.

The core problem of this lesson is: how to evaluate classification quality so that specific error types are exposed—especially under class imbalance—and how to select thresholds aligned with real-world operating costs.

Theory

From Probabilities to Labels: Decision Thresholds

Let $p^{(i)}$ be the model’s estimated probability that sample $i$ belongs to the positive class. To assign a predicted label $\hat{y}^{(i)}$, we compare $p^{(i)}$ against a decision threshold $t$:

$$\hat{y}^{(i)} = \begin{cases} 1 & \text{if } p^{(i)} \ge t \\ 0 & \text{if } p^{(i)} < t \end{cases}$$

Scikit-learn’s predict method applies $t = 0.5$ by default (strictly predicting positive when $p > 0.5$, differing only at exact ties), but nothing obligates using 0.5. The model learns probabilities $p$; choosing $t$ is your decision based on asymmetric error costs. All metrics discussed below (except AUC) are calculated after choosing $t$; shifting $t$ shifts the metrics.

Confusion Matrix

Comparing predicted label $\hat{y}$ with true label $y$, every sample falls into one of four distinct categories:

True Label	Predicted Positive ($\hat{y} = 1$)	Predicted Negative ($\hat{y} = 0$)
Positive ($y = 1$)	TP: True Positive	FN: False Negative
Negative ($y = 0$)	FP: False Positive	TN: True Negative

• TP (True Positive): Has disease, predicted positive.
• FN (False Negative): Has disease, predicted negative. A missed detection (Type II error).
• FP (False Positive): Healthy, predicted positive. A false alarm (Type I error).
• TN (True Negative): Healthy, predicted negative.

The second letter (P or N) represents the model’s prediction. The first letter (T or F) indicates whether that prediction is correct. This layout is the confusion matrix.

For our 20 patients with $t = 0.5$: TP comprises patients 1, 2, 4, 6; FP includes 3, 5, 7; FN contains 8 and 11; the remaining eleven individuals are TN.

True Label	Predicted Positive	Predicted Negative
Positive (6 patients)	TP = 4	FN = 2
Negative (14 patients)	FP = 3	TN = 11

Row sums represent actual class sizes in the dataset and remain constant regardless of threshold. Column sums represent the number of predicted positives and negatives, varying with $t$.

Accuracy

Accuracy is the proportion of correctly classified instances across all samples—the sum of the diagonal divided by the total:

$$\text{Accuracy} = \frac{\text{TP} + \text{TN}}{\text{TP} + \text{TN} + \text{FP} + \text{FN}} = \frac{4 + 11}{20} = 0.75$$

The “always negative” model has $\text{TP} = 0, \text{FP} = 0, \text{FN} = 6, \text{TN} = 14$, giving accuracy $14/20 = 0.70$. The vast majority of points contributing to accuracy stem from the TN quadrant—the dominant class. The more imbalanced the classes, the more TN overwhelms the score, rendering accuracy uninformative regarding the minority positive class.

Precision and Recall

To evaluate the positive class specifically, we use two complementary metrics:

Precision: Among all samples predicted positive, what proportion is truly positive? We divide TP by the entire “predicted positive” column:

$$\text{Precision} = \frac{\text{TP}}{\text{TP} + \text{FP}} = \frac{4}{4 + 3} \approx 0.571$$

Recall (also known as sensitivity or true positive rate): Among all samples that are truly positive, what proportion did the model catch? We divide TP by the entire “actual positive” row:

$$\text{Recall} = \frac{\text{TP}}{\text{TP} + \text{FN}} = \frac{4}{4 + 2} \approx 0.667$$

Both formulas share the numerator TP, differing only in the denominator. Precision is degraded by false alarms (FP). Recall is degraded by missed detections (FN).

For the negative class, an analogous metric to recall is specificity (true negative rate): the fraction of actual negatives correctly classified as negative:

$$\text{Specificity} = \frac{\text{TN}}{\text{TN} + \text{FP}} = \frac{11}{11 + 3} \approx 0.786$$

In medicine, diagnostic tests are commonly characterized by sensitivity and specificity. In machine learning, the precision–recall pair is more prevalent.

For the trivial “always negative” baseline: $\text{recall} = 0/6 = 0$, immediately unmasking what 70% accuracy disguised. Its precision evaluates to $0/0$: having zero positive predictions, it is mathematically undefined. Scikit-learn issues a warning and returns 0 by default (customizable via the zero_division parameter).

F1: Combining Precision and Recall into a Single Metric

Comparing models with a single number is often convenient. The arithmetic mean $(P + R)/2$ is tempting, but flawed. Consider threshold $t = 0.9$: only patient 1 is predicted positive, yielding $P = 1/1 = 1.0$ and $R = 1/6 \approx 0.167$. Their arithmetic mean is $0.583$—higher than at $t = 0.5$, despite missing 5 out of 6 sick individuals.

The F1 score resolves this by computing the harmonic mean: averaging the reciprocals, then inverting:

$$\frac{1}{F_1} = \frac{1}{2}\left(\frac{1}{P} + \frac{1}{R}\right) \quad\Longrightarrow\quad F_1 = \frac{2PR}{P + R}$$

The right-hand side follows by finding a common denominator $\frac{1}{P} + \frac{1}{R} = \frac{P + R}{PR}$ and taking the reciprocal.

Why does the harmonic mean prevent distortion? Small numbers have large reciprocals. When $R \approx 0.167$, $1/R = 6$ dominates $1/P = 1$, driving the average reciprocal high and compressing $F_1$ low. At $t = 0.9$:

$$F_1 = \frac{2 \cdot 1.0 \cdot 0.167}{1.0 + 0.167} \approx 0.286$$

F1 is strongly pulled toward the smaller of the two values. To achieve a high F1 score, both precision and recall must be high.

Expressing $P$ and $R$ directly in terms of TP, FP, and FN produces an alternative representation:

$$\begin{aligned} \frac{1}{F_1} &= \frac{1}{2}\left(\frac{\text{TP} + \text{FP}}{\text{TP}} + \frac{\text{TP} + \text{FN}}{\text{TP}}\right) \\ &= \frac{2\,\text{TP} + \text{FP} + \text{FN}}{2\,\text{TP}} \end{aligned}$$

Inverting yields:

$$F_1 = \frac{2\,\text{TP}}{2\,\text{TP} + \text{FP} + \text{FN}}$$

At $t = 0.5$: $F_1 = 8/(8 + 3 + 2) = 8/13 \approx 0.615$. Notice that the TN term is absent: a flood of true negatives cannot inflate F1 the way it inflates accuracy. Additionally, this form is computable even when positive predictions are zero: when $\text{TP} = 0$, $F_1 = 0$, whereas precision would be $0/0$.

F-beta. When one error type carries higher stakes, the generalized $F_\beta$ score balances them:

$$F_\beta = \frac{(1 + \beta^2)\,P R}{\beta^2 P + R}$$

Setting $\beta = 1$ recovers $F_1$. Setting $\beta > 1$ weights recall more heavily (e.g., $F_2$ for medical screening), whereas $\beta < 1$ favors precision (e.g., $F_{0.5}$). In scikit-learn, this is accessible via fbeta_score.

Key Takeaway

$\text{Precision} = \text{TP}/(\text{TP}+\text{FP})$ measures fidelity among positive predictions. $\text{Recall} = \text{TP}/(\text{TP}+\text{FN})$ measures coverage among actual positive cases. F1 is their harmonic mean, bound closely to the weaker metric. Accuracy includes TN, creating false optimism on imbalanced negative-heavy data.

The Precision–Recall Trade-off Across Thresholds

Varying $t$ across our 20 patients demonstrates this trade-off. The second code cell prints full sweeps from 0.1 to 0.9; representative values are summarized below:

Threshold $t$	TP	FP	FN	TN	Accuracy	Precision	Recall	F1
0.30	6	5	0	9	0.750	0.545	1.000	0.706
0.50	4	3	2	11	0.750	0.571	0.667	0.615
0.70	3	1	3	13	0.800	0.750	0.500	0.600
0.80	2	0	4	14	0.800	1.000	0.333	0.500
0.95	0	0	6	14	0.700	undefined	0.000	0.000

Raising $t$ causes fewer samples to be flagged as positive:

• Recall never increases. Samples can only drop out of the positive set, so TP is non-increasing while the denominator $\text{TP} + \text{FN}$ remains fixed at 6.
• Precision usually rises, but non-monotonically. While FP decreases, TP can also fall. Moving from $t = 0.30$ to $0.35$, the sample eliminated is patient 11 (a true positive), causing precision to drop from $0.545$ to $0.500$.
• Accuracy can rise even as model clinical utility plummets. At $t = 0.80$, accuracy climbs to $0.800$ (higher than at $0.50$), yet recall drops to $0.333$, abandoning 4 out of 6 sick patients.

There is no universally optimal threshold. The question is: which mistake is more costly?

• Prioritize recall when misses are expensive. In clinical screening, a false alarm merely requires a follow-up test, whereas a missed diagnosis may prove fatal. Lower the threshold. Fraud detection often adopts the same stance.
• Prioritize precision when false alarms are costly. In spam filtering, an occasional junk email in the inbox is minor, but diverting a critical job offer to spam can be disastrous. Raise the threshold.

Selecting $t$ tunes a hyperparameter. Therefore, optimize it on the validation set (or via cross-validation as taught in Overfitting and Underfitting), and evaluate on the test set once with the finalized threshold. Searching for the highest test F1 across thresholds commits the pitfall of “selecting models using test data” highlighted in Introduction to Machine Learning. Scikit-learn provides TunedThresholdClassifierCV to automate this procedure cleanly.

Precision–Recall and ROC Curves

Rather than inspecting discrete thresholds, sweeping across all possible thresholds traces continuous evaluation curves.

The precision–recall curve plots recall on the horizontal axis and precision on the vertical axis. Every point corresponds to a threshold. A high-performing classifier hugs the upper-right corner (high recall with sustained high precision). A random guessing baseline forms a flat horizontal line at the dataset’s positive prevalence rate ($6/20 = 0.3$).

The ROC curve (Receiver Operating Characteristic) plots the false positive rate (FPR) on the x-axis:

$$\text{FPR} = \frac{\text{FP}}{\text{FP} + \text{TN}} = 1 - \text{Specificity}$$

against recall (true positive rate) on the y-axis. Lowering $t$ from 1 to 0 traces a curve from $(0, 0)$ (predicting nobody positive) to $(1, 1)$ (predicting everyone positive). Random guessing follows the diagonal.

AUC (Area Under the Curve) measures the area under the ROC curve, ranging from 0 to 1. It carries an intuitive probabilistic interpretation: given a randomly chosen positive sample and negative sample, AUC is the probability that the model ranks the positive sample higher than the negative one. Among our 20 patients, there are $6 \times 14 = 84$ positive-negative pairs. Inspecting the table: patient 4 is ranked below 1 healthy patient (patient 3), patient 6 below 2, patient 8 below 3, patient 11 below 5. In total, 11 pairs are inverted:

$$\text{AUC} = \frac{84 - 11}{84} \approx 0.869$$

AUC depends solely on prediction rank ordering, completely independent of any chosen threshold. It is ideal for comparing model ranking capabilities prior to selecting an operating threshold. When the positive class is heavily outnumbered, the vast number of TNs keeps FPR perpetually small, making ROC curves appear deceptively optimistic. Under severe class imbalance, the precision–recall curve provides a much more faithful reflection of minority-class performance.

Multi-class: Macro, Micro, and Weighted Averages

For $K$ classes (e.g., digit recognition from 0 to 9, covered in Softmax Regression), one calculates precision, recall, and F1 for each individual class by treating it as positive and all others as negative. These $K$ scores are then aggregated using one of three strategies:

• Macro average: The unweighted arithmetic mean across all $K$ classes, weighting each class equally. Poor performance on a rare class significantly penalizes the macro score.
• Weighted average: The mean weighted by the true support (sample count) of each class. Dominant classes control the score.
• Micro average: Aggregates total TP, FP, and FN across all classes before computing a global metric. When every sample has exactly one ground truth label, micro precision, recall, and F1 all mathematically equal overall accuracy.

To diagnose whether a model neglects minority classes, always check macro avg. The third code example illustrates how widely these aggregations can diverge.

Limitations

• Small sample sizes cause high variance in metrics. With only 6 positive patients, catching or missing one individual shifts recall by $1/6 \approx 0.167$. Treat figures on 20 patients as educational illustrations.
• Metrics do not incorporate dollar costs directly. F1 treats FP and FN symmetrically; $F_\beta$ adjusts their weighting heuristically. When real business costs are known, select the threshold that minimizes expected financial loss.
• Threshold-dependent metrics do not verify probability calibration. A well-calibrated model ensures that among cases assigned $p = 0.9$, approximately 90% are truly positive. Calibration is an orthogonal property not captured by thresholded counts.
• Measuring imbalance is not equivalent to remedying it. Techniques for handling class imbalance (resampling, class-weighted loss) are detailed in Data Preprocessing.

Visualization

The interactive tool below displays all 20 patients along a probability axis from 0 to 1, colored by their true label (diseased or healthy). The slider adjusts threshold $t$ from 0 to 1 in steps of 0.05. A vertical guideline marks the current threshold; all points to the right (including those landing directly on the boundary) are predicted positive. Alongside, the $2 \times 2$ confusion matrix and the four metrics (accuracy, precision, recall, F1) update dynamically as you drag. Try the following:

1. Initial state at $t = 0.50$: $\text{TP} = 4, \text{FP} = 3, \text{FN} = 2, \text{TN} = 11$. Accuracy is 0.750, precision 0.571, recall 0.667, F1 0.615. Two positive dots lie left of the line: these represent missed cases.
2. Drag down to $t = 0.30$: all 6 diseased patients fall to the right, raising recall to 1.000. In return, FP increases to 5, lowering precision to 0.545, while F1 climbs to 0.706.
3. Drag up to $t = 0.80$: false alarms vanish ($\text{FP} = 0$), driving precision to 1.000. However, recall plummets to 0.333, yet accuracy increases to 0.800: accuracy rewards missing sick patients because healthy patients dominate the population.
4. Drag up to $t = 0.95$: nobody is predicted positive. Precision displays ”—” ($0/0$), recall and F1 hit 0, and accuracy registers 0.700—matching the dummy baseline.

Drag the threshold t. Each dot is one patient; a dot on the line or to its right is predicted ill. The confusion matrix and the four metrics below follow.

From the line rightwards (p ≥ t): predicted ill

0.00.20.40.60.81.0

Probability of being ill, p

• Ill (y = 1)
• Not ill (y = 0)
• Threshold line t

Threshold $0.50

Confusion matrix at t = 0.50. Rows are the true labels, columns the predictions.

True label	Predicted positive	Predicted negative
Positive (6)	TP = 4	FN = 2
Negative (14)	FP = 3	TN = 11

Accuracy15/200.750

Precision4/70.571

Recall4/60.667

F18/130.615

No positive predictions: precision is 0/0, undefined.

20 made-up patients, 6 ill and 14 not ill, placed by the probability of being ill that the model predicts. Most ill patients get a high probability, but the two groups overlap between about 0.3 and 0.8.

Code Examples

The three cells below run independently, each handling its own imports and synthetic datasets. The first two replicate the 20-patient dataset. Loading scikit-learn in the browser adds a brief delay on initial execution.

The first cell sets threshold $t = 0.5$, tallies the four confusion matrix cells with NumPy, computes metrics manually from definitions, and validates them against sklearn.metrics. The syntax (p >= t).astype(int) converts booleans to binary indicator arrays. The expression (y_pred == 1) & (y_true == 1) masks matching conditions, and np.sum counts them.

Confusion matrix at threshold 0.5

import numpy as np
from sklearn.metrics import (accuracy_score, confusion_matrix, f1_score,
                             precision_score, recall_score)

# 20 patients (made-up illustration data), sorted by probability, high first.
# y_true: true label (1 = has the disease). p: predicted disease probability.
y_true = np.array([1, 1, 0, 1, 0, 1, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0])
p = np.array([0.93, 0.86, 0.78, 0.71, 0.63, 0.58, 0.54, 0.46, 0.42, 0.36, 0.31, 0.27, 0.22, 0.18, 0.14, 0.12, 0.09, 0.07, 0.04, 0.02])

t = 0.5  # decision threshold
y_pred = (p >= t).astype(int)  # predict positive when p >= t

# Count the four cells of the confusion matrix.
tp = np.sum((y_pred == 1) & (y_true == 1))
fp = np.sum((y_pred == 1) & (y_true == 0))
fn = np.sum((y_pred == 0) & (y_true == 1))
tn = np.sum((y_pred == 0) & (y_true == 0))
print(f"TP = {tp}, FP = {fp}, FN = {fn}, TN = {tn}")

accuracy = (tp + tn) / len(y_true)
precision = tp / (tp + fp)
recall = tp / (tp + fn)
f1 = 2 * precision * recall / (precision + recall)  # harmonic mean
print(f"by hand: acc = {accuracy:.3f}, P = {precision:.3f}, "
      f"R = {recall:.3f}, F1 = {f1:.3f}")

# Check against scikit-learn.
print(confusion_matrix(y_true, y_pred))  # rows: true label 0, 1
print(f"sklearn: acc = {accuracy_score(y_true, y_pred):.3f}, "
      f"P = {precision_score(y_true, y_pred):.3f}, "
      f"R = {recall_score(y_true, y_pred):.3f}, "
      f"F1 = {f1_score(y_true, y_pred):.3f}")

# The "always predict healthy" model: accuracy is still high.
print(f"always 0: acc = {np.mean(y_true == 0):.3f}")

RunEdit code

The first run downloads Python, about 10 MB.

The output confirms the theory section: $\text{TP} = 4, \text{FP} = 3, \text{FN} = 2, \text{TN} = 11$. Hand calculations and scikit-learn outputs match identically: accuracy 0.750, precision 0.571, recall 0.667, F1 0.615. The final row reflects the “always healthy” dummy baseline scoring 0.700 accuracy.

Note scikit-learn’s quadrant orientation in confusion_matrix: rows represent true labels, columns represent predictions, and class 0 appears first. Thus, top-left 11 is TN, and bottom-right 4 is TP.

The second cell sweeps thresholds from 0.1 through 0.9, prints precision, recall, and F1 at each point, and plots both performance curves. Built-in utilities precision_recall_curve and roc_curve automatically evaluate all relevant candidate thresholds from continuous probabilities $p$.

Threshold sweeping, PR curve, and ROC curve

import matplotlib.pyplot as plt
import numpy as np
from sklearn.metrics import (f1_score, precision_recall_curve,
                             precision_score, recall_score, roc_auc_score,
                             roc_curve)

# 20 patients (made-up illustration data), sorted by probability, high first.
y_true = np.array([1, 1, 0, 1, 0, 1, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0])
p = np.array([0.93, 0.86, 0.78, 0.71, 0.63, 0.58, 0.54, 0.46, 0.42, 0.36, 0.31, 0.27, 0.22, 0.18, 0.14, 0.12, 0.09, 0.07, 0.04, 0.02])

# Sweep the threshold from 0.1 to 0.9 in steps of 0.1.
print("   t      P      R     F1")
for t in np.round(np.arange(0.1, 0.95, 0.1), 2):
    y_pred = (p >= t).astype(int)
    P = precision_score(y_true, y_pred, zero_division=0)  # report 0 for 0/0
    R = recall_score(y_true, y_pred)
    F = f1_score(y_true, y_pred)
    print(f"{t:.2f}  {P:.3f}  {R:.3f}  {F:.3f}")

# Two curves: each point on a curve is one threshold.
prec, rec, _ = precision_recall_curve(y_true, p)
fpr, tpr, _ = roc_curve(y_true, p)
auc = roc_auc_score(y_true, p)
print(f"ROC AUC = {auc:.3f}")

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(9, 4))
ax1.plot(rec, prec, marker="o")
ax1.axhline(y_true.mean(), ls="--", c="gray")  # random-guess level
ax1.set(xlabel="recall", ylabel="precision", title="Precision-recall")
ax2.plot(fpr, tpr, marker="o", label=f"model (AUC = {auc:.2f})")
ax2.plot([0, 1], [0, 1], ls="--", c="gray", label="random")
ax2.set(xlabel="false positive rate", ylabel="recall (TPR)", title="ROC")
ax2.legend()
plt.tight_layout()
plt.show()

RunEdit code

The first run downloads Python, about 10 MB.

Reading across the threshold table columns:

• Recall holds steady at 1.000 until threshold 0.3, then declines to 0.167 at 0.9. It never increases as thresholds rise.
• Precision rises from 0.375 toward 1.000 non-monotonically, exhibiting plateaus (0.545, 0.556, 0.571 across thresholds 0.3, 0.4, 0.5).
• F1 reaches its peak at threshold 0.3 with approximately 0.706, confirming that on this dataset, operating below the 0.5 default is beneficial.

The reported ROC AUC = 0.869 confirms our manual ranking derivation (73 out of 84 pairs). On the left plot, the sawtooth pattern in the precision–recall curve reflects discrete data points: crossing a healthy patient lowers precision, while crossing a sick patient advances recall rightward and lifts precision. On large datasets, these steps diminish into smooth curves.

The third cell tests realistic class imbalance: 2,000 synthetic samples generated via make_classification, with only 5% positive instances. Data is split into training, validation, and test subsets (60%, 20%, 20%) with stratified class proportions. The code compares three approaches: a DummyClassifier predicting majority class, logistic regression under default threshold 0.5, and logistic regression with its threshold tuned on the validation set.

Imbalanced classification and threshold tuning

import numpy as np
from sklearn.datasets import make_classification
from sklearn.dummy import DummyClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report, f1_score, recall_score
from sklearn.model_selection import train_test_split

# 2000 samples, only about 5% in the positive class (class 1).
X, y = make_classification(n_samples=2000, n_features=10, n_informative=4,
                           weights=[0.95, 0.05], class_sep=2.0,
                           random_state=0)
# Three sets: train 60%, validation 20%, test 20%, same class ratio.
X_tmp, X_test, y_tmp, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=0)
X_tr, X_val, y_tr, y_val = train_test_split(
    X_tmp, y_tmp, test_size=0.25, stratify=y_tmp, random_state=0)

# A "dumb" model: always predict the majority class (class 0).
dummy = DummyClassifier(strategy="most_frequent").fit(X_tr, y_tr)
d_pred = dummy.predict(X_test)
print(f"dummy: acc = {np.mean(d_pred == y_test):.3f}, "
      f"R = {recall_score(y_test, d_pred):.3f}")

# Logistic regression with the default 0.5 threshold (predict).
model = LogisticRegression().fit(X_tr, y_tr)
print(classification_report(y_test, model.predict(X_test), digits=3))

# Choose the threshold on the VALIDATION set (not the test set).
p_val = model.predict_proba(X_val)[:, 1]
ts = np.round(np.arange(0.05, 0.95, 0.05), 2)
f1s = [f1_score(y_val, (p_val >= t).astype(int)) for t in ts]
best_t = ts[int(np.argmax(f1s))]
print(f"best threshold on validation: {best_t:.2f}")

# Score once on the test set with the chosen threshold.
p_test = model.predict_proba(X_test)[:, 1]
y_hat = (p_test >= best_t).astype(int)
print(classification_report(y_test, y_hat, digits=3))

RunEdit code

The first run downloads Python, about 10 MB.

The test set contains 400 instances, including 22 positive samples (support column):

• Dummy model: Accuracy is 0.945 yet recall is 0.000—failing to capture a single positive case.
• Default logistic regression ($t = 0.5$): Accuracy is 0.960, barely 1.5 percentage points above the dummy model. Inspecting class 1 reveals reality: precision is 1.000 but recall is only 0.273 (capturing merely 6 of 22 positive cases). The model is excessively timid.
• Validation-tuned threshold ($t = 0.20$): Evaluated once on the held-out test set, recall jumps to 0.500 (11 of 22 caught), precision settles at 0.647, and class 1 F1 rises from 0.429 to 0.564. Overall accuracy remains virtually flat (0.958 vs. 0.960). Relying on accuracy alone completely conceals this doubling in minority-class detection.

The bottom rows illustrate aggregation strategies. Under threshold 0.5, macro avg F1 is 0.704, whereas weighted avg is 0.949. Weighted averaging assigns overwhelming weight to class 0 (378 instances), concealing poor performance on the minority class. Macro averaging weights both classes equally, immediately highlighting the deficiency.

Common Pitfalls

Reporting Accuracy Without a Baseline

A test accuracy of 0.960 sounds impressive until one learns that an uninformative majority-class predictor achieves 0.945. Always contextualize accuracy against DummyClassifier(strategy="most_frequent"), and report precision, recall, and F1 for minority classes (classification_report displays all three by default).

Tuning Decision Thresholds on the Test Set

Sweeping thresholds directly on test data to report the best obtainable F1 invalidates evaluation by treating the test set as validation data. The reported metric will be over-optimistic compared to future real-world performance. Optimize thresholds on a validation set or via cross-validation (TunedThresholdClassifierCV), evaluating the test set strictly once.

Reporting Precision or Recall in Isolation

Either metric is trivial to game in isolation. Predicting positive on every sample ($t = 0$) guarantees 1.000 recall despite miserable precision. Conversely, predicting positive on only the single most confident instance ($t = 0.9$) guarantees 1.000 precision while sacrificing recall. Always report both together alongside the operating threshold, or summarize them via F1 and precision–recall curves.

Quick Check

Question 1 of 4 There are 1,000 credit card transactions, with 10 fraudulent transactions (positive class). A model always predicts "legitimate". What are its accuracy and recall?

Accuracy 99%, recall 99%Accuracy 1%, recall 0Accuracy 99%, recall 0Accuracy 99%, recall is undefined due to division by zero

Check

Question 2 of 4 A model has precision $0.9$ and recall $0.1$. Which value is closest to its F1 score?

$0.5$$0.18$$0.9$$0.1$

Check

Question 3 of 4 You raise the decision threshold from $0.5$ to $0.7$ (predicting positive when $p \ge t$). Which statement is always true?

Precision strictly increases and recall strictly decreasesAccuracy strictly increases because the model is more conservativeBoth precision and recall increaseRecall cannot increase; precision usually increases but may decrease

Check

Question 4 of 4 You are building a spam filter where the positive class is "spam". Filtered emails move to the spam folder and are rarely checked by users. Which metric should you prioritize when setting the threshold?

Precision, because an email flagged as spam must genuinely be spamRecall, because as much spam as possible must be caughtAccuracy, because most emails are legitimateNeither, because a 0.5 threshold is always optimal

Check

Summary and Further Reading

Key Takeaways:

• Accuracy $= (\text{TP}+\text{TN})/m$ is easily distorted by class imbalance: a majority-class predictor matches the majority prevalence (70% on 20 patients, 94.5% in our synthetic experiment). Always compare against this baseline.
• A confusion matrix partitions errors into false alarms (FP) and missed detections (FN). $\text{Precision} = \text{TP}/(\text{TP}+\text{FP})$ penalizes false alarms, while $\text{Recall} = \text{TP}/(\text{TP}+\text{FN})$ penalizes misses. $F_1 = 2PR/(P+R) = 2\text{TP}/(2\text{TP}+\text{FP}+\text{FN})$ is their harmonic mean, tethered closely to the weaker score.
• The model outputs continuous probabilities $p$; selecting threshold $t$ is your choice. Raising $t$ cannot increase recall and typically raises precision. Prioritize recall when misses are catastrophic (medical screening); prioritize precision when false alarms are intolerable (spam filtering). Optimize $t$ on validation data.
• Precision–recall and ROC curves sweep across all thresholds. AUC represents the probability that a random positive instance is ranked above a random negative one; under severe imbalance, precision–recall curves provide more truthful feedback. For multi-class settings, macro averages treat all classes equally, weighted averages scale by support, and micro averages equate to accuracy.

Further Reading:

• Vũ Hữu Tiệp (2017). Machine Learning cơ bản, Bài 33: Các phương pháp đánh giá một hệ thống phân lớp. Vietnamese resource covering confusion matrices, ROC/AUC, precision–recall, F1, micro and macro averaging.
• Google (n.d.). Machine Learning Crash Course: Accuracy, recall, precision, and related metrics and ROC and AUC. Concise tutorials explaining trade-offs and the probabilistic ranking intuition behind AUC.
• scikit-learn developers (2007–2026). scikit-learn 1.8 User Guide: Metrics and scoring, Section 3.4.4. Formal definitions for macro, micro, weighted, and zero_division. See also Section 3.3 Tuning the decision threshold detailing TunedThresholdClassifierCV.
• Fawcett, T. (2006). An introduction to ROC analysis. Pattern Recognition Letters 27(8), 861–874. Foundational paper on ROC and AUC analysis.
• Saito, T., Rehmsmeier, M. (2015). The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets. PLOS ONE 10(3), e0118432. Demonstrates why precision–recall curves are superior to ROC under extreme class imbalance.

I have finished this lesson