ROC Curves and AUC Explained

An ROC (receiver operating characteristic) curve plots a classifier's true positive rate against its false positive rate at every possible decision threshold, and AUC (area under the curve) condenses that curve into one number between 0 and 1. AUC is the probability that the model scores a randomly chosen positive example higher than a randomly chosen negative one, so 0.5 means random guessing and 1.0 means perfect ranking.

What does an ROC curve show?

An ROC curve shows the tradeoff between catching positives and raising false alarms: each point is one threshold, with the false positive rate on the x-axis and the true positive rate (recall) on the y-axis. Lowering the threshold moves you up and to the right along the curve, catching more positives at the cost of more false positives.

# ROC plots:
# X-axis: False Positive Rate (FPR) = FP / (FP + TN)
# Y-axis: True Positive Rate (TPR) = TP / (TP + FN) = Recall

# Shows tradeoff between catching positives and avoiding false alarms

How do you plot an ROC curve in Python?

Get the predicted probability of the positive class with predict_proba(X_test)[:, 1], then pass the true labels and those scores to scikit-learn's roc_curve() for the curve's points and to roc_auc_score() for the area. Plot TPR against FPR and add the diagonal as the random-guess baseline.

from sklearn.metrics import roc_curve, roc_auc_score
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
import matplotlib.pyplot as plt

# Create dataset
X, y = make_classification(n_samples=1000, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=42
)

# Train model
model = LogisticRegression()
model.fit(X_train, y_train)

# Get prediction probabilities (not just 0/1)
y_scores = model.predict_proba(X_test)[:, 1]

# Calculate ROC curve points
fpr, tpr, thresholds = roc_curve(y_test, y_scores)

# Calculate AUC
auc = roc_auc_score(y_test, y_scores)

# Plot
plt.plot(fpr, tpr, label=f'Model (AUC = {auc:.2f})')
plt.plot([0, 1], [0, 1], 'k--', label='Random (AUC = 0.50)')
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('ROC Curve')
plt.legend()
plt.show()

What is a good AUC score?

As a rough guide, an AUC above 0.9 is excellent, 0.8 to 0.9 is good, 0.7 to 0.8 is fair and 0.5 is no better than random guessing. What counts as good depends on how hard the problem is, so compare against a simple baseline model on the same test set rather than a fixed cutoff.

# AUC = 1.0: Perfect classifier
# AUC = 0.9-1.0: Excellent
# AUC = 0.8-0.9: Good
# AUC = 0.7-0.8: Fair
# AUC = 0.6-0.7: Poor
# AUC = 0.5: Random guessing (no better than coin flip)
# AUC < 0.5: Worse than random (scores are likely inverted;
#            flipping them turns AUC into 1 - AUC)

What does AUC actually measure?

AUC measures ranking: it is the share of all (positive, negative) pairs in which the model gives the positive example the higher score, with ties counting as half. With three positives scored 0.9, 0.8 and 0.4 and three negatives scored 0.7, 0.3 and 0.2, eight of the nine pairs are ranked correctly (only 0.4 vs 0.7 is wrong), so AUC = 8/9 = 0.889.

from sklearn.metrics import roc_auc_score

y_true_demo   = [1, 1, 1, 0, 0, 0]
y_scores_demo = [0.9, 0.8, 0.4, 0.7, 0.3, 0.2]

print(f"{roc_auc_score(y_true_demo, y_scores_demo):.3f}")  # 0.889

Because only the order of the scores matters, AUC stays the same under any strictly increasing transformation of the scores, and it says nothing about whether the predicted probabilities are calibrated. It is also threshold-free: a model with a high AUC can still perform badly at the one threshold you deploy, so check the confusion matrix at that cutoff as well.

Comparing Multiple Models

Plot every model's ROC curve on the same axes, computed from the same test set, and compare their AUCs. The curve that sits higher and further left is better across all thresholds; if two curves cross, the better model depends on which false positive rate you will actually operate at.

from sklearn.ensemble import RandomForestClassifier
from sklearn.naive_bayes import GaussianNB

# Train multiple models
models = {
    'Logistic Regression': LogisticRegression(),
    'Random Forest': RandomForestClassifier(),
    'Naive Bayes': GaussianNB()
}

plt.figure(figsize=(10, 6))

for name, model in models.items():
    model.fit(X_train, y_train)
    y_scores = model.predict_proba(X_test)[:, 1]

    fpr, tpr, _ = roc_curve(y_test, y_scores)
    auc = roc_auc_score(y_test, y_scores)

    plt.plot(fpr, tpr, label=f'{name} (AUC = {auc:.2f})')

plt.plot([0, 1], [0, 1], 'k--', label='Random')
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('ROC Curves - Model Comparison')
plt.legend()
plt.show()

# Higher AUC = Better model

Finding Optimal Threshold

A common default is Youden's J statistic: choose the threshold that maximizes TPR − FPR, the point on the curve furthest above the diagonal. It gives the true positive rate and the false positive rate equal weight, which treats each error equally only when the classes are balanced; with 1% positives it implicitly weighs one missed positive the same as 99 false alarms. When your real error costs differ, pick the threshold from a target false positive rate or recall instead, as in the credit scoring example below.

from sklearn.metrics import roc_curve
import numpy as np

# Get ROC curve
fpr, tpr, thresholds = roc_curve(y_test, y_scores)

# Find threshold that maximizes TPR - FPR (Youden's J)
optimal_idx = np.argmax(tpr - fpr)
optimal_threshold = thresholds[optimal_idx]

print(f"Optimal threshold: {optimal_threshold:.3f}")
print(f"TPR at optimal: {tpr[optimal_idx]:.3f}")
print(f"FPR at optimal: {fpr[optimal_idx]:.3f}")

# Use this threshold for predictions
y_pred_optimal = (y_scores >= optimal_threshold).astype(int)

When should you use a precision-recall curve instead of ROC?

Use a precision-recall curve when positives are rare and you mainly care about the positive class. When negatives vastly outnumber positives, even a small false positive rate means many false alarms, so the ROC curve can look strong while precision is poor.

For example, with 10,000 cases of which 100 are positive, a threshold that catches 80 of them (TPR = 0.80) while flagging 495 of the 9,900 negatives (FPR = 0.05) sits near the top-left corner of the ROC plot, yet precision is only 80 / (80 + 495) = 13.9%. A random model's precision-recall curve is a flat line at the positive rate (0.01 here), not the diagonal line (AUC = 0.5) that a random model traces on an ROC plot.

# ROC Curve: Use when classes are balanced
# Precision-Recall Curve: Use when classes are imbalanced

from sklearn.metrics import precision_recall_curve, average_precision_score

# For imbalanced data
precision, recall, _ = precision_recall_curve(y_test, y_scores)
ap = average_precision_score(y_test, y_scores)

plt.plot(recall, precision, label=f'AP = {ap:.2f}')
plt.xlabel('Recall')
plt.ylabel('Precision')
plt.title('Precision-Recall Curve')
plt.legend()
plt.show()

Multi-Class ROC

ROC is defined for two classes, so with more classes the usual approach is one-vs-rest: binarize the labels, draw one curve per class against all the others, and summarize with roc_auc_score(y_true, y_proba, multi_class='ovr'), which averages the per-class AUCs (a macro average by default). The alternative is one-vs-one, multi_class='ovo', which averages the AUC over every pair of classes.

from sklearn.datasets import load_iris
from sklearn.preprocessing import label_binarize

# 3-class example (iris)
X3, y3 = load_iris(return_X_y=True)
X3_train, X3_test, y3_train, y3_test = train_test_split(
    X3, y3, test_size=0.3, random_state=42, stratify=y3
)

clf = LogisticRegression(max_iter=1000).fit(X3_train, y3_train)
y3_scores = clf.predict_proba(X3_test)  # one probability column per class

# One-vs-rest: one ROC curve per class
y3_test_bin = label_binarize(y3_test, classes=[0, 1, 2])
for i in range(3):
    fpr, tpr, _ = roc_curve(y3_test_bin[:, i], y3_scores[:, i])
    auc = roc_auc_score(y3_test_bin[:, i], y3_scores[:, i])
    plt.plot(fpr, tpr, label=f'Class {i} (AUC = {auc:.2f})')

plt.plot([0, 1], [0, 1], 'k--', label='Random')
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('Multi-Class ROC Curves (One-vs-Rest)')
plt.legend()
plt.show()

# One summary number: macro-averaged one-vs-rest AUC
print(roc_auc_score(y3_test, y3_scores, multi_class='ovr'))

Cross-Validated ROC

Cross-validation gives a more reliable AUC than a single train/test split: cross_val_predict(..., method='predict_proba') scores every row with a model that never saw it, and you build the ROC curve from those out-of-fold probabilities. See cross-validation in machine learning for how the folds work.

from sklearn.model_selection import cross_val_predict

# Get cross-validated predictions
y_scores_cv = cross_val_predict(
    LogisticRegression(), X, y,
    cv=5,
    method='predict_proba'
)[:, 1]

# Calculate ROC from CV predictions
fpr, tpr, _ = roc_curve(y, y_scores_cv)
auc_cv = roc_auc_score(y, y_scores_cv)

print(f"Cross-validated AUC: {auc_cv:.3f}")

# More reliable estimate of true performance

Practical Example: Credit Scoring

When the business can tolerate only a fixed share of false alarms, read the threshold off the ROC curve at that false positive rate instead of maximizing a generic score. Here the positive class is "will default", so a false positive rejects a good borrower and a false negative approves a loan that later defaults.

# Scenario: predict loan default (1 = default)
# False positive = reject a good borrower (lost business)
# False negative = approve a borrower who defaults (lost money)

# Policy: reject at most 10% of good borrowers (FPR <= 10%)
# and catch as many defaults as possible within that limit
fpr, tpr, thresholds = roc_curve(y_test, y_scores)

target_fpr = 0.10
# fpr only increases along the curve, so the last point within
# the limit is the one with the highest TPR
idx = np.where(fpr <= target_fpr)[0][-1]

selected_threshold = thresholds[idx]
selected_tpr = tpr[idx]

print(f"Threshold: {selected_threshold:.3f}")
print(f"FPR: {fpr[idx]:.1%}, TPR (Recall): {selected_tpr:.1%}")
print(f"We catch {selected_tpr:.1%} of defaults while rejecting at most {target_fpr:.0%} of good borrowers")

Key Concepts

Five ideas cover most of what you need to read an ROC chart: the two rates on its axes, the diagonal baseline, the ideal top-left corner and the area under the curve.

• TPR (Recall): % of actual positives correctly identified
• FPR: % of actual negatives incorrectly flagged
• Diagonal line: Random classifier performance
• Top-left corner: Perfect classifier (TPR=1, FPR=0)
• AUC: Single number summarizing overall performance

What are the most common ROC curve mistakes?

The most common mistake is computing ROC and AUC from predicted 0/1 labels instead of probabilities. With labels there is only one threshold, so the curve is two straight lines and AUC collapses to balanced accuracy: for the six-example dataset above, thresholding the scores at 0.5 gives an AUC of 0.667 instead of 0.889.

• Using predict() output instead of predict_proba() or decision_function() scores
• Calculating AUC on the training data, which overstates performance
• Not using stratified splits with imbalanced data, leaving too few positives in the test set for a stable curve
• Comparing AUCs computed on different datasets or different test splits
• Reading AUC as accuracy: an AUC of 0.90 does not mean 90% of predictions are correct
• Relying on ROC alone when positives are rare (check the precision-recall curve too)