Common ML Failure Modes: Overfitting, Leakage, and Class Imbalance

Common ML Failure Modes: Overfitting, Leakage, and Class Imbalance

Building a machine learning model is only half the job. The other half is making sure the model actually works on data it has never seen. Three failure modes destroy more deployed models than almost any other cause: overfitting, data leakage, and class imbalance. Understanding each one—and knowing how to fix it—is a core competency tested by CompTIA Data+, the Google Data Analytics Certificate, and the IBM Data Science Professional Certificate.

Overfitting

A model overfits when it memorizes the training data instead of learning the underlying pattern. The symptom is a large gap between training accuracy and validation (or test) accuracy. For example, a decision tree with no depth limit can grow one branch per training sample, achieving nearly 100% training accuracy while performing little better than a coin flip on new data.

Why it happens. Overfitting occurs when a model has too many parameters relative to the amount of training data, or when training continues too long without a stopping criterion.

How to fix it.

• Regularization penalizes model complexity. Ridge regression (L2 regularization) adds a penalty proportional to the sum of squared coefficients, shrinking large weights toward zero. In scikit-learn, set `alpha` in `Ridge()` to control the penalty strength. For tree-based models, restrict `max_depth` or increase `min_samples_leaf`.
• Cross-validation gives a more honest estimate of generalization error than a single train/test split.
• More data or data augmentation dilutes noise that the model would otherwise memorize.

What to check in a notebook. Look at the gap between training score and validation score. If training accuracy is 0.98 and validation accuracy is 0.62, the model is overfitting. After applying regularization, both numbers should converge—even if the training score drops.

Data Leakage

Data leakage is the accidental inclusion of information in training that would not be available at prediction time. It produces suspiciously good metrics during development and catastrophic performance in production.

Common sources.

• Fitting a scaler, encoder, or imputer on the entire dataset before splitting into train and test sets. The scaler then carries information about the test distribution into the training step.
• Including a feature that is a direct consequence of the target variable (e.g., using a discharge code to predict whether a patient was admitted).
• Time-series data where future values inadvertently appear in features for past observations.

How to fix it. Wrap all preprocessing steps inside a scikit-learn `Pipeline`. When the pipeline is fit on training data only, the scaler sees only training statistics. When it transforms the test set, it uses those same training statistics—mimicking real-world deployment accurately. Moving `StandardScaler()` from outside the pipeline to inside is often the single line change that eliminates leakage.

What to check in a notebook. Look for `scaler.fit(X)` or `scaler.fit_transform(X)` called before `train_test_split`. That is the classic leakage bug. Also look for features with suspiciously high correlations to the target.

Class Imbalance

Class imbalance occurs when one class (the majority class) vastly outnumbers another (the minority class). A fraud-detection dataset might have 990 legitimate transactions for every 10 fraudulent ones. A naive model that always predicts "legitimate" achieves 99% accuracy while being completely useless.

Why accuracy is misleading here. Accuracy counts all correct predictions equally. In imbalanced data, the majority class dominates. Use precision, recall, and F1-score for the minority class, or the ROC-AUC score, to measure real performance.

How to fix it.

• Class weights. In scikit-learn classifiers, set `class_weight='balanced'`. The algorithm then assigns higher penalty to misclassifying the minority class, proportional to how underrepresented it is.
• SMOTE (Synthetic Minority Over-sampling Technique). The `imbalanced-learn` library (`imblearn`) provides `SMOTE`, which generates synthetic minority-class samples by interpolating between existing ones. Apply SMOTE inside a pipeline using `imblearn.pipeline.Pipeline` (not scikit-learn's pipeline) so that synthetic samples are created only from training data, never from the test set. Applying SMOTE before splitting is a form of data leakage.
• Under-sampling the majority class is an alternative when data are plentiful.

What to check in a notebook. Run `y.value_counts()` early. If the ratio is 10:1 or worse, imbalance is a concern. Check whether the reported metric is accuracy alone—if so, also compute minority-class recall.

Connecting Failure Modes to Real-World Risk

These are not academic edge cases. Production ML systems have failed in high-stakes settings because of each of these bugs. A model deployed on data that has shifted away from its training distribution (a related problem called distribution shift) behaves like an overfitted model in reverse: it learned real patterns, but those patterns no longer match the new world. Recognizing the red flags early—large train/validation gaps, implausibly high metrics, poor minority-class recall—is what separates a professional data analyst from someone who just runs code.

A Note on Reproducibility and Safety

When sharing or submitting notebooks, always set a random seed (`random_state=42` is conventional) so results are reproducible. Never include real personally identifiable information (PII) or protected health information (PHI) in practice notebooks, even anonymized versions. Confirm your instructor's data-handling policy before loading any external dataset.