Handling Missing Data

Why Missing Data Matters

Real-world datasets are almost never complete. Patient records get skipped, sensors drop readings, and survey respondents leave questions blank. If you ignore missing values or handle them carelessly, every downstream analysis—summary statistics, machine learning models, dashboards—can produce results that are wrong or misleading. Before you impute (fill in) anything, you must ask why the data are missing, because the reason determines the correct remedy.

The Three Missingness Mechanisms

Missing Completely at Random (MCAR) means the probability that a value is missing has nothing to do with any variable in your dataset, observed or unobserved. A lab machine randomly jams and drops a reading regardless of the patient's condition—that is MCAR. Because the missing rows are a random subset of all rows, you can often remove them or use simple imputation without introducing bias. Little's MCAR test (available in Python via the `pyampute` or `statsmodels` ecosystem, and in R) provides a formal statistical check: a non-significant p-value suggests MCAR.

Missing at Random (MAR) means the probability of missingness depends on other observed variables, but not on the missing value itself. For example, older patients may skip a pain-scale question, but once you know age, whether the value is missing is unrelated to what the pain score would have been. MAR is the most common assumption in practice. Model-based methods—multiple imputation, regression imputation—work well here because they can use the related observed variables to estimate the missing values.

Missing Not at Random (MNAR) means the probability of missingness depends on the unobserved value itself. A patient with very high blood pressure may skip reporting it precisely because it is high. MNAR is the most dangerous mechanism because no standard imputation method fully corrects for it. You must either collect more data, use domain knowledge, or at minimum flag the missingness and report its potential impact on conclusions.

Diagnosing Missingness in Practice

Start visually. The `missingno` library in Python produces matrix plots and bar charts that reveal which columns have missing values and whether missingness is clustered in the same rows across columns—a pattern that suggests MAR. After the visual check, apply Little's MCAR test. If the test rejects the null hypothesis (p < 0.05), MCAR is unlikely; proceed to examine correlations between missingness indicators and other columns to distinguish MAR from MNAR.

Create a binary indicator column for each variable with missingness (1 = missing, 0 = observed). Correlate those indicators with other observed variables. Strong correlations suggest MAR. When you have theoretical reasons—supported by subject-matter experts—to believe the missing value itself drives missingness, classify the variable as MNAR.

Four Imputation Strategies

Mean, median, or mode substitution replaces each missing value with the column's central tendency. It is fast and simple, but it shrinks variance and distorts correlations. Use mean for roughly symmetric continuous variables under MCAR; use median when the column is skewed; use mode for nominal or ordinal categorical columns. Never use mean/median/mode for MNAR data.

Pandas forward-fill (`ffill`) propagates the last observed value forward in time-ordered data—useful for sensor readings or time-series vitals. It assumes values change slowly and continuously. Only apply it when the data have a meaningful time or sequence order.

scikit-learn `SimpleImputer` provides a pipeline-friendly way to apply mean, median, mode, or a constant fill value. It is especially valuable when you need the same imputation logic applied consistently during both training and inference in a machine learning pipeline.

Binary missingness-indicator flags do not fill the gap—they add a new column that records whether the original value was missing. This is the primary tool for MNAR variables and can also supplement other methods. The flag lets a model learn that missingness itself is informative.

Evaluating Your Imputation

After imputing, compare the before-and-after distribution of each column using histograms and summary statistics. A good imputation preserves the overall shape of the distribution. If the post-imputation histogram looks dramatically different—particularly if variance collapses to a spike—your method is distorting the data. For MAR variables with many missing values, consider multiple imputation (generating several filled datasets and pooling results) rather than single-value substitution.

Documentation and Rationale

Every imputation decision must be documented in your code. Write comments that state: (1) the diagnosed mechanism, (2) the evidence you used to reach that diagnosis, and (3) why the chosen method is appropriate given that mechanism. This documentation is not optional—it is a professional and ethical obligation. Analyses built on undocumented imputation choices cannot be reproduced or audited, which violates data integrity standards expected in healthcare, finance, and regulated industries.

Data Safety and Integrity Reminders

When working with any dataset that contains personal or health-related information—even synthetic data modeled on real records—follow your organization's data governance policies. Never store real patient data on personal devices. Synthetic datasets used for training should be clearly labeled as such and not mixed with production data. Maintain version control on your notebooks so that imputation decisions are traceable. Accuracy in data preparation is a form of professional safety: a poorly imputed dataset fed into a clinical or financial model can cause real harm.