Handling Missing Data in Python: MCAR, MAR, MNAR and Imputation

A small finance company in Indore receives 1,000 loan applications through its app. When the analyst opens the data, the monthly income field is blank for 180 people.

She looks closer. 30 blanks came from a single day when the app crashed. 110 belong to self-employed shopkeepers, whose form screen made income optional. The remaining 40 are salaried applicants who simply chose not to answer, and the collection team later found those applicants earned less than average.

Same blank cell, three different stories. Filling all 180 with the average income would be right for the first group, wrong for the second and dangerously wrong for the third.

What is missing data, and why does the reason matter?

Map the loan story to the terms:

• The app crash is missing completely at random (MCAR): the gap has nothing to do with anything in the data.
• The shopkeepers are missing at random (MAR): the gap depends on another column you do have (occupation), not on the income itself.
• The silent low earners are missing not at random (MNAR): the gap depends on the missing value itself.

Handling missing data is the process of finding empty values, working out why they are missing, and then dropping, imputing or flagging them so analysis and models are not biased.

These three categories were formalised by the statistician Donald Rubin in 1976 and are still the standard way to reason about gaps.

Under MCAR you lose rows but not truth; under MAR other columns can repair the gap; under MNAR the blank itself carries information.

You can never prove MNAR from the data alone, because the evidence is the value you don't have. You argue it from domain knowledge, as the collection team did. This is why talking to the people who collected the data is part of the job.

This article builds on the EDA steps from the previous lesson, which is where you first count the gaps. Basic pandas cleaning such as dropna and fillna is covered in data cleaning with pandas; here the focus is on choosing a strategy that keeps models honest. Code targets pandas 2.x and scikit-learn 1.4+.

How do you find missing values in pandas?

Three questions matter: how many, where, and do the gaps follow a pattern?

import numpy as np, pandas as pd

rng = np.random.default_rng(7)
n = 1000
df = pd.DataFrame({
    "age": rng.integers(21, 60, n),
    "occupation": rng.choice(["salaried", "self-employed"], n, p=[0.7, 0.3]),
    "income_k": rng.normal(45, 15, n).round(1),
})
self_emp = df["occupation"] == "self-employed"
df.loc[self_emp & (rng.random(n) < 0.35), "income_k"] = np.nan   # MAR
df.loc[rng.random(n) < 0.03, "income_k"] = np.nan                # MCAR

print("Missing count:", df["income_k"].isna().sum())
print("Missing %:", round(df["income_k"].isna().mean() * 100, 1))
print("\nMissing % by occupation:")
print(df.groupby("occupation")["income_k"].apply(lambda s: s.isna().mean()).round(3))

The last table is the key diagnostic. If one group is missing far more often than another, the data is not MCAR, and simply dropping rows would remove mostly self-employed applicants. That changes who your model learns from.

Hidden missing values

Not every gap shows up as NaN. Watch for placeholders such as 0 for age, -1, "NA", "unknown", "-" or an empty string. Convert them with df.replace({"-": np.nan, "unknown": np.nan}) before counting, or your counts will be too low.

Strategies for handling missing data

Start with how much is missing and why; only then pick a method.

Why mean imputation can mislead

Filling every gap with one number piles many rows onto a single value. The column's standard deviation drops, and its correlation with other columns weakens. Run this to see it.

import numpy as np, pandas as pd

rng = np.random.default_rng(1)
age = rng.integers(22, 60, 2000)
income = 10 + 1.1 * age + rng.normal(0, 8, 2000)
df = pd.DataFrame({"age": age, "income_k": income})
print("True std:", round(df["income_k"].std(), 2),
      "| true corr:", round(df["age"].corr(df["income_k"]), 3))

holes = df.copy()
holes.loc[rng.random(2000) < 0.3, "income_k"] = np.nan
filled = holes["income_k"].fillna(holes["income_k"].mean())
print("After mean fill std:", round(filled.std(), 2),
      "| corr:", round(df["age"].corr(filled), 3))

With 30% of values replaced by the mean, the spread and the age-income relationship both shrink. A regression trained on this would underestimate how much income rises with age.

Using other columns to impute

When data is MAR, the columns you have can predict the one you don't. The simplest version is a group-wise median: fill a self-employed applicant's income with the median of other self-employed applicants.

scikit-learn offers two general tools. KNNImputer fills a gap with the average of the k most similar rows. IterativeImputer fits a regression for each column using the others, and repeats a few rounds. It is still marked experimental, so it needs an extra import line.

import numpy as np
from sklearn.experimental import enable_iterative_imputer  # noqa: F401
from sklearn.impute import SimpleImputer, KNNImputer, IterativeImputer

rng = np.random.default_rng(3)
age = rng.integers(22, 60, 600).astype(float)
income = 10 + 1.1 * age + rng.normal(0, 6, 600)
X = np.column_stack([age, income])
mask = rng.random(600) < 0.25
X_missing = X.copy()
X_missing[mask, 1] = np.nan

for name, imp in [("median", SimpleImputer(strategy="median")),
                  ("knn", KNNImputer(n_neighbors=5)),
                  ("iterative", IterativeImputer(random_state=0))]:
    filled = imp.fit_transform(X_missing)
    err = np.abs(filled[mask, 1] - X[mask, 1]).mean()
    print(f"{name:>9}: mean absolute error on filled cells = {err:.2f}")

Because income depends on age here, the methods that use age land much closer to the truth than the plain median. KNN needs scaled inputs on real data, since it uses distances between rows.

Flag the gap as a feature

For MNAR, the fact that a value is missing is information. Low earners skipped the question, so "income was blank" predicts risk. SimpleImputer(add_indicator=True) fills the gap and adds a 0/1 column marking which rows were filled. Tree models such as random forests use these flags well.

How to avoid data leakage when imputing

If you compute the median on the full dataset and then split into train and test, the test rows have already influenced the training data. That is data leakage, and it makes your scores look better than reality.

Follow these steps:

1. Split into training and test sets first.
2. Put the imputer inside a scikit-learn Pipeline with the model.
3. Call fit on training data only; the pipeline learns the fill values there.
4. Call predict or score on test data; the pipeline reuses the training fill values.
5. Use the same pipeline in cross-validation, so each fold learns its own fill values.

import numpy as np
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline

rng = np.random.default_rng(5)
income = rng.normal(45, 15, 1500)
default = (income + rng.normal(0, 10, 1500) < 35).astype(int)
skipped = (income < 35) & (rng.random(1500) < 0.5)          # MNAR
X = income.reshape(-1, 1).copy()
X[skipped, 0] = np.nan

X_tr, X_te, y_tr, y_te = train_test_split(X, default, test_size=0.3, random_state=0)
for flag in (False, True):
    pipe = make_pipeline(SimpleImputer(strategy="median", add_indicator=flag),
                         LogisticRegression())
    pipe.fit(X_tr, y_tr)
    print(f"add_indicator={flag}: test accuracy {pipe.score(X_te, y_te):.3f}")

The indicator lifts accuracy because skipping the question is itself a warning sign. If you would like to work through messy real datasets like this with guidance, Debugshala's hands-on data science classes in Old Palasia include imputation and pipeline exercises.

Common mistakes with missing values

• Dropping rows by reflex. If gaps cluster in one group, dropna() quietly removes that group.
• Imputing before splitting. Fill values must come from training data only.
• Missing the hidden placeholders. Zeros, "-" and "unknown" are gaps too.
• Mean-filling skewed columns. Income and prices are skewed; the median is safer.
• Filling the target column. Rows without a label should be set aside, not given an invented label.
• Forgetting to record what you did. Write down the share imputed and the method per column.

Key takeaways

• Ask why a value is missing before deciding how to fill it.
• MCAR can be dropped or simply filled; MAR benefits from using other columns; MNAR needs a missing indicator and domain thinking.
• Mean and median fills shrink spread and weaken correlations, so use them as a baseline, not a default.
• KNNImputer and IterativeImputer use related columns to fill gaps more accurately.
• Put imputation inside a Pipeline and fit on training data only.

Missing values are one kind of problem; extreme values are another. The next lesson covers outlier detection. To test yourself on these rules, check your data cleaning knowledge on Debugshala Test.