Cross-Validation Explained: K-Fold, Stratified and Time Series

Picture the selectors for a Ranji Trophy side choosing between two young batters from Madhya Pradesh. One scored 140 in a single practice match on a flat pitch at Holkar Stadium. The other scored 55, 62, 38, 71 and 49 across five matches on five different pitches.

One innings tells you about one pitch on one day. Five innings on five pitches tell you how the batter plays in general, and how much the scores swing.

A single train/test split is the one innings. Cross-validation is the five-match record.

What is cross-validation in machine learning?

Map the cricket story to model building:

• Each match is one fold, a slice of your data the model has not trained on.
• The runs scored in each match are the fold scores.
• The batting average is the cross-validated score, and the spread of runs is its standard deviation.

Cross-validation is a resampling method that trains and evaluates a model several times on different train/validation splits of the same data and averages the results, giving a more reliable estimate of performance on unseen data than one split.

It sits inside the workflow from model evaluation metrics: you still keep a final test set aside, and cross-validation runs on the training portion to compare models and settings. It is also the main tool for spotting the problem described in overfitting and the bias-variance trade-off.

This tutorial uses scikit-learn 1.7, the version that runs in the browser on this site.

Why is one train/test split not enough?

A random split decides which rows land in the test set. With small or medium datasets, that luck of the draw moves the score a lot. Run the same model on the same 200 rows with ten different random splits:

import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression

# 200 loan applicants, 6 numeric features, label 1 = repaid on time
X, y = make_classification(n_samples=200, n_features=6, n_informative=3,
                           flip_y=0.1, random_state=42)

scores = []
for seed in range(10):
    X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=seed)
    model = LogisticRegression().fit(X_tr, y_tr)
    scores.append(model.score(X_te, y_te))

print("accuracy on 10 different random splits:")
print(np.round(scores, 3))
print(f"lowest {min(scores):.3f}, highest {max(scores):.3f}")

Nothing about the model changed, yet accuracy ranges from 0.80 to 0.95. If you had compared two models using one split each, a 5-point difference could be pure luck. That is the problem cross-validation solves.

How does k-fold cross-validation work?

K-fold is the standard method. Follow these steps:

1. Shuffle the rows (unless the data has a time order).
2. Cut them into k equal parts called folds. Five or ten is typical.
3. Train on k−1 folds and score on the one left out.
4. Repeat until every fold has been the validation fold exactly once.
5. Report the mean of the k scores and their standard deviation.

In 5-fold cross-validation every row is used for validation exactly once and for training four times. The final score is the average of the five fold scores.

import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import KFold, cross_val_score
from sklearn.linear_model import LogisticRegression

X, y = make_classification(n_samples=200, n_features=6, n_informative=3,
                           flip_y=0.1, random_state=42)

kf = KFold(n_splits=5, shuffle=True, random_state=0)
for i, (train_idx, val_idx) in enumerate(kf.split(X), start=1):
    print(f"fold {i}: train on {len(train_idx)} rows, validate on {len(val_idx)} rows")

scores = cross_val_score(LogisticRegression(), X, y, cv=kf, scoring="accuracy")
print("fold scores:", np.round(scores, 3))
print(f"cross-validated accuracy = {scores.mean():.3f} +/- {scores.std():.3f}")

The result, about 0.88 ± 0.04, is a far more honest summary than any single number from the previous block. The ± part tells you how much to trust a small gap between two models.

Types of cross-validation and when to use each

Plain k-fold assumes rows are independent and shuffled freely. Real data often breaks that assumption, and scikit-learn has a splitter for each case.

Stratified k-fold for classification

When one class is rare, random folds can end up with very few or none of it. A fold with zero fraud cases cannot measure fraud recall at all.

import numpy as np
from sklearn.model_selection import KFold, StratifiedKFold

# 100 insurance claims, only 10 are fraud (label 1)
y = np.array([1] * 10 + [0] * 90)
X = np.zeros((100, 1))                      # features don't matter for splitting

for name, cv in [("KFold", KFold(5, shuffle=True, random_state=3)),
                 ("StratifiedKFold", StratifiedKFold(5, shuffle=True, random_state=3))]:
    counts = [int(y[val].sum()) for _, val in cv.split(X, y)]
    print(f"{name:>16}: fraud cases in each validation fold = {counts}")

Plain KFold gives one fold no fraud at all. StratifiedKFold puts exactly 2 in each. When you pass an integer such as cv=5 with a classifier, scikit-learn already uses stratified folds, but without shuffling. Passing your own splitter makes the behaviour explicit. Rare classes get a full treatment in handling imbalanced data.

Time series split for dated data

If you shuffle a year of sweet-shop sales, the model trains on December and is tested on March. In real life you never know the future when you predict, so this score is too optimistic.

TimeSeriesSplit always trains on earlier months and validates on the months that come right after, so the model never sees the future.

import numpy as np
from sklearn.model_selection import TimeSeriesSplit

# 12 months of sales for a sweet shop, Jan to Dec
months = np.array(["Jan", "Feb", "Mar", "Apr", "May", "Jun",
                   "Jul", "Aug", "Sep", "Oct", "Nov", "Dec"])
tscv = TimeSeriesSplit(n_splits=4, test_size=2)

for i, (tr, te) in enumerate(tscv.split(months), start=1):
    print(f"split {i}: train {months[tr][0]}-{months[tr][-1]} ({len(tr)} months)"
          f" -> test {', '.join(months[te])}")

The training window grows each round, and validation always comes later. For more on dated data, see time series analysis basics.

Group k-fold when rows are not independent

Suppose a hospital in Indore gives you 400 scans from 40 patients, 10 scans each. Scans from the same patient look alike. If one patient's scans land in both train and validation, the model can recognise the patient instead of learning the condition.

import numpy as np
from sklearn.model_selection import KFold, GroupKFold, cross_val_score
from sklearn.neighbors import KNeighborsClassifier

rng = np.random.default_rng(0)
# 40 patients, 10 scans each. Each patient has a personal "signature",
# and the label (condition yes/no) is per patient, unrelated to the features.
patients = np.repeat(np.arange(40), 10)
signature = rng.normal(0, 3, (40, 4))
X = signature[patients] + rng.normal(0, 0.3, (400, 4))
y = rng.integers(0, 2, 40)[patients]

model = KNeighborsClassifier(n_neighbors=3)
plain = cross_val_score(model, X, y, cv=KFold(5, shuffle=True, random_state=0))
grouped = cross_val_score(model, X, y, cv=GroupKFold(5), groups=patients)
print(f"KFold (same patient in train and test): {plain.mean():.3f}")
print(f"GroupKFold (patients kept apart):       {grouped.mean():.3f}")

The labels here are random, so no honest model can beat about 50%. Plain KFold reports 99% because the model simply matches each scan to its patient's other scans. GroupKFold reports about 56%, the truth.

How do you avoid data leakage during cross-validation?

Leakage happens when information from the validation fold reaches training. The usual culprit is preprocessing on the full dataset first: fitting a scaler, an imputer or a feature selector on all rows, then cross-validating. The validation rows have already influenced the training step.

The fix is a Pipeline. When you pass a pipeline to cross_validate, scikit-learn re-fits every step on each fold's training rows only.

import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import StratifiedKFold, cross_validate
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

X, y = make_classification(n_samples=300, n_features=8, n_informative=4,
                           random_state=1)
X[:, 0] *= 1000                                 # one feature in rupees, others small

pipe = make_pipeline(StandardScaler(), SVC())   # scaler is re-fitted inside every fold
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
res = cross_validate(pipe, X, y, cv=cv, scoring=["accuracy", "f1"],
                     return_train_score=True)

for key in ["train_accuracy", "test_accuracy", "test_f1"]:
    print(f"{key:>15}: {res[key].mean():.3f}")

cross_validate returns several metrics at once, plus training scores. A training accuracy of 0.96 against a validation accuracy of 0.92 is a small, healthy gap. A gap of 20 points would point to overfitting. Scaling itself is covered in data preprocessing: scaling and encoding.

What happens after cross-validation?

Cross-validation scores a recipe (model type plus settings), not a final model. Once you have picked the best recipe:

1. Re-train it on the whole training set.
2. Score it once on the held-out test set.
3. Ship that model.

The five fold models are thrown away. When you also use cross-validation to pick settings, you are doing hyperparameter tuning, the next topic in this series, and GridSearchCV automates exactly these steps.

Common cross-validation mistakes

• Shuffling time-ordered data. Use TimeSeriesSplit, or your score will look better than real forecasting performance.
• Ignoring groups. Rows from the same customer, patient or device must stay in one fold.
• Preprocessing before splitting. Fit scalers, imputers and selectors inside a pipeline.
• Reporting only the mean. Always show the standard deviation. 0.88 ± 0.01 and 0.88 ± 0.09 are very different results.
• Tuning on the test set after cross-validation. The test set is for one final check.

Choosing the right splitter is easier when someone reviews your setup on real project data, which is part of the practice in Debugshala's machine learning classes in Old Palasia. You can also solve cross-validation MCQs on Debugshala Test.

Key takeaways

• One random split gives a noisy score; cross-validation averages several splits for a steadier estimate.
• K-fold uses each row for validation once and for training k−1 times. Five or ten folds is the usual choice.
• Match the splitter to the data: stratified for classes, time series for dates, group for repeated entities.
• Report mean and standard deviation, and treat differences within one standard deviation as ties.
• Wrap preprocessing in a Pipeline so nothing leaks from the validation fold.
• After choosing a recipe, re-train on all training data and test once.