Ensemble Learning: Bagging, Boosting and Stacking in Python

Your college has entered a three-person team in the inter-college quiz at an Indore fest. Priya is strong on science, Arjun on sport and films, and Neha on history and current affairs. For each question, the three discuss and go with the majority, or with whoever sounds most sure.

The team beats each member on their own, because they rarely get the same question wrong. Three science toppers would share the same blind spots.

What is ensemble learning?

Map the quiz team to machine learning:

• Each team member is a base model (also called a base learner or weak learner).
• Their different strengths are diversity: models that err on different rows.
• Taking the majority answer is voting; listening more to whoever is most confident is soft voting.
• A captain who learns over many rounds whom to trust for which topic is a meta-model, as in stacking.

Ensemble learning is a machine learning technique that trains several models and combines their predictions, by voting, averaging or a learned meta-model, to get better accuracy and stability than any single model.

You have already met two ensembles in this series: the random forest and gradient boosting and XGBoost. This article covers the general idea and the third family, stacking, with scikit-learn 1.7.

Why does ensemble learning work?

Suppose five models are each right 70% of the time, and their mistakes are independent. The majority is wrong only when at least three of the five are wrong at once, which is much rarer than one model being wrong.

from math import comb
import numpy as np

def majority_right(n, p):
    """Chance that more than half of n independent models are right."""
    return sum(comb(n, k) * p**k * (1 - p)**(n - k) for k in range(n // 2 + 1, n + 1))

print("models | independent 70% models | simulated, errors shared 60% of the time")
rng = np.random.default_rng(0)
for n in [1, 3, 5, 11, 25, 101]:
    shared = rng.random(20000) < 0.7                     # one common "opinion" per question
    own = rng.random((20000, n)) < 0.7                   # each model's own opinion
    use_shared = rng.random((20000, n)) < 0.6            # how often a model copies the common one
    right = np.where(use_shared, shared[:, None], own)
    sim = (right.sum(axis=1) > n / 2).mean()
    print(f"{n:6d} | {majority_right(n, 0.7):22.3f} | {sim:.3f}")

The middle column is the ideal case. Five independent 70% models give a majority that is right about 84% of the time, and 25 such models reach about 98%.

The right column is closer to real life: each model copies a shared opinion 60% of the time. There the gain stops at about 0.73 with three models, and large ensembles slip back to 0.70, the accuracy of the shared opinion itself.

That is the core lesson of ensembles. Accuracy of the members matters, but diversity between them matters just as much. Every technique below is a way to create diversity: different rows, different features, different algorithms or different targets.

What are the types of ensemble methods?

There are three main families, plus simple voting:

Bagging works side by side, boosting works in a line, and stacking adds a second layer that learns how to combine.

Voting and averaging ensembles

The simplest ensemble takes models you have already built and combines their outputs. Hard voting counts predicted labels. Soft voting averages predicted probabilities, so a confident model counts for more than a hesitant one.

from sklearn.datasets import make_moons
from sklearn.ensemble import VotingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures, StandardScaler
from sklearn.tree import DecisionTreeClassifier

X, y = make_moons(n_samples=600, noise=0.35, random_state=11)
members = [
    ("curvy logreg", make_pipeline(PolynomialFeatures(3), StandardScaler(), LogisticRegression())),
    ("knn", make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=25))),
    ("tree", DecisionTreeClassifier(max_depth=5, random_state=0)),
]
for name, m in members:
    print(f"{name:12s} CV accuracy {cross_val_score(m, X, y, cv=5).mean():.3f}")
for vote in ["hard", "soft"]:
    ens = VotingClassifier(members, voting=vote)
    print(f"{vote + ' vote':12s} CV accuracy {cross_val_score(ens, X, y, cv=5).mean():.3f}")

The three models score between about 0.86 and 0.88 on their own. The soft vote reaches about 0.885, matching or slightly beating the best member. The practical point: you didn't need to know in advance which model would be best, and the vote is usually more stable across folds.

Soft voting needs predict_proba on every member. For regression, VotingRegressor averages predictions.

Bagging: which models benefit?

Bagging (bootstrap aggregating) trains many copies of one algorithm on bootstrap resamples of the rows and averages them. It cuts variance, so it only helps a model whose predictions swing when the training data changes slightly.

from sklearn.datasets import make_classification
from sklearn.ensemble import BaggingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.tree import DecisionTreeClassifier

X, y = make_classification(n_samples=600, n_features=12, n_informative=5,
                           flip_y=0.08, random_state=2)
for name, base in [("deep decision tree", DecisionTreeClassifier(random_state=0)),
                   ("logistic regression", LogisticRegression(max_iter=500))]:
    alone = cross_val_score(base, X, y, cv=5).mean()
    bagged = cross_val_score(BaggingClassifier(base, n_estimators=25, random_state=0),
                             X, y, cv=5).mean()
    print(f"{name:20s} alone {alone:.3f} | bagged x25 {bagged:.3f} | gain {bagged - alone:+.3f}")

A deep decision tree jumps by about 6 percentage points when bagged, from about 0.76 to about 0.82. Logistic regression gains nothing, because 25 versions trained on slightly different rows draw almost the same line.

High-variance models such as deep trees and KNN with small k are good bagging candidates; stable models are not. A random forest is bagging of trees plus random feature selection.

Boosting in one paragraph

Boosting trains models one after another, giving more attention to the rows the ensemble still gets wrong. AdaBoost does this by re-weighting rows; gradient boosting does it by fitting each new tree to the remaining errors.

Boosting turns many weak, shallow models into a strong one, which reduces bias. The gradient boosting tutorial covers it in depth, including learning rate and early stopping.

What is stacking in machine learning?

Stacking (stacked generalisation) trains several different base models, then trains a meta-model that takes their predictions as input features and learns how to combine them.

The critical detail is how the meta-model's training data is built. If you train the base models on all rows and then feed their predictions on those same rows to the meta-model, the base models look far better than they are, and the meta-model learns to trust overfitted outputs.

Stacking avoids this with out-of-fold predictions: each row's base prediction comes from a model that didn't see that row.

Every row gets a prediction from a model that never saw it. Those honest predictions are what the meta-model learns from.

scikit-learn's StackingClassifier does this for you with its cv argument, then refits each base model on all the training rows for use at prediction time.

from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier, StackingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

X, y = make_classification(n_samples=800, n_features=12, n_informative=5,
                           flip_y=0.08, random_state=3)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0)
base = [
    ("forest", RandomForestClassifier(n_estimators=60, random_state=0)),
    ("svm", make_pipeline(StandardScaler(), SVC(random_state=0))),
    ("knn", make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=11))),
]
for name, m in base:
    print(f"{name:8s} test accuracy {m.fit(Xtr, ytr).score(Xte, yte):.3f}")

stack = StackingClassifier(estimators=base, final_estimator=LogisticRegression(), cv=5)
stack.fit(Xtr, ytr)
print(f"stacked  test accuracy {stack.score(Xte, yte):.3f}")
print("meta-model weights (forest, svm, knn):", stack.final_estimator_.coef_.round(2))

The stacked model scores about 0.90, a little above the best base model at about 0.89. The meta-model's weights show how it combines them: it trusts the forest most and leans on KNN more than its standalone score suggests, presumably because KNN is right on rows where the others slip.

Gains from stacking are usually small like this, and with only 240 test rows one percentage point is three rows, so confirm a stack's benefit with cross-validation before you ship it.

When should you not use an ensemble?

Ensembles cost training time, memory and latency, and are harder to explain. Prefer a single model when:

1. A simpler model is already within noise of the ensemble's score.
2. You need clear, auditable rules, such as for a loan decision that must be explained to a customer.
3. Predictions must be returned in a few milliseconds on small hardware.
4. Your members are near-copies of each other, so the ensemble adds cost without adding diversity.

Common ensemble learning mistakes

• Combining near-identical models. Five similar boosted trees add little; mix algorithms or data views.
• Training a meta-model on in-sample predictions. Always use out-of-fold predictions, as StackingClassifier does.
• Bagging a stable model. Logistic regression and linear SVMs barely change; bag trees instead.
• Soft voting with uncalibrated members. One overconfident model can drown out the rest.
• Tuning the ensemble on the test set. Keep a final hold-out untouched until the very end, as shown in cross-validation.

Ensembles appear in almost every serious ML project, and machine learning lessons at Debugshala, Old Palasia build them on real datasets. If you would rather cover statistics, ML and AI in one track, see Debugshala's full Data Science & AI course.

The next section of this series moves to unsupervised learning, starting with k-means clustering. Before that, test your ensemble basics on Debugshala Test.

Key takeaways

• An ensemble combines models so that their errors cancel; it works only when members make different mistakes.
• Bagging averages parallel copies to cut variance; boosting chains models to cut bias; stacking learns how to combine different models.
• Soft voting averages probabilities and usually beats hard voting when members are calibrated.
• Stacking must train its meta-model on out-of-fold predictions.
• Use a single model when the ensemble's gain is within noise or when you need explainability and speed.