Random Forest Explained: Bagging, OOB Score and Importance

Your college placement cell wants to guess which final-year students will clear campus interviews. One senior coordinator could judge from experience, but every coordinator has blind spots.

So the cell asks 100 seniors instead. Each has seen a different, random batch of past students. And to stop them all leaning on CGPA, each senior is handed only a few columns of a student's profile at a time.

Then everyone votes. Individually the seniors are a bit erratic, but their errors point in different directions, and the majority is right far more often than any one of them.

What is a random forest?

Map the placement cell to machine learning:

• Each senior is a decision tree.
• The random batch each senior saw is a bootstrap sample of the training rows.
• Seeing only a few profile columns at a time is random feature selection at each split.
• The final count of votes is the ensemble prediction.

A random forest is a supervised ensemble learning algorithm that trains many decision trees on bootstrap samples of the data, considers a random subset of features at every split, and combines the trees by majority vote for classification or averaging for regression.

If decision trees are new to you, read that tutorial first: a forest is only as good as the trees it averages. This article uses scikit-learn 1.7's RandomForestClassifier and RandomForestRegressor.

How does the random forest algorithm work?

Training a forest of T trees takes four steps:

1. Draw a bootstrap sample: pick n rows from the n training rows with replacement, so some rows appear twice and others not at all.
2. Grow a decision tree on that sample. At every split, consider only a random subset of the features (by default the square root of the feature count for classification) and choose the best split among them.
3. Grow the tree deep, usually without pruning.
4. Repeat steps 1 to 3 for every tree, then predict by majority vote or by averaging the trees' outputs.

Each tree sees different rows and different candidate features, so each makes different mistakes. The vote keeps what they agree on.

Step 1 on its own is called bagging (bootstrap aggregating). Step 2's feature sampling is what turns bagged trees into a random forest. Ensemble learning compares bagging with boosting and stacking in general; this article stays with forests.

What is bootstrap sampling and the out-of-bag score?

A bootstrap sample draws n rows from n, with replacement. How many distinct rows does it contain?

import numpy as np

rng = np.random.default_rng(0)
print("   rows | in bootstrap | left out (OOB) | formula (1 - 1/n)^n")
for n in [10, 100, 1000, 10000]:
    left_out = []
    for _ in range(200):
        sample = rng.integers(0, n, n)              # draw n row numbers WITH replacement
        left_out.append(1 - len(np.unique(sample)) / n)
    oob = np.mean(left_out)
    print(f"{n:7d} | {1 - oob:12.3f} | {oob:14.3f} | {(1 - 1 / n) ** n:.3f}")

Roughly 63% of the rows land in each tree's sample, and about 37% are left out. The formula (1 − 1/n)ⁿ approaches 1/e ≈ 0.368 as n grows, so this holds for any dataset of a reasonable size.

Those left-out rows are called out-of-bag (OOB) rows for that tree. Every training row is out-of-bag for about a third of the trees, so you can predict each row using only the trees that never saw it. The accuracy of those predictions is the OOB score, a validation estimate you get without holding out any data.

Row 3 was left out by trees 1 and 3, so only those two trees vote on it when computing the OOB score.

Why is a random forest better than a single decision tree?

A single deep tree has low bias but high variance: change a few training rows and the splits near the top change, taking the whole tree with them. Averaging many such trees keeps the low bias and cuts the variance.

How much it cuts depends on how correlated the trees are. If every tree made the same mistakes, averaging would fix nothing. That is why the random forest also samples features: without it, every tree would pick the same strong feature for its first split and look alike.

import numpy as np
from sklearn.datasets import make_classification
from sklearn.ensemble import BaggingClassifier, RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier

X, y = make_classification(n_samples=900, n_features=16, n_informative=6,
                           flip_y=0.05, random_state=3)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.4, random_state=0)

models = {
    "one deep tree": DecisionTreeClassifier(random_state=0),
    "bagged trees (all features)": BaggingClassifier(DecisionTreeClassifier(), n_estimators=100, random_state=0),
    "random forest (sqrt features)": RandomForestClassifier(n_estimators=100, random_state=0),
}
for name, m in models.items():
    m.fit(Xtr, ytr)
    print(f"{name:30s} test accuracy {m.score(Xte, yte):.3f}")

for name in ["bagged trees (all features)", "random forest (sqrt features)"]:
    votes = np.array([t.predict(Xte) for t in models[name].estimators_])
    corr = np.corrcoef(votes)[np.triu_indices(len(votes), k=1)].mean()
    print(f"{name:30s} average correlation between trees {corr:.2f}")

One deep tree scores about 0.77 on the test rows. A hundred bagged trees jump to about 0.86, and on this dataset the full random forest matches them.

The last two lines show the mechanism: the forest's trees agree with each other less (an average correlation of about 0.35 against 0.42), which leaves more room for the vote to cancel errors. On datasets with one dominant feature, that gap in correlation turns into a gap in accuracy.

How many trees does a random forest need?

More trees never make a forest overfit; they only stabilise its average. The score climbs quickly, then flattens, and extra trees just cost compute. The OOB score lets you see that point without touching the test set.

from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=900, n_features=16, n_informative=6,
                           flip_y=0.05, random_state=3)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.4, random_state=0)

forest = RandomForestClassifier(oob_score=True, warm_start=True, random_state=0)
print("trees | OOB accuracy | test accuracy")
for n in [25, 50, 100, 150, 200]:
    forest.set_params(n_estimators=n)
    forest.fit(Xtr, ytr)                       # warm_start adds trees, keeps the old ones
    print(f"{n:5d} | {forest.oob_score_:12.3f} | {forest.score(Xte, yte):.3f}")

warm_start=True keeps the grown trees and adds new ones on each fit. The OOB accuracy rises from about 0.80 with 25 trees to about 0.84 with 200 and is settling by then.

The OOB estimate is a little lower than the test score here; it's a slightly pessimistic estimate, since each row is judged by only about a third of the forest.

Random forest hyperparameters

Forests work well with defaults. When you tune, max_features and min_samples_leaf matter most; hyperparameter tuning shows how to search them.

Random forest feature importance

A forest reports feature_importances_, which adds up how much each feature reduced impurity across all splits (mean decrease in impurity, MDI). It is free to compute, but it has a known bias: features with many unique values get extra chances to split and look more important than they are.

import numpy as np
import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.inspection import permutation_importance
from sklearn.model_selection import train_test_split

rng = np.random.default_rng(5)
n = 800
df = pd.DataFrame({
    "cgpa": rng.normal(7.2, 1.0, n).round(2),
    "aptitude": rng.integers(20, 100, n),
    "mock_interviews": rng.integers(0, 6, n),
    "roll_no": rng.permutation(n),            # an ID column: pure noise, many unique values
})
score = 0.9 * (df.cgpa - 7.2) + 0.04 * (df.aptitude - 60) + 0.5 * df.mock_interviews
placed = (score + rng.normal(0, 1, n) > 1.2).astype(int)

Xtr, Xte, ytr, yte = train_test_split(df, placed, test_size=0.3, random_state=0)
rf = RandomForestClassifier(n_estimators=100, random_state=0).fit(Xtr, ytr)
perm = permutation_importance(rf, Xte, yte, n_repeats=5, random_state=0)
print(pd.DataFrame({"impurity (MDI)": rf.feature_importances_,
                    "permutation (test)": perm.importances_mean},
                   index=df.columns).round(3))

The roll_no column is pure noise, yet impurity importance gives it about 0.16, almost as much as mock interviews. Permutation importance shuffles one column at a time on test data and measures the drop in score.

It scores roll_no at about 0.01, close to nothing, which is the honest answer. Use permutation importance before you act on a ranking; feature selection covers it further.

Random forest regression and its limits

RandomForestRegressor averages the trees' numeric predictions. Every tree predicts the average of training values in a leaf, so a forest can never predict outside the range of targets it has seen.

import numpy as np
from sklearn.ensemble import RandomForestRegressor
from sklearn.linear_model import LinearRegression

# Monthly sales grew steadily for 24 months. Can each model predict months 30 and 36?
month = np.arange(1, 25).reshape(-1, 1)
rng = np.random.default_rng(1)
sales = 50 + 4 * month.ravel() + rng.normal(0, 3, 24)     # in thousand rupees

rf = RandomForestRegressor(n_estimators=100, random_state=0).fit(month, sales)
lr = LinearRegression().fit(month, sales)
for m in [12, 24, 30, 36]:
    print(f"month {m}: forest {rf.predict([[m]])[0]:6.1f} | linear {lr.predict([[m]])[0]:6.1f}")
print("highest sales seen in training:", round(sales.max(), 1))

At months 30 and 36 the forest predicts about 147 both times, stuck near the highest value it saw, while the linear model continues the trend. For data with a growing trend, such as sales over time, remove the trend first or use a model that extrapolates; time series analysis covers this.

Common random forest mistakes

• Trusting feature_importances_ blindly. High-cardinality columns such as IDs look important. Check with permutation importance.
• Tuning n_estimators like a regulariser. More trees don't overfit. Pick enough for a stable OOB score and move on.
• Leaving ID or date columns in. Trees will happily split on them.
• Forecasting trends with a forest. It flattens beyond the training range.
• Using one tree's rules to explain the forest. No single tree represents the vote.

Forests are often the first strong model on a tabular project, and Debugshala's Indore machine learning classes use them on business datasets with trainer review. Next comes gradient boosting and XGBoost, which builds trees one after another instead of side by side. To revise, try random forest quizzes on Debugshala Test.

Key takeaways

• A random forest averages many deep trees, each grown on a bootstrap sample with random features at each split.
• Bootstrap sampling plus feature sampling make the trees disagree, so averaging cuts variance.
• About 37% of rows are out-of-bag for each tree, which yields a free OOB validation score.
• More trees don't overfit; stop when the OOB score flattens.
• Prefer permutation importance to impurity importance, and don't expect a forest to extrapolate.