Decision Trees Explained: Gini, Entropy and scikit-learn Lab

Think about the dispatcher at a food delivery hub near Palasia Square. Before every order leaves, she makes a quick call: "Will this one be late?" She does not run a formula. She asks herself questions.

"Is it more than 5 km away? Yes. Is it peak dinner time? Yes. Then it will probably be late, so I'll warn the customer." For a short order she asks something else: "Is it raining?" Each answer sends her down a different path until she reaches a verdict.

Over months she has learned which questions to ask first. Distance tells her the most, so she asks it first. Rain only matters for some orders, so it comes later. Her mental flowchart is a decision tree.

What a Decision Tree Is

Map the dispatcher to the terms:

• Each order is a sample; distance, rain and peak hour are features; late or on time is the label.
• Each question ("distance ≤ 5.5 km?") is a split at an internal node.
• The first question is the root. The final verdicts are leaves.
• Choosing which question to ask first is what the training algorithm does.

A decision tree is a model that predicts a label by routing each sample through a sequence of feature-based yes/no questions, where each question is chosen to make the groups it creates as pure as possible.

Trees work for classification (late or not) and regression (how many minutes late). This lab focuses on classification.

Splitting: How a Tree Chooses Its Questions

Suppose you have 12 past deliveries: 7 late and 5 on time. You could split on distance at many thresholds, or on rain, or on peak hour. Which is best?

The tree's rule is simple: prefer the split whose child groups are the least mixed. A group with 4 late and 0 on-time orders is perfectly pure. A group with 3 and 3 is as mixed as it gets. To compare splits you need a number for "mixed", and there are two popular ones.

Gini impurity

Gini is the chance that two random picks from the group (with replacement) have different labels. For class proportions p1, p2, ...:

Gini = 1 - (p1² + p2² + ...)

A 50/50 group gives 1 - (0.25 + 0.25) = 0.5, the worst case for two classes. A pure group gives 1 - 1 = 0.

Entropy

Entropy comes from information theory and measures surprise:

Entropy = -(p1·log2 p1 + p2·log2 p2 + ...)

A 50/50 group has entropy 1.0 bit; a pure group has 0. Entropy is a little more sensitive to small minority classes, but in practice Gini and entropy usually choose very similar splits.

Compute them by hand

from math import log2

def gini(labels):
    n = len(labels)
    return 1 - sum((labels.count(c) / n) ** 2 for c in set(labels))

def entropy(labels):
    n = len(labels)
    return abs(-sum((labels.count(c) / n) * log2(labels.count(c) / n) for c in set(labels)))

half = ["late"] * 6 + ["on_time"] * 6
pure = ["on_time"] * 5
skew = ["late"] * 1 + ["on_time"] * 4
for name, labels in [("6 vs 6 node", half), ("pure node", pure), ("1 vs 4 node", skew)]:
    print(f"{name:12s} gini={gini(labels):.3f}  entropy={entropy(labels):.3f}")

Notice the ordering is the same for both measures: the 50/50 node is worst, the 1 vs 4 node is in between, and the pure node is perfect.

Finding the Best Split in Plain Python

To score a split, compute the impurity of each child and take a weighted average by child size. A big impure child should count more than a tiny impure one. The drop from the parent's impurity to this weighted value is the information gain.

The script below tries every distance threshold plus the two yes/no features and reports the weighted Gini of each:

# 12 deliveries: distance in km, raining (1/0), peak hour (1/0), late?
rows = [(2,0,0,0),(3,0,1,0),(4,1,0,0),(5,0,0,0),(6,0,1,1),(7,1,1,1),
        (8,0,0,0),(9,1,0,1),(10,0,1,1),(11,1,1,1),(3,1,1,1),(12,0,0,1)]

def gini(labels):
    n = len(labels)
    return 1 - sum((labels.count(c) / n) ** 2 for c in set(labels)) if n else 0

def split_score(feature, threshold):
    left  = [r[3] for r in rows if r[feature] <= threshold]
    right = [r[3] for r in rows if r[feature] >  threshold]
    n = len(rows)
    return len(left)/n * gini(left) + len(right)/n * gini(right)

print(f"parent gini = {gini([r[3] for r in rows]):.3f}")
names = ["distance", "raining", "peak_hour"]
candidates = [(0, t) for t in range(3, 12)] + [(1, 0), (2, 0)]
for f, t in candidates:
    print(f"{names[f]:>9} <= {t:2d}: weighted gini {split_score(f, t):.3f}")
best = min(candidates, key=lambda c: split_score(*c))
print("best first question:", names[best[0]], "<=", best[1])

Distance ≤ 5 wins with a weighted Gini of about 0.276, down from 0.486 at the parent. Rain alone barely helps (0.419). The tree then repeats the same search inside each child, which is why the method is called recursive partitioning.

The Tree, Drawn Out

Here is the full tree that this data produces, with the number of deliveries reaching each leaf.

The decision tree asks about distance first because that split reduces Gini impurity the most, then about peak hour and rain inside each branch.

Read any path from top to bottom as an if-else rule. "Distance over 5.5 km and peak hour, so late" covers four deliveries. If you can write if-else statements, you can already read a tree.

Lab: DecisionTreeClassifier in scikit-learn

Now let scikit-learn do the search. export_text prints the learned rules so you can compare them with the diagram.

from sklearn.tree import DecisionTreeClassifier, export_text

# distance_km, raining, peak_hour  ->  late (1) or on time (0)
X = [[2,0,0],[3,0,1],[4,1,0],[5,0,0],[6,0,1],[7,1,1],
     [8,0,0],[9,1,0],[10,0,1],[11,1,1],[3,1,1],[12,0,0]]
y = [0, 0, 0, 0, 1, 1, 0, 1, 1, 1, 1, 1]
features = ["distance_km", "raining", "peak_hour"]

tree = DecisionTreeClassifier(max_depth=3, criterion="gini", random_state=0)
tree.fit(X, y)
print(export_text(tree, feature_names=features))
print("training accuracy:", tree.score(X, y))
print("predictions for new orders:", tree.predict([[4, 1, 1], [9, 0, 0], [2, 0, 1]]))
for name, imp in zip(features, tree.feature_importances_):
    print(f"importance of {name}: {imp:.2f}")

A few things to notice in the output:

• The root is distance_km <= 5.50. scikit-learn places thresholds halfway between observed values (5 and 6), so it matches your hand result of "≤ 5".
• Training accuracy is 1.0. With 12 rows and depth 3, the tree can fit every example, which is a warning sign, not a victory.
• Feature importance shows distance carrying about two-thirds of the total impurity reduction.

Try criterion="entropy" and compare the printed tree. On this data the structure barely changes, which is typical.

Controlling tree size

An unlimited tree keeps splitting until every leaf is pure, even if that means a leaf for one odd delivery. The main brakes are:

These trade a little training accuracy for much better behaviour on new data. The overfitting and bias-variance lab shows that trade-off with numbers.

Common Mistakes with Decision Trees

• Trusting 100% training accuracy. Always score on a held-out test set. See model evaluation metrics for how.
• Scaling features unnecessarily. Trees only compare a feature with a threshold, so standardising does not change the splits. It is harmless, just wasted effort.
• Feeding raw text categories. scikit-learn trees need numbers. Encode categories such as area names first, for example with one-hot encoding.
• Reading importance as cause. High importance means the tree used that feature a lot, not that it causes lateness.
• Ignoring instability. Removing a couple of rows can change the root split. Random forests average many trees to fix this.

Building trees, forests and boosting models on real business datasets is a core part of the ML course at Debugshala's Old Palasia campus, where you practise reading and pruning trees with a trainer.

Key takeaways

• A decision tree is a chain of yes/no feature questions ending in leaf predictions.
• Each split is chosen to minimise the weighted impurity of the child groups.
• Gini = 1 − Σp²; entropy = −Σp·log2 p. Both are 0 for pure nodes and maximal for evenly mixed ones.
• DecisionTreeClassifier plus export_text gives you a model you can read line by line.
• Control depth and leaf size, and always check accuracy on unseen data.