Feature Selection in Machine Learning: Filter, Wrapper, Embedded

Suppose you are packing one cabin bag for a four-day work trip from Indore to Bengaluru. You lay out everything you own on the bed: three laptops' worth of chargers, two identical black shirts, a winter jacket, a cricket bat, your documents.

You don't pack all of it. You drop the jacket because Bengaluru is mild, keep one of the two identical shirts, and leave the bat at home. The bag gets lighter and you find things faster.

Your dataset's columns are that pile on the bed, and feature selection is the packing.

What is feature selection in machine learning?

Map the packing to data:

• Everything on the bed is your full feature set.
• The jacket is an irrelevant feature: it has no link to the target.
• The two identical shirts are redundant features that carry the same information, like a bill in rupees per month and per year.
• The cabin bag size is the real cost of extra features: slower training, more overfitting and harder explanations.

Feature selection is the process of choosing a subset of the available input features that keeps or improves a model's predictive performance, by removing features that are irrelevant, redundant or too noisy to help.

It is different from creating features. Feature engineering builds new columns such as ratios or date parts; feature selection then decides which columns to keep. It is also different from PCA and dimensionality reduction, which combines features into new ones instead of keeping original columns.

The code here uses scikit-learn 1.7 and pandas 2.x.

Why does feature selection matter?

More columns are not automatically better. Each extra feature brings costs:

• Overfitting. With many features and few rows, a model finds patterns in noise. This is the high-variance problem from overfitting and the bias-variance trade-off.
• Speed. Training and prediction slow down, and production systems must collect every input.
• Explanations. A credit model with 8 clear inputs is easier to defend to a manager than one with 300.
• Data quality risk. Every column is another place for missing values or leaks to sneak in.

What are the three types of feature selection methods?

Filters are fast and model-agnostic. Wrappers are accurate but expensive. Embedded methods get selection as a by-product of fitting one model.

Filter methods: quick statistical checks

Start with basic hygiene before any statistics. Drop columns that never change, and drop one of any pair that are near copies:

import numpy as np
import pandas as pd
from sklearn.feature_selection import VarianceThreshold

rng = np.random.default_rng(1)
n = 300
df = pd.DataFrame({
    "monthly_bill": rng.normal(800, 200, n).round(),
    "data_gb": rng.gamma(4, 5, n).round(1),
    "country_code": np.full(n, 91),                     # every customer is in India
    "calls_per_day": rng.poisson(6, n),
})
df["yearly_bill"] = df["monthly_bill"] * 12 + rng.normal(0, 50, n)   # same info again

vt = VarianceThreshold(threshold=0.0).fit(df)
print("zero-variance columns:", list(df.columns[~vt.get_support()]))

corr = df.drop(columns="country_code").corr().abs()
upper = corr.where(np.triu(np.ones(corr.shape, dtype=bool), k=1))
to_drop = [c for c in upper.columns if (upper[c] > 0.9).any()]
print("highly correlated, drop:", to_drop)
print("kept:", [c for c in df.columns if c not in to_drop + ["country_code"]])

country_code is the same for every row, so it can't explain anything. yearly_bill is just monthly_bill times 12, so one of them is enough.

F-test vs mutual information

Next, score each feature against the target. The ANOVA F-test (f_classif) measures how far apart the class means are, so it only sees straight-line, mean-shift relationships. Mutual information (mutual_info_classif) measures how much knowing a feature reduces uncertainty about the target, so it also catches curved or two-sided patterns.

import numpy as np
import pandas as pd
from sklearn.feature_selection import SelectKBest, f_classif, mutual_info_classif

rng = np.random.default_rng(0)
n = 600
y = rng.integers(0, 2, n)                           # 1 = customer left
X = pd.DataFrame({
    "complaints": rng.poisson(1 + 2 * y),           # linear link: more complaints, more churn
    "tenure": rng.normal(30 - 10 * y, 8),           # linear link: newer customers churn
    "usage_change": rng.normal(0, 0.5, n) + y * rng.choice([-2, 2], n),  # big change either way
})
for i in range(4):
    X[f"noise_{i}"] = rng.normal(0, 1, n)           # no link at all

f_scores, _ = f_classif(X, y)
mi = mutual_info_classif(X, y, random_state=0)
print(pd.DataFrame({"F_test": f_scores.round(1), "mutual_info": mi.round(3)}, index=X.columns))
best = SelectKBest(mutual_info_classif, k=3).fit(X, y)
print("SelectKBest with mutual information keeps:", list(X.columns[best.get_support()]))

Customers who leave either sharply raise or sharply cut their usage. The averages cancel out, so the F-test scores usage_change at about 1, no better than noise.

Mutual information ranks it as the most informative feature of all (about 0.61). Filters judge one feature at a time, so neither test would notice a feature that only matters in combination with another.

Wrapper methods: let a model choose

Recursive feature elimination (RFE) fits a model, removes the weakest feature by coefficient size or importance, and repeats. RFECV adds cross-validation at every size so it can pick the best number of features for you.

import numpy as np
from sklearn.datasets import make_classification
from sklearn.feature_selection import RFECV
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold

X, y = make_classification(n_samples=400, n_features=12, n_informative=4,
                           n_redundant=0, shuffle=False, random_state=4)

rfe = RFECV(LogisticRegression(), step=1, cv=StratifiedKFold(5, shuffle=True, random_state=0),
            scoring="accuracy", min_features_to_select=1)
rfe.fit(X, y)
scores = rfe.cv_results_["mean_test_score"]
for k in [1, 2, 3, 4, 6, 8, 12]:
    print(f"{k:2d} features: CV accuracy {scores[k - 1]:.3f}")
print("best number of features:", rfe.n_features_)
print("kept columns:", np.flatnonzero(rfe.support_).tolist(), "(true informative: 0-3)")

Here shuffle=False places the four truly useful features in columns 0 to 3. RFECV keeps exactly those four, at about 0.945 accuracy, and the score slips slightly as noise columns come back. The cost is many model fits, so wrappers suit dozens of features better than thousands.

SequentialFeatureSelector is the other wrapper in scikit-learn. It adds (forward) or removes (backward) one feature at a time based on the CV score directly, which is slower but works with any model.

Embedded methods: selection during training

Some models select features as part of fitting:

• Lasso pushes weak coefficients exactly to zero, as shown in ridge, lasso and Elastic Net.
• Tree ensembles report how much each feature reduced impurity across all splits.
• SelectFromModel wraps either one and keeps features above an importance threshold.

Why permutation importance is safer

A tree's built-in feature_importances_ (impurity importance) is computed on training data, and it is biased towards features with many unique values, such as IDs or exact amounts. Permutation importance shuffles one column on held-out data and measures how much the score drops. A feature the model truly relies on causes a big drop; a useless one causes none.

import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.inspection import permutation_importance
from sklearn.model_selection import train_test_split

rng = np.random.default_rng(0)
n = 500
y = rng.integers(0, 2, n)
X = np.column_stack([
    y + rng.normal(0, 0.8, n),              # useful
    y + rng.normal(0, 1.5, n),              # weakly useful
    rng.integers(0, 2, n),                  # yes/no column, pure noise
    rng.permutation(n),                     # customer ID: unique, pure noise
])
names = ["useful", "weak", "noise_yes_no", "customer_id"]

X_tr, X_te, y_tr, y_te = train_test_split(X, y, random_state=0)
rf = RandomForestClassifier(n_estimators=100, random_state=0).fit(X_tr, y_tr)
perm = permutation_importance(rf, X_te, y_te, n_repeats=10, random_state=0)

print(f"{'feature':>12} | impurity | permutation (on test data)")
for name, imp, p in zip(names, rf.feature_importances_, perm.importances_mean):
    print(f"{name:>12} | {imp:8.3f} | {p:11.3f}")

Impurity importance gives the meaningless customer_id about 0.20, far more than the yes/no noise column, simply because it has 500 unique values to split on. Permutation importance on the test data puts it at roughly zero. The scikit-learn permutation importance guide explains the method in detail.

How do you avoid leakage in feature selection?

This is the most important section. If you pick features using all rows and then cross-validate, the validation rows have already helped choose the features. With enough columns, some noise columns will correlate with the target by chance, and your score will look excellent.

Left: the selector sees the validation rows, so chance correlations survive into the score. Right: each fold selects using its own training rows only.

import numpy as np
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline

rng = np.random.default_rng(42)
X = rng.normal(size=(100, 2000))        # 2,000 columns of pure noise
y = rng.integers(0, 2, 100)             # labels are coin flips

# WRONG: pick the 20 "best" columns using all rows, then cross-validate
X_sel = SelectKBest(f_classif, k=20).fit_transform(X, y)
wrong = cross_val_score(LogisticRegression(), X_sel, y, cv=5).mean()

# RIGHT: selection is a pipeline step, so it is redone inside every fold
pipe = make_pipeline(SelectKBest(f_classif, k=20), LogisticRegression())
right = cross_val_score(pipe, X, y, cv=5).mean()
print(f"select first, then CV : {wrong:.2f} accuracy on pure noise")
print(f"select inside the CV  : {right:.2f} accuracy (the truth is 0.50)")

Every column and every label is random, yet the wrong workflow reports 82% accuracy. Inside a pipeline, the honest answer of about 50% comes back. Genomics, text and survey data with thousands of columns are especially prone to this trap.

Common feature selection mistakes

• Selecting before splitting. Put the selector in a Pipeline so it re-fits on each fold.
• Trusting one filter score. The F-test misses non-linear links; check mutual information too.
• Reading impurity importance as truth. It favours high-cardinality columns. Use permutation importance on held-out data.
• Dropping a feature because lasso zeroed it. Its correlated twin may carry the same signal; the pair might matter together.
• Selecting too aggressively. If CV accuracy is flat from 6 to 12 features, keeping 6 is fine but keeping 2 may hurt.

Choosing features for a real business dataset, and defending that choice, is a skill that grows with guided project work such as Debugshala's project-driven ML training. To revise the ideas here, attempt feature selection questions on Debugshala Test.

Key takeaways

• Feature selection keeps the useful columns and removes constant, redundant and noisy ones.
• Filter methods are fast single-feature tests; wrapper methods search subsets with a model; embedded methods select while training.
• Mutual information catches non-linear links that the F-test misses.
• Permutation importance on held-out data is more trustworthy than impurity importance.
• Selection must happen inside cross-validation, or noise can look highly predictive.