Classical machine learning models

In plain English

Before deep learning dominated images and text, classical machine learning solved lots of business problems—and still does on spreadsheets, logs, and sensor tables.

Linear and logistic regression fit weighted sums (with optional nonlinear features). Decision trees split data on simple rules. Random forests and gradient boosting combine many trees for stronger predictions.

These models are often fast to train, easier to interpret, and need less data than huge neural nets.

How common models behave

Linear regression predicts numbers; logistic regression predicts class probabilities via a sigmoid. Regularization (L1/L2) keeps weights small for stability.

Decision trees greedily choose splits that reduce impurity (Gini, entropy). Single trees overfit; ensembles average many trees trained on random subsets (bagging) or sequential error correction (boosting).

k-nearest neighbors classifies by majority vote among closest training points—simple baseline with no training phase beyond storing data.

from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X = [[0, 1], [1, 0], [3, 3], [4, 4]]
y = [0, 0, 1, 1]

log_reg = make_pipeline(StandardScaler(), LogisticRegression())
forest = RandomForestClassifier(n_estimators=50, random_state=0)

log_reg.fit(X, y)
forest.fit(X, y)
print("linear preds:", log_reg.predict([[2, 2]]))
print("forest preds:", forest.predict([[2, 2]]))

Going deeper

Gradient boosting libraries (XGBoost, LightGBM, CatBoost) dominate many Kaggle tabular competitions and enterprise risk models.

Interpretability tools (coefficients, SHAP for trees) help regulated industries explain decisions—an advantage over large opaque nets.

Classical models can feed deep systems: hand-crafted features into a net, or a tree model as a sanity-check baseline before deploying transformers.

Common misconceptions

Classical ML is obsolete after deep learning.

On structured tabular data with moderate size, boosted trees often outperform deep nets with less tuning.

Linear models can only learn straight lines.

Feature engineering (polynomials, interactions) lets linear models capture rich patterns while staying fast.

Random forests are always interpretable.

Single trees are readable; large forests are ensembles requiring summary tools like feature importance or SHAP.

Key facts

• Linear and logistic regression are fast, interpretable baselines for many tasks.
• Decision trees split feature space with axis-aligned rules.
• Random forests and gradient boosting combine many trees for robust predictions.
• Classical methods remain strong on tabular and small-to-medium datasets.
• Ensemble tree models are widely deployed in industry for structured data.