Ensembles and boosting (deeper)

Ensembles combine multiple models. Boosting builds models in sequence, each focusing more on previous mistakes (e.g., gradient boosting families).

Tabular ML in industry still often wins with gradient boosting. Knowing why prevents “deep learning everything” mistakes.

Bagging averages parallel diverse models (random forests). Boosting adds specialists for residuals. Diversity + aggregation reduce error—until you overfit or leak.

A panel of reviewers catches more issues than one reviewer—if they aren’t clones.

Compare a single decision stump story vs a sequence of stump corrections on a tiny example.

⚠️ Myth: Neural nets always beat boosting on spreadsheets.

✓ Reality: For many tabular tasks, boosting remains extremely strong.

⚠️ Myth: More trees always help.

✓ Reality: Diminishing returns + overfit—use validation.

• Course 03 classical-ml-models
• scikit-learn ensemble docs: https://scikit-learn.org/ ↗
• Friedman gradient boosting literature (cite when teaching)