Model selection and nested CV

Model selection chooses among algorithms/hyperparameters using validation. Nested cross-validation reduces the optimistic bias from tuning and evaluating on the same splits.

Leaderboard-chasing on one validation set invents fake winners. Nested CV is the careful pattern when data is small.

Outer loop: estimate generalization. Inner loop: tune. Final model: refit with chosen settings on appropriate data. Keep a true holdout if you can afford it.

Picking a phone after testing cameras—don’t use the same one photo to both choose and advertise.

Describe your last tuning session: what was train/val/test—and did test stay untouched?

⚠️ Myth: CV always matches production.

✓ Reality: Time/group structure can invalidate random CV.

⚠️ Myth: Nested CV is mandatory for every big-data job.

✓ Reality: It’s most critical when n is small and tuning is heavy.

• Course 02 splits; Course 03 overfitting
• scikit-learn model selection guides: https://scikit-learn.org/ ↗