Features, loss, and optimization

Three ideas power most learning systems:

1. Features — numbers that describe an input (or learned representations).
2. Loss — a score for “how wrong” a prediction is.
3. Optimization — the process of adjusting the model to make loss smaller.

When a model fails, the bug is often here: bad features, the wrong loss for the business goal, or optimization that never really learned.

Imagine grading a quiz. Loss is the red marks. Optimization is studying to get fewer red marks next time. Features are what the student pays attention to—handwriting length vs actual math.

Deep learning often learns features automatically. Classical ML often uses human-designed features.

Spam model features might include link count and keyword flags. Loss rises when spam is called ham. Optimization nudges weights so that mistake happens less on the training mail.

For “predict rain tomorrow,” list five possible features. Mark which need sensors vs which are calendar/season quirks. Notice how feature choice embeds assumptions.

⚠️ Myth: Lower training loss always means a better product.

✓ Reality: Wrong loss (optimizing the wrong thing) or overfitting can make low loss misleading.

⚠️ Myth: Optimization “understands” the problem.

✓ Reality: It follows the math of the loss on the data it sees.

• Google ML Crash Course (descending into loss/gradient ideas): https://developers.google.com/machine-learning/crash-course ↗
• ANN live: https://www.ainerdnetwork.com/learn/features-loss-and-optimization ↗