Features, loss, and optimization

In plain English

Models do math on numbers. Features turn raw inputs—pixels, words, spreadsheet columns—into those numbers. Good features make patterns easier to learn; deep networks can learn features automatically.

A loss function scores how wrong predictions are. Big loss means bad fit; small loss means predictions match labels (or targets) closely.

Optimization adjusts model parameters step by step to lower loss. Gradient descent follows the slope downhill, like finding the bottom of a valley in fog by feeling which way tilts down.

How training steps fit together

Forward pass: compute predictions from current parameters. Loss: compare predictions to truth. Backward pass: compute gradients (how loss changes if each parameter nudges). Update: move parameters a small step opposite the gradient.

Learning rate controls step size—too large oscillates or diverges; too small crawls. Optimizers (SGD, Adam) adapt steps for faster, stabler training.

Different tasks use different losses: cross-entropy for classification, mean squared error for regression, token cross-entropy for language modeling.

# Predict y from x with weight w, bias b
w, b = 0.5, 0.0
lr = 0.1

for step in range(20):
    x, y_true = 2.0, 3.0
    y_pred = w * x + b
    loss = (y_pred - y_true) ** 2  # mean squared error on one point

    # Gradients of loss w.r.t. w and b
    d_loss_dw = 2 * (y_pred - y_true) * x
    d_loss_db = 2 * (y_pred - y_true)

    w -= lr * d_loss_dw
    b -= lr * d_loss_db
    print(f"step {step}: loss={loss:.4f} w={w:.4f} b={b:.4f}")

Going deeper

Non-convex losses in deep nets have many local basins; good initialization, batch normalization, and architecture choices matter as much as the optimizer name.

Regularization (weight decay, dropout) penalizes complexity so optimization finds simpler solutions that generalize better.

Automatic differentiation frameworks compute gradients for millions of parameters—what makes large-scale deep learning practical.

Common misconceptions

Loss near zero on training data is always the goal.

Perfect training fit can overfit. Validation loss guides when to stop and which model to ship.

You must hand-engineer all features for deep learning.

Deep nets learn hierarchical features, but tabular problems still benefit from thoughtful feature design and cleaning.

Key facts

• Features are numeric representations fed into models.
• Loss functions quantify prediction error during training.
• Gradient-based optimizers iteratively reduce loss by updating parameters.
• Learning rate and optimizer choice strongly affect training stability.
• Task-appropriate loss functions align training with deployment goals.