Backpropagation

In plain English

After a forward pass produces a wrong answer, the network needs to know which weights to nudge and in which direction. Backpropagation computes those sensitivities efficiently.

It applies the chain rule from calculus layer by layer, starting at the loss and moving backward. Each weight learns how much it contributed to the error.

Modern frameworks hide the calculus, but the idea—local gradients multiplied along paths—is what makes deep learning trainable at scale.

How backward passes work

Compute loss L comparing prediction to target. Derive ∂L/∂output at the last layer. Propagate ∂L/∂activations backward through each layer using derivatives of activations and weight matrices.

Weight gradient is activation from the previous layer times upstream gradient—intuitively, how much that weight influenced the mistake.

Batching averages gradients over many examples for stable updates. Optimizers then apply those gradients with learning rates and momentum.

# y = f(g(x)); dL/dx = dL/dy * dy/dx
x = 2.0
g = 3 * x + 1      # g(x)
f = g ** 2         # f(g) = loss-like scalar

dL_df = 1.0
df_dg = 2 * g
dg_dx = 3.0

dL_dx = dL_df * df_dg * dg_dx
print("gradient w.r.t. x:", dL_dx)  # backprop multiplies local grads like this

Going deeper

Vanishing gradients slowed very deep sigmoid nets; ReLU, residuals, and better init helped. Exploding gradients need clipping in RNN training.

Automatic differentiation builds a computation graph during forward pass, then traverses it backward—PyTorch and JAX implement this pattern.

Second-order methods and meta-learning reuse backprop structure; most large models still rely on first-order SGD variants for scale.

Common misconceptions

Backpropagation is how the brain learns.

It is an efficient engineering algorithm for differentiable networks, not a claim about biology.

You must derive every gradient by hand to train nets.

Frameworks autograd handles derivatives; understanding concepts helps debug NaNs and shape errors.

Key facts

• Backpropagation computes gradients of the loss with respect to all weights.
• The chain rule links gradients layer by layer from output to input.
• Automatic differentiation in frameworks implements backprop on computation graphs.
• Gradient vanishing and exploding motivated modern activations and architectures.
• Optimizers consume backprop gradients to update parameters each step.