Batch norm vs layer norm (deeper)

A deeper comparison of batch normalization vs layer normalization (and friends): what statistics they use and when each shows up.

Vision CNNs historically loved batch norm; transformers often use layer/RMS norm. Mixing them blindly breaks recipes.

Batch norm: normalize across the batch for each channel/feature (train vs eval behavior differs). Layer norm: normalize across features for each example—friendlier for variable NLP batches and decoders.

Grading on a curve within a class (batch-ish) vs standardizing each student’s own answer vector (layer-ish)—imperfect metaphor, useful for memory.

When reading a paper diagram, label which norm sits where.

⚠️ Myth: Norm layers are optional cosmetics.

✓ Reality: They often decide whether deep stacks train at all.

• Course 05 residuals-and-normalization; Course 07 transformers
• Ioffe & Szegedy BatchNorm; Ba et al. LayerNorm (cite when teaching)