Scaling laws and compute

Scaling laws are empirical patterns relating model size, data, and compute to performance on training objectives. They are guides—not destiny and not proof of understanding. Hardware reality (accelerator availability, energy, networking) constrains which scale-ups are practical.

Labs use them to budget training runs. Citizens should treat headlines like “10× bigger = 10× smarter” with skepticism. IEA’s energy–AI work highlights that larger deployment also raises electricity and grid questions—scaling is not only a loss-curve story.

Within a setup, more compute/data/parameters often improves loss predictably—until data quality, architecture, or eval mismatch intervenes. Optimal allocation among size/data/compute matters. Serving scale (inference traffic) can dominate lifetime compute even when training was the headline.

Studying longer helps—until you’re exhausted or using bad materials.

When you see a “scaling” claim, ask: which metric, which data, which compute budget, compared to what baseline—and is energy counted?

⚠️ Myth: Scaling laws guarantee AGI on a calendar date.

✓ Reality: They describe trends under assumptions; capability jumps and plateaus both happen.

⚠️ Myth: More GPUs always linear-speed training.

✓ Reality: Networking and utilization bend the curve (see distributed networking unit).

• Course 24 compute / accelerators units
• IEA — Energy and AI: https://www.iea.org/reports/energy-and-ai/ ↗
• MLPerf Training (empirical system scaling): https://mlcommons.org/benchmarks/training/ ↗