Compute and accelerators

Compute is the work hardware does to train and run models. Accelerators (GPUs, TPUs, Trainium, Instinct MI300-class devices, etc.) speed the linear algebra AI relies on. Google’s TPU docs contrast CPUs (flexible, memory-bottlenecked), GPUs (thousands of ALUs, still general-purpose), and TPUs (matrix-specialized systolic arrays).

Hardware shapes what is affordable, who can train frontier models, how much inference costs, and—via data centers—electricity demand (IEA *Energy and AI*, Apr 2025; LBNL US data-center reports).

CPUs are flexible generalists. GPUs shine at many parallel math ops. Custom ASICs (TPUs, Trainium) trade generality for matrix throughput. Training burns large one-time compute; inference at scale often dominates product bills. Memory bandwidth, HBM capacity, and networking matter as much as peak FLOPs marketing. MLPerf Training/Inference give industry-comparable timing-to-quality metrics—not a single “best chip” crown.

A kitchen blender vs a hand whisk—same job family, different throughput and energy.

For one AI feature you use, ask: is the bill driven by training, by chat tokens, by embedding search, or by humans reviewing outputs?

⚠️ Myth: Parameter count alone predicts your cloud bill.

✓ Reality: Architecture (dense vs MoE), context length, and traffic matter.

⚠️ Myth: Energy stories are simple single numbers.

✓ Reality: Boundaries and methods differ—see IEA/LBNL chapters; do not invent kWh.

• Google Cloud — TPU architecture: https://docs.cloud.google.com/tpu/docs/system-architecture-tpu-vm ↗
• NVIDIA H100 product page: https://www.nvidia.com/en-us/data-center/h100/ ↗
• AWS Trainium getting started: https://aws.amazon.com/ai/machine-learning/trainium/getting-started/ ↗
• AMD Instinct MI300 microarchitecture: https://instinct.docs.amd.com/develop/gpu-arch/mi300.html ↗
• MLPerf Training: https://mlcommons.org/benchmarks/training/ ↗
• IEA — Energy and AI: https://www.iea.org/reports/energy-and-ai/ ↗