GPUs, TPUs, and training compute
Why modern AI needs special chips—and what “compute” means in practice.
What you'll learn
- Explain why matrix-heavy neural training favors GPUs over CPUs.
- Define compute roughly as chip time spent on training or inference.
- Compare inference vs training hardware needs at a high level.
In plain English
Training big neural networks is mostly huge batches of matrix math. Graphics Processing Units (GPUs) were built for parallel pixel work and happen to excel at those operations.
Tensor Processing Units (TPUs) are Google's custom chips optimized for tensor workloads common in deep learning. Cloud providers also offer various AI accelerators.
When researchers say compute, they often mean how much processor time and energy a training run consumed—not abstract intelligence.
How hardware maps to training
CPUs handle flexible sequential logic well; GPUs run thousands of small cores in parallel for dense linear algebra. Frameworks (CUDA, ROCm, XLA) compile ops to run on device memory bandwidth limits.
Training memory must hold model weights, activations, and optimizer states—large models need many gigabytes per device or sharding across chips.
Inference can use smaller dtypes (FP16, INT8), distillation, or specialized serving hardware to cut latency and cost versus full training rigs.
- GPU: flexible parallel training for most labs and startups.
- TPU: tightly integrated stacks in Google Cloud for large tensor jobs.
- Memory bandwidth often bottlenecks before raw FLOPs.
Going deeper
Mixed-precision training uses lower numerical precision where safe to double effective throughput on supported hardware.
Energy and cost drive model choices: a baseline that trains on one GPU beats a giant model nobody can afford to finetune.
Hardware roadmaps (HBM memory, chiplet designs, optical interconnects) shape which model sizes are economical each year.
Common misconceptions
- More FLOPs automatically means a smarter model.
- Compute enables scale, but data quality, algorithms, and evaluation determine usefulness.
- You need a TPU to do any deep learning.
- Consumer and datacenter GPUs train most models worldwide; TPUs are one cloud option among many.
- Training and inference have identical hardware needs.
- Training demands high memory and sustained throughput; inference optimizes latency, batching, and cost per query.
Key facts
- Deep learning training relies heavily on parallel matrix operations.
- GPUs accelerate neural network training and inference via massive parallelism.
- TPUs are specialized accelerators optimized for tensor computations.
- Device memory capacity limits model size per chip without sharding.
- Compute budget influences which models teams can train and maintain.
Sources used
These free resources informed this page. ANN writes original explainers; we do not copy course text behind paywalls.
Also explore AI companies, Live Feed, and Weekly Brief.