Distributed training basics

In plain English

When a model or batch is too large for one GPU, teams spread work across many devices and machines. Distributed training synchronizes gradients or splits layers so everyone stays aligned.

The goal is near-linear speedup: twice the chips should not mean twice the headaches—but networking and memory often eat gains.

Large language model training runs are cluster jobs orchestrated with fault tolerance, checkpointing, and careful batch scaling.

Main parallelism patterns

Data parallelism: each replica holds a full model copy, processes different micro-batches, then all-reduce gradients to update weights together. Simple and widely used.

Model parallelism: split layers or tensors across devices so no single chip holds the entire model. Tensor parallelism shards big matrix multiplies; pipeline parallelism assigns layer groups to stages.

Optimizer state and activations also consume memory—techniques like ZeRO sharding partition optimizer tensors across ranks to fit bigger models.

# Two replicas compute gradients on different mini-batches
grad_a = {"w": 0.4}
grad_b = {"w": 0.8}

# All-reduce: average so both replicas apply the same update
avg = {k: (grad_a[k] + grad_b[k]) / 2 for k in grad_a}
lr = 0.1
w = 1.0
w -= lr * avg["w"]
print("updated weight:", w)

Going deeper

Frameworks (PyTorch FSDP, DeepSpeed, Megatron-LM) implement sharding strategies; cloud schedulers place jobs on racks with fast interconnects (NVLink, InfiniBand).

Global batch size affects convergence—scaling learning rates with batch size is standard but not automatic. Reproducibility across cluster sizes is hard.

Checkpointing to durable storage recovers from node failures; training pauses are expensive at thousand-GPU scale.

Common misconceptions

More GPUs always means proportionally faster training.

Communication, I/O, and stragglers limit speedup; small models may not benefit from many devices.

Distributed training is only for tech giants.

Mid-size teams use multi-GPU data parallel finetuning daily; extreme model parallelism is what requires giant clusters.

Key facts

• Data parallelism replicates the model and splits batches across devices.
• Model parallelism partitions layers or tensors when models exceed device memory.
• Gradient all-reduce synchronizes updates in data-parallel training.
• Network bandwidth and topology strongly affect multi-node efficiency.
• Checkpointing and fault tolerance are essential for long cluster jobs.