Mixture of experts and modern variants

In plain English

Scaling language models means more parameters and more compute. Mixture-of-experts (MoE) is a trick: instead of one giant feed-forward block firing for every token, a router picks a small set of “expert” subnetworks to handle each token.

You get a model with many total parameters but only part of them active per token—like a hospital where not every specialist joins every appointment.

How it works

In an MoE layer, a gating network scores experts and selects top-k (often one or two). Only those experts run for that token, which saves compute versus a dense layer of the same total size. Training must balance load so experts all get used—not just a favorite few.

Other modern tweaks appear across families: grouped-query attention (fewer key/value heads), rotary position embeddings (RoPE), SwiGLU activations, and better normalization. Together they improve stability, length generalization, or inference speed.

• MoE: router + many expert FFNs + sparse activation.
• GQA: share KV heads to cut memory during inference.
• RoPE: encode position by rotating query/key vectors.

Going deeper

MoE models can be fast per token but tricky to serve—routing patterns affect batching, and large expert counts stress memory bandwidth on GPUs.

Architecture labels in product marketing (“128 experts”) describe capacity and routing—not a guarantee of quality on your specific task without testing.

Common misconceptions

MoE means the model “calls different AIs” with separate personalities.

Experts are sublayers inside one model; routing is a math optimization, not separate agents.

Total parameter count equals cost per token.

Active parameters and memory traffic drive inference cost; MoE decouples total vs active size.

Every modern LLM uses MoE.

Many popular dense models still exist; MoE is one scaling path among several.

Key facts

• MoE routes each token through a subset of expert feed-forward networks.
• Sparse activation aims for large capacity with lower compute per token.
• Load-balancing losses help prevent router collapse onto few experts.
• Grouped-query and rotary embeddings are common efficiency and length tweaks.
• Architecture choice interacts with hardware, serving stack, and fine-tuning recipe.