Transformers

In plain English

A transformer is a neural network design built around attention: each token can look at other tokens in the input and decide what to focus on. That makes it easier to handle long sentences, pronouns, and dependencies far apart in text.

Most LLMs you hear about—GPT-style decoders, many chat models, and plenty of multimodal systems—are transformer stacks trained to predict tokens.

How it works

A typical decoder-only transformer (common in LLMs) repeats layers of self-attention and feed-forward networks. Self-attention produces a new representation for each token by mixing information from the whole context so far. Feed-forward layers apply the same small MLP to each position. Residual connections and layer normalization help training stay stable.

Causal (masked) attention ensures each position only looks at earlier tokens during training—so the model cannot cheat by peeking at the answer when learning next-token prediction.

• Token + position information enters the stack.
• Each layer: self-attention → add & norm → feed-forward → add & norm.
• Final layer outputs logits (scores) for the next token at each position.

Going deeper

The original “Attention Is All You Need” architecture also introduced encoder–decoder stacks for translation. Many LLMs use only the decoder stack because autoregressive language modeling fits that shape naturally.

Transformers are parallel during training (all positions processed together with masks), which is a major speed win over step-by-step RNNs—important when scaling to billions of parameters.

Common misconceptions

Transformers “understand” language like humans.

They learn statistical representations that work well for prediction and many downstream prompts—not a full cognitive model.

Attention is optional icing on the cake.

For modern LLMs, attention is the core mechanism that mixes context at every layer.

Key facts

• Transformers stack identical-ish blocks of attention and feed-forward layers.
• Decoder-only LLMs use causal masking for next-token training.
• Position information is added because attention alone is order-agnostic.
• Output logits at the last position drive the next generated token in chat use.
• The design enabled much larger models and datasets than RNN-era pipelines.