Self-attention

In plain English

Self-attention lets every token ask a question: “Which other tokens should I pay attention to right now?” Tokens that matter get higher weight; irrelevant ones fade. The result is an updated representation that carries context from the whole sequence.

When the model sees “The trophy did not fit in the suitcase because it was too big,” attention helps link “it” to the right noun—something older sequence models struggled with at long distance.

How it works

For each token, the model builds three vectors: Query (what am I looking for?), Key (what do I offer?), and Value (what information do I pass if selected?). Attention scores compare queries to keys (often dot products, scaled, then softmax). Those scores weight the values— a weighted average that becomes the token's new context-aware vector.

Multi-head attention runs several attention patterns in parallel so different heads can track syntax, coreference, or local phrases.

import math

def softmax(xs):
    m = max(xs)
    exps = [math.exp(x - m) for x in xs]
    s = sum(exps)
    return [e / s for e in exps]

# Keys and queries for 3 tokens (simplified)
keys = [[1.0, 0.0], [0.0, 1.0], [1.0, 1.0]]
queries = [[1.0, 0.0], [0.0, 1.0], [1.0, 1.0]]
values = [[1.0, 0.0], [0.0, 1.0], [1.0, 1.0]]  # info each token would pass on

def attend(q):
    scores = [sum(q[k] * keys[i][k] for k in range(2)) for i in range(3)]
    weights = softmax(scores)
    out = [0.0, 0.0]
    for w, v in zip(weights, values):
        out[0] += w * v[0]
        out[1] += w * v[1]
    return weights, out

for i, q in enumerate(queries):
    w, mixed = attend(q)
    print(f"token {i} weights:", [round(x, 3) for x in w], "-> mixed", [round(x, 3) for x in mixed])

Going deeper

Scaled dot-product attention divides scores by √d to keep softmax from saturating when dimensions grow. Causal masks set scores to negative infinity for future positions so generation stays autoregressive.

Attention cost grows with context length—roughly O(n²) in naive form— which is why long contexts, sparse attention, and hardware optimizations are active engineering areas.

Common misconceptions

Attention weights are a perfect explanation of model reasoning.

Weights are useful visuals but can mislead; models distribute information across layers and heads.

More attention heads always help without tradeoffs.

Heads add parameters and compute; returns depend on model size and task.

Key facts

• Self-attention compares every token to every other token (with masks in decoders).
• Queries, keys, and values are learned linear projections of embeddings.
• Softmax turns scores into weights that sum to 1 for each query token.
• Output for each token is a weighted mix of value vectors.
• Multiple heads let the model attend in parallel along different patterns.