Tokenization

In plain English

Computers do not read English letters the way you do. Before a language model sees your message, a tokenizer chops text into tokens—small chunks that might be a whole word, part of a word, or punctuation.

Tokenization matters in real products: it affects how long your prompt can be, how much you pay per request, and whether the model “sees” a rare word as one piece or several awkward pieces.

How it works

Most modern LLMs use subword tokenizers. Byte Pair Encoding (BPE) and similar methods start from characters or bytes and merge the most frequent pairs until they build a vocabulary of thousands of tokens. Common words become single tokens; rare words split into readable pieces (e.g. “un” + “believ” + “able”).

Each token maps to an integer ID. The model never sees the string “hello”—it sees ID 15339 (example only). Decoding runs the process backward when text is generated.

# Start with characters, merge frequent pairs (BPE idea)
text = "low lower lowest"
tokens = list(text)  # character-level starting point

merges = [("l", "o"), ("lo", "w")]  # pretend these were learned from a corpus
for a, b in merges:
    merged = a + b
    tokens = [
        merged if tokens[i] == a and i + 1 < len(tokens) and tokens[i + 1] == b
        else tokens[i]
        for i in range(len(tokens))
    ]
    # collapse consecutive merged spans (simplified)
    out = []
    i = 0
    while i < len(tokens):
        if i + 1 < len(tokens) and tokens[i] == a and tokens[i + 1] == b:
            out.append(merged)
            i += 2
        else:
            out.append(tokens[i])
            i += 1
    tokens = out

print("Tokens:", tokens)
print("Token count:", len(tokens))

Going deeper

Different models use different tokenizers, so the same English sentence can have different token counts on different APIs. That is why “max tokens” limits are not identical to “max words.”

Tokenizer boundaries can change behavior: spacing, emoji, code, and non-English scripts may consume more tokens than you expect. For production apps, measure with the model's official tokenizer when possible.

Common misconceptions

One token equals one word.

Tokens are subword pieces. Short common words are often one token; long or rare strings may be many.

Tokenization is just a formatting detail.

It defines the model's alphabet. Bad splits can hurt math, code, and multilingual tasks.

Key facts

• Tokenization converts text into integer IDs from a fixed vocabulary.
• BPE-style methods merge frequent character pairs into subword tokens.
• The same visible text can tokenize differently across models.
• Context windows and billing are counted in tokens, not characters.
• Rare strings often split into multiple tokens, which can weaken model performance on them.