Language models

In plain English

A language model is a program trained on huge amounts of text to guess what word or piece of word comes next. If you show it “The sky is ___,” it learns that “blue” is a strong guess—not because it looked outside, but because that pattern appeared often in its training data.

That sounds simple, but next-token prediction is flexible. The same machinery can finish a sentence, answer a question, translate, summarize, or write code—depending on how you phrase the input and what extra training the model received.

Today's popular chatbots are large language models (LLMs): the same core idea, scaled up with transformers, lots of data, and follow-up training so replies feel helpful and on-topic.

How it works

Training shows the model many sequences of tokens (pieces of text). At each step the model outputs scores for every possible next token. The training goal is to raise the score of the token that actually appeared and lower the rest. After enough examples, the model internalizes grammar, facts, styles, and common reasoning patterns—whatever was statistically useful for prediction.

At use time you give a prompt (your text so far). The model predicts one token, appends it, predicts the next, and repeats until it hits a stop condition. There is no separate “understanding module”—the behavior emerges from prediction.

• Input: a sequence of tokens (your prompt plus anything already generated).
• Output: a probability distribution over the vocabulary for the next token.
• Generation: sample or pick the top token, append, repeat.

Going deeper

Language modeling predates transformers—older RNNs and n-gram models did similar jobs with smaller context and weaker long-range links. Transformers made it practical to train much bigger models on much more text, which is why LLMs took off in the last few years.

Useful capabilities are side effects of compression: to predict well, the model must represent syntax, some world knowledge, and task-like patterns seen in training (Q&A formats, code, dialogue). That is powerful but not the same as guaranteed correctness or intentional reasoning on every turn.

Common misconceptions

Language models look up answers in a database.

They compute next-token probabilities from learned weights. They can recall training-like text, but they are not querying a curated fact store unless you add retrieval (RAG) or tools.

If it sounds fluent, it must be right.

Fluency comes from language statistics. A model can produce smooth, confident text that is still wrong—especially on rare facts or math.

Bigger always means smarter in every way.

Scale helps many tasks, but size does not remove hallucinations, bias, or the need for evaluation on your actual use case.

Key facts

• A language model assigns scores to possible next tokens given everything so far.
• Chat is implemented by conditioning on a prompt (instructions plus conversation text).
• Training is self-supervised: the “labels” are the next tokens in the training text itself.
• The model has no direct access to the live web unless a product wires in search or tools.
• Capabilities depend on architecture, data, training recipe, and post-training—not prediction alone.