Context windows and memory
How much text a model can “see” at once—and why that is not the same as human memory.
What you'll learn
- Define context window in tokens and why it is a hard limit.
- Contrast in-context conversation with long-term agent memory.
- List practical strategies when prompts exceed the window.
In plain English
The context window is how many tokens the model can consider in one forward pass—your instructions, chat history, retrieved documents, and its own draft answer all compete for that space.
It feels like memory when a chat remembers earlier messages, but that is usually because the product resends recent text inside the window—not because the model permanently stored your story.
How it works
During generation, each new token attends to everything currently in context (subject to architecture limits). When text falls off the left edge because the window filled up, the model literally stops seeing it—unless the app summarizes, truncates cleverly, or retrieves it again.
Longer windows help with big code files and document Q&A, but they cost more compute and do not automatically fix forgetting: distant text can still get weak attention (“lost in the middle” effects show up in practice).
- Everything in one request must fit: system prompt + tools + user + assistant turns + RAG chunks.
- Tokenizers count the same whether the text is instructions or filler.
- Products may compress history with summaries or vector retrieval.
Going deeper
Research explores sliding windows, recurrence, and external memory, but mainstream APIs still expose a max context length you must design around.
Agent systems combine short context with external stores (databases, files, RAG)—treating the LLM as a reasoning front-end, not the only memory.
Common misconceptions
- A huge context window means the model remembers everything equally well.
- Attention spreads over more tokens; important details can still be under-weighted if buried.
- Chat history in an app equals model long-term memory.
- Persistent memory is a product feature—stored externally and injected when needed.
- Summaries never lose information.
- Summaries trade fidelity for space; critical facts may need structured storage or citations.
Key facts
- Context limits are measured in tokens and apply per request or session turn.
- Older chat turns drop out when they no longer fit unless archived elsewhere.
- RAG and tool results consume the same window as user messages.
- Long-context models reduce but do not remove the need for retrieval design.
- Human-like “memory” in products is engineered outside the raw transformer.
Sources used
These free resources informed this page. ANN writes original explainers; we do not copy course text behind paywalls.
- Hugging Face LLM Course — Intro to language models and modern NLP stacks.
- The Illustrated Transformer — Visual walkthrough of attention and transformer blocks.
- Dive into Deep Learning — Textbook-style coverage of deep learning and NLP.
Also explore AI companies, Live Feed, and Weekly Brief.