RAG (retrieval-augmented generation)

In plain English

Retrieval-augmented generation (RAG) means: search your documents for passages related to the user's question, paste the best matches into the prompt, then ask the language model to answer using that material.

The model weights stay the same—you are giving it a open-book exam instead of relying on memory from training.

How it works

Offline: split sources into chunks, embed each chunk, store vectors in an index. Online: embed the query, retrieve top matches, optionally rerank, inject chunks into the prompt with citations, generate an answer conditioned on that context.

Good RAG systems tune chunk size, metadata filters, and refusal behavior when nothing relevant is found.

# Offline index (pretend we already embedded chunks)
index = [
    {"id": "doc1", "text": "Our refund policy allows returns within 30 days."},
    {"id": "doc2", "text": "Shipping is free on orders over $50."},
]

def retrieve(query, k=1):
    # Real systems use vector search; here: naive keyword overlap
    q = set(query.lower().split())
    scored = []
    for row in index:
        words = set(row["text"].lower().split())
        scored.append((len(q & words), row))
    scored.sort(reverse=True)
    return [row for _, row in scored[:k]]

query = "Can I return my order after two weeks?"
hits = retrieve(query)

context = "\n".join(f"[{h['id']}] {h['text']}" for h in hits)
prompt = f"""Use only the context below.

Context:
{context}

Question: {query}
Answer:"""

print(prompt)
# Next: send prompt to LLM; model generates answer grounded in context

Going deeper

Failure modes include retrieving wrong chunks, partial context that omits caveats, and the model ignoring provided text. Hybrid search (keywords + vectors) and citation-required formats mitigate some issues.

RAG complements but does not replace evaluation—you still measure answer correctness against sources.

Common misconceptions

RAG fine-tunes the model on your docs automatically.

Classic RAG injects text at inference time; updating knowledge means updating the index.

If it retrieved text, the answer must be faithful.

Models can misread or blend chunks; always check citations against sources.

Bigger chunks are always better for context.

Oversized chunks dilute signal; tiny chunks lose sentences needed for meaning.

Key facts

• RAG separates knowledge storage (index) from generation (LLM).
• Embeddings enable semantic retrieval beyond exact keyword match.
• Retrieved passages consume tokens in the context window.
• Freshness comes from updating documents and re-embedding.
• Grounding reduces but does not eliminate hallucinations.