Embeddings

In plain English

An embedding turns a word, sentence, or document into a list of numbers—a vector. The idea is that items with similar meaning should land in similar directions in that number space, so a computer can compare “dog” and “puppy” as closer than “dog” and “spreadsheet.”

Embeddings power semantic search, recommendation, clustering, and retrieval-augmented generation (RAG). They are one bridge between human language and linear algebra.

How it works

Inside a language model, each token ID is looked up in an embedding table—a matrix whose rows are learned vectors. As training progresses, tokens that appear in similar contexts get pushed toward similar representations.

For whole passages, systems often pool token vectors (mean, last token, or a dedicated “CLS” style token) or use a separate embedding model trained to match queries with relevant documents.

import math

def cosine(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    na = math.sqrt(sum(x * x for x in a))
    nb = math.sqrt(sum(x * x for x in b))
    return dot / (na * nb)

# Pretend these came from an embedding model
dog = [0.9, 0.1, 0.0]
puppy = [0.85, 0.15, 0.05]
spreadsheet = [0.0, 0.2, 0.95]

print("dog vs puppy:", round(cosine(dog, puppy), 3))
print("dog vs spreadsheet:", round(cosine(dog, spreadsheet), 3))

Going deeper

Embedding quality depends on training data and objective. A model trained on general web text may embed legal or medical phrases poorly until fine-tuned or replaced with a domain-specific embedder.

Similarity is not truth: two false statements can embed closely if they paraphrase each other. RAG still needs good source documents and answer checking.

Common misconceptions

Embeddings store English definitions inside the numbers.

They encode patterns useful for the training task. Interpretability is partial, not a dictionary lookup.

Higher dimension always means better search.

More dimensions can help, but data quality, chunking, and the embedding model matter as much as size.

LLM token embeddings and search embeddings are the same thing.

They may come from related training, but production RAG often uses dedicated embedding models and indexes.

Key facts

• Embeddings map discrete text into continuous vectors.
• Similar vectors (by cosine or dot product) suggest related meaning for retrieval.
• LLMs learn token embeddings as the first step of the forward pass.
• Document embeddings enable retrieve-then-generate pipelines (RAG).
• Domain mismatch between queries and indexed text hurts retrieval quality.