Prompt versioning in production
Treating prompts like code: versions, reviews, rollbacks, and eval gates—alongside model versions. In LLM products the prompt is part of the “model system,” just as configuration debt is part of classic ML systems (Sculley et al.). <!-- ...
What it is
Treating prompts like code: versions, reviews, rollbacks, and eval gates—alongside model versions. In LLM products the prompt is part of the “model system,” just as configuration debt is part of classic ML systems (Sculley et al.).
<!-- IMAGE: prompt@v3 + model@v12 pinned in config -->
Visual Spec & Architecture Diagram
Prompt version board: prompt_id v1/v2/v3, owner, eval score, % traffic, rollback. Treat prompts like code artifacts.
Why it matters
Silent prompt edits cause silent regressions. A registry that tracks weights but not prompts cannot explain what users saw yesterday.
How it works (plain)
- Store prompts in git (or a config service with history).
- Require PR review for production prompts.
- Run an eval script / golden set before promote.
- Pin
prompt_id@versionnext tomodel@versionin deploy config. - Roll back the pair together when quality drops.
Everyday example
Changing a restaurant’s allergy disclaimer on the menu without telling the kitchen—or rewriting the recipe card without a version number.
Try it
Move one live prompt into a versioned file with a one-line changelog. Point staging at the new pin; keep prod on the old pin until eval passes.
Myths
- ⚠️ Myth: Prompts are too soft to version.
- ✓ Reality: Softness is why regressions hide—version them.
- ⚠️ Myth: The model vendor’s changelog covers your prompt.
- ✓ Reality: Your system prompt and tool instructions are your product.
Sources
- Course 17 registry / MLflow registry patterns: https://mlflow.org/docs/latest/ml/model-registry/ ↗
- Google Cloud MLOps (config + metadata as first-class): https://docs.cloud.google.com/architecture/mlops-continuous-delivery-and-automation-pipelines-in-machine-learning ↗
- Course 08 prompt eval; Course 21 eval script lab
- Hidden technical debt (config debt): https://research.google/pubs/hidden-technical-debt-in-machine-learning-systems/ ↗
