Regex and rules still matter
Deterministic **rules and regular expressions** catch crisp structure—IDs, formats, allowlists, required legal phrases—either instead of ML or as **gates** around ML/LLM systems. SLP3’s early chapters on words/tokens and classic classifi...
What it is
Deterministic rules and regular expressions catch crisp structure—IDs, formats, allowlists, required legal phrases—either instead of ML or as gates around ML/LLM systems. SLP3’s early chapters on words/tokens and classic classifiers sit beside modern LLMs for a reason: not every problem is fuzzy language.
<!-- IMAGE: rule gate before/after model -->
Visual Spec & Architecture Diagram
Decision tree: structured pattern (dates, IDs) → regex/rules; fuzzy semantics → ML. Hybrid box in middle. Example fake invoice line with regex capture groups highlighted.
Why it matters
Hybrid systems are adult engineering. Rules enforce invariants models approximate poorly (invoice shapes, jurisdiction codes, “must include risk disclosure”). They are testable like code.
How it works (plain)
- List invariants that must never be “model-guessed.”
- Implement rules/regex with unit tests.
- Use ML/LLM for fuzzy language around those rails.
- Log rule hits vs model decisions for audits.
- Retire rules that became unmaintainable—don’t let the pile rot.
CS224N focuses on neural methods; production NLP still needs this chapter’s discipline when shipping.
Everyday example
Reject any “account number” that fails a known checksum before an LLM drafts a support reply.
Try it
List three checks in your domain that should never be model-guessed. Write one regex or deterministic function for the easiest one, plus two failing tests.
Myths
- ⚠️ Myth: Deep learning made rules obsolete.
- ✓ Reality: Rules enforce invariants models approximate poorly.
- ⚠️ Myth: More regex always means more safety.
- ✓ Reality: Unmaintainable piles create blind spots.
- ⚠️ Myth: LLMs can replace validators.
- ✓ Reality: Validators should be boring and exact.
Sources
- SLP3 (words/tokens, classification foundations): https://web.stanford.edu/~jurafsky/slp3/ ↗
- Course 15 NLP overview; Course 10 tools; Course 06 logic
