Measuring progress without hype

How to read AI progress claims: benchmarks, demos, product metrics, and scientific papers—without buying calendar-date prophecies.

Hype cycles waste money and erode trust. Measurement literacy is civic skill.

Prefer: clear task, baseline, data regime, failure cases, cost/latency, and who evaluated. Distrust: vibes, single viral clips, and “human-level” without definition.

A model that aces a quiz but fails your customer emails is not “done.”

Take one headline. Rewrite it as: task / metric / baseline / caveat.

⚠️ Myth: Leaderboard #1 means best for you.

✓ Reality: Domain mismatch and contamination happen.

⚠️ Myth: If it’s impressive on video, it’s reliable.

✓ Reality: Demos hide selection and supervision.

• Course 18 evaluation; Course 23 reading papers
• HELM / academic eval projects (cite specifically when teaching)
• NIST AI RMF: https://www.nist.gov/itl/ai-risk-management-framework ↗