Evals for People Who Ship Code
The habit that turns 'it looked good in my chat window' into engineering.
An eval is a test for model behavior. Not always a perfect test. Not always automated. But a test.
Without evals, you are changing prompts by vibes. That works until the demo becomes a product and nobody knows whether the latest "improvement" broke refunds, citations, or tool calls.
Start small
Collect 20 to 50 real examples. Use cases that represent your product, not generic benchmark prompts. Include boring successes and painful failures.
For each example, define what good means. Exact answer? Contains required fields? Refuses unsafe action? Calls the correct tool? Cites the right document?
Types of evals
Deterministic checks: JSON parses, required fields exist, answer includes a source, tool argument matches schema.
Human review: a person grades correctness, tone, usefulness, or risk.
LLM-as-judge: another model grades output with a rubric. Useful, but needs spot checks because now your test also has opinions.
The useful loop
Run evals before and after prompt changes, model changes, retrieval changes, and tool changes. Track failures. Read traces. Fix the highest-impact failure class first.
That is the line between "AI feature" and "AI system."