A 30-Minute Eval Setup That Catches Most Prompt Regressions

For months, my testing process for prompt changes was: tweak the prompt, try three examples by hand, feel good, ship. Then a user emailed to say summaries had started dropping the customer's actual question. I had broken it two deploys earlier and not noticed.

That was when I finally set up evals. Not a big platform. Just enough structure to know when a change makes things worse.

Step 1: Collect 40 real examples

Synthetic examples are tempting because they are clean. Real ones are better because they are messy in the ways your users are messy. I pulled 40 anonymised inputs from logs, deliberately including:

• Very short inputs
• Very long inputs
• Inputs in mixed languages
• Inputs where earlier versions had failed

Forty is not statistically perfect. It is enough to catch obvious regressions, which is the whole goal.

Step 2: Write checks, not scores

For each example, I wrote one to three simple checks that a good output must pass. Things like:

1. The summary mentions the order number if one exists.
2. The reply does not promise a refund unless the input mentions one.
3. The output is under 120 words.

Some checks are plain string or length rules. A few use a small model as a judge with a yes or no question. I avoid asking a judge for a 1 to 10 score, because I found those numbers drifted around without meaning much.

Step 3: A tiny runner

The script is short. In plain terms it does this:

for example in examples:
    output = run_prompt(example.input)
    for check in example.checks:
        record(example.id, check.name, check(output))
print(pass_rate_by_check())

It runs in about two minutes and costs a few cents.

Step 4: Compare before and after

The key habit is running the suite on both the current prompt and the new one, then looking at the difference. My output is a short summary in the terminal: which checks went from pass to fail, and which went from fail to pass.

When I tested a new, more concise prompt last month, it improved length compliance from 78 to 95 percent, and silently broke order number extraction on 6 examples. Without the comparison, I would have shipped it happily.

Step 5: Grow it from failures

Every time a user reports a bad output, it becomes a new example with a check. My set is now at 63 examples. That is the real value: the suite slowly becomes a memory of every mistake I do not want to repeat.

An eval suite is just your past bugs, written down so they cannot sneak back.

If you are shipping prompts with no tests at all, start with twenty examples tonight. It will feel too small. It will still catch something within the first week.

Cover: Photo by Jakub Żerdzicki on Unsplash.