The advice gets given in two sentences and then everybody moves on. Take thirty real examples. Work out what the correct answer is for each one. Run the system against them and count how often it gets them right.
It is the hour that returns most in an AI project and the thing that most reliably separates a system a business trusts from one it hopes about. A vendor who can tell you their system is right on 94% of a hundred real cases from your business, and can show you the six it missed, is having a completely different conversation from one who says it works well.
This piece is about how to actually do it: how to pick the cases, what to do when there is no single right answer, what accuracy is good enough, and why the test has to keep running after launch.
One scoping note. Everything below tests a step. Testing a whole sequence of steps is harder, because a chain can pass every step-level check and still produce a wrong finished job, so those cases have to run end to end and be graded on what came out the far side.
Picking the cases
The instinct is to grab the first thirty items in the folder. That produces a test set full of easy examples, because most work is easy, and the result is a reassuring score that tells you nothing about the cases that will cause problems.
A useful set is built deliberately, in roughly these proportions.
Half ordinary. The routine cases that make up the bulk of the volume. These establish the baseline and catch gross failures.
A quarter awkward. The ones that are genuinely harder: the document that is a scan of a fax, the customer with two accounts, the invoice with a credit note attached, the enquiry that asks three questions at once. Go and find these deliberately, because they will not turn up by chance in a sample of thirty.
A quarter that should be refused. Cases where the correct behaviour is to stop and hand it to a person. A document that is not what it claims to be, a value that is missing, a case outside the system’s remit. This quarter is the one almost everyone omits, and it tests the most important behaviour a business system has, which is knowing when to decline.
Two rules about how you choose them. Pick the cases before anyone sees what the system produces, because a set assembled after looking at the output is a set that quietly avoids the failures. And ask the person who does the work now to nominate the hard ones, because they know exactly which cases cause trouble and nobody has ever asked them.
On size: thirty to fifty is enough to be useful for most small-business workflows and honest about its limits. Thirty cases cannot distinguish 92% accuracy from 95%, because the difference between them at that size is under a single case. It can distinguish 92% from 60%, which is the distinction that actually decides whether to proceed.
That limit matters again later. A set of thirty will not reliably show a fall from 94% to 87% either, since that is a two-case difference and sits inside the noise. For catching gradual decline you need either a larger set, upward of a hundred, or the live indicators covered in the piece on drift, with the fixed set catching gross regressions instead of gentle ones. If a category matters enormously, refunds, say, or anything touching a regulated record, weight the set toward it rather than growing the whole thing.
Settling the right answers
This is the part that takes the time, and it is where most of the value turns up before the system is even run.
Someone who knows the work goes through each case and records what the correct output is, specifically instead of roughly: this field is 4,200, this is a category B, this one should have been refused because the date is missing.
Two things happen when a business does this properly. The first is that they end up with an answer key, which is the point. The second is that they discover their own process is less defined than they thought. Two experienced people will disagree about a handful of the thirty, and the disagreement is not a problem with the test, it is a problem with the specification that has been invisible until now. Resolving those disagreements improves the eventual system more than any amount of engineering, because a system built to an ambiguous rule produces ambiguous output and nobody can say it is wrong.
Write the resolutions down. That document is the specification, and it is worth more than the test set.
When there is no single right answer
Extraction has an answer key. A drafted reply does not, and pretending otherwise is why a lot of evaluation gets skipped for exactly the tasks that most need it.
The move is to grade against a rubric instead. For each output, three questions:
What must it contain? The customer’s actual question addressed. The correct price. The condition that applies. Anything absent from this list is a failure regardless of how well the thing reads.
What must it never contain? A commitment nobody authorised. A price that was not checked. Information about a different customer. Speculation stated as fact. These are the ones that cost money.
What would make it unusable? Wrong tone for the relationship, wrong length, wrong language, so long that the recipient will not read it.
Then two people grade the same ten outputs independently and compare. Where they agree, the rubric is working. Where they disagree, the rubric is vague, and that vagueness is currently being resolved differently by every person and every run. Tightening it is the most valuable edit available.
Worth separating two things that get confused here. This grades the output. Judging whether the call underneath it was sound is a different exercise, and it is why two experienced graders can disagree about an output that happened to turn out fine.
This is slower than counting correct answers and it is the only honest way to evaluate generative work. A system that produces fluent text is very easy to feel good about and very hard to assess without a rubric, which is precisely why the assessment gets skipped.
What score is good enough
There is no benchmark answer, and anyone quoting one is quoting a number from a context that is not yours.
The threshold comes from arithmetic about consequences. Take the error rate, multiply by what an error costs, compare against what the work saves. A system sorting enquiries into three buckets at 90% accuracy is clearly worth having, because a misfiled enquiry costs somebody a minute and the sorting saves hours. A system extracting payment amounts at 90% accuracy is not worth having on its own, because one wrong payment in ten is a disaster, and the same system paired with a checkpoint on anything it is unsure about might be excellent.
That relationship is the useful one. Accuracy and where the human checkpoint sits are two halves of one design, and evaluating either alone gives you the wrong answer. The question is never “is 94% good” but “is 94% good given that a person confirms the flagged ones and an error here costs eleven dollars.”
Two other things worth measuring alongside the headline rate, because they change decisions.
Which way it fails. Thirty errors that are all cautious refusals is a very different system from thirty errors that are confident wrong answers, even though the score is identical. The first is usable today, the second is dangerous. Count them separately.
Whether the confidence signal is honest. If the system reports high confidence on cases it got wrong, the flagging mechanism that the whole checkpoint design depends on is broken, and the score is hiding it. Models trained on human feedback tend toward overconfidence in the confidence they state, so this is worth measuring instead of assuming.
Re-running it
A test set run once at launch describes a system that no longer exists.
Four things change underneath a working system. Prompts get edited. The model version moves, sometimes without notice. The documents behind a retrieval system accumulate, and what the system finds depends entirely on what is in there. And upstream systems change their formats, which is the most common and least announced of the four.
So the same set gets re-run on a schedule, and after every change. Quarterly is a reasonable default for a stable system; after any change, without exception. The cost is small once the set exists, which is the argument for building it properly the first time.
The pattern worth watching is the movement rather than the absolute score, within the limits of the sample size. A thirty-case set will catch a system that has fallen off a cliff. A hundred-case set will catch a meaningful slide. Neither replaces watching the live traffic, which is a different instrument covered separately.
What it is not
Two clarifications, because these get conflated and the conflation wastes effort.
A test set is not the return-on-investment baseline. The baseline measures how long the work takes now and what it costs, which is what decides whether the project is worth doing. The test set measures whether the system is right, which is a different question, and a system can be highly accurate on work that was never worth automating.
And a test set is not monitoring. The test set is a fixed set of known cases you re-run deliberately. Monitoring watches the live traffic for signs that something has changed. Both are needed, they catch different things, and a business that has one usually believes it has both.
The part worth carrying out the door
The hour that decides whether an AI system is trustworthy happens before the system exists, and it consists of assembling thirty real cases and writing down what the right answer is.
Pick them deliberately, weighted toward the hard ones and the ones that should be refused. Settle the answers with someone who knows the work, and write down the disagreements, because those are specification defects you have been carrying invisibly. Grade generative work against a rubric instead of an answer key, and have two people grade the same ten to find out where the rubric is vague. Set the accuracy threshold from what an error costs instead of from a benchmark. Count the confident wrong answers separately from the cautious refusals. And re-run the whole thing after every change and quarterly regardless.
The question to put to anyone building this for you is simple, and the answer is diagnostic. How was it tested, on how many real cases from this business, what did it get wrong, and when was that last run. A build with a specific answer has done the work. A build that says it performs well has an opinion.