What is an evaluation harness?
Evaluation harness definition
An evaluation harness is a set of cases, each with an input and an expected outcome, that an AI system’s output is scored against every time the system changes: a new prompt, a new model version, a new tool, a new integration. The harness runs in the build pipeline, like a test suite, and a change that lowers the score does not ship. For a system whose output is text or a decision rather than a number, it is the only way to know whether a change made things better or worse.
It is the difference between a demo and a system. A demo works on the examples its builder tried. A system works on the cases the business meets, including the ones nobody thought of until a customer sent them. The harness is where those cases accumulate: every failure found in production becomes a case, and the system can never fail that way silently again.
What a good case looks like
A real input, taken from production or written by the person who does the job, not an invented one. An expected outcome that a reviewer can check without a judgement call: the correct classification, the fields that must be extracted, the action that must or must not be taken, the facts an answer must contain. A note of why the case matters, so the next person understands what it guards. A case that scores a free-text answer needs a rubric, and the rubric is written down with the case.
Who writes the cases
The client's team. The engineers build the harness, the scoring and the pipeline that runs it, but the people who know what a correct outcome looks like are the people who did the job before the system did. In a Traide engagement the first cases are written in the Diagnose sprint, from the real data it measures, and the set grows for as long as the system runs, under an AI ops retainer.