Tests, With a Probabilistic Twist

Every serious system I have shipped traveled with a second system behind it: the tests. The suite was my contract with my own future self – green meant sleep, red meant not yet, and keeping it honest was half the craft. Then the systems started answering in prose. Same input, different output, on purpose. The first time a good test suite met a language model, the suite lost – not because testing stopped mattering, but because every assertion in it rested on a promise the model refuses to make: that there is exactly one right answer.

Stage 05 of the Route is blunt about the stakes: most teams running agents have observability, only about half have evals, and that gap is where production quality dies. Chapter 12 installs evaluation into the delivery machine – the fourth layer of the test pyramid, the new SLO vocabulary, the runbook. But it assumes you already know what an eval is made of. This piece is that vocabulary, and the good news is that you have carried most of it for years under a different name.

When assertEquals Stopped Working

A classical test suite rests on three quiet assumptions, so old they feel like physics. Determinism: the same input produces the same output, every run, forever. Enumerable cases: the inputs worth testing can be listed, because the interface constrains them. And a binary verdict: the assertion either holds or it does not; there is no such thing as forty percent equal.

A language model politely declines all three. Same input, different output – variation is not a bug, it is sampling working as designed. The input space is human language, which does not enumerate. And the output is not right or wrong but graded: an answer can be accurate, safe, on-format, and concise – or accurate but rambling, or fluent but subtly wrong, which is the failure mode Chapter 12 names confidently wrong. You cannot assert on prose. You can only grade it. And the moment grading replaces asserting, three new parts appear.

Three Parts, Every Eval

Cases, grader, bar. Everything else in the eval ecosystem – the platforms, the dashboards, the tracing hooks – is packaging around these three parts.

The Judge Is Also on Trial

The scorer that makes evals scale is the strange one: a model grading a model. It works for a reason worth internalising – verifying is easier than generating, the same asymmetry that makes code review faster than writing code. A judge with a written rubric can grade thousands of outputs overnight for less than the cost of one human afternoon.

But the judge is also on trial, and its biases are documented, not hypothetical. Judges lean toward answers that resemble their own style. They mistake length for quality often enough that verbosity bias has a name. In side-by-side comparisons, the order of presentation moves the verdict, so careful setups grade both orders and average. None of this disqualifies the method; it puts obligations on it. Calibrate the judge against human labels before you trust it. Keep its rubric written, versioned, and reviewed – it is configuration, not conversation. And spot-check it forever. A judge you never audited is just a second opinion you decided to like.

Offline Gates, Online Truth

Evals run on two clocks, and mature teams run both.

Years ago I ran an A/B test instead of an argument: a CEO with a strong instinct, me with a different view, and a split of real traffic to settle what the meeting could not. Online evaluation is that move, institutionalised. You stop asking the room whether the new prompt is better. You ask the traffic, with a measurement plan you wrote before you saw the results – because evidence gathered after the argument has a way of agreeing with whoever gathered it.

Grade the Journey, Not Just the Answer

For a single completion, grading the answer is enough. For a loop, it is dangerously incomplete. An agent can land on the right answer down a wrong path – three unnecessary tool calls, a permission gate grazed, a lucky guess that will not be lucky twice. And it can fail honourably, every step sane, defeated by a broken tool. If you grade only the destination, both lessons are lost. So agent evals grade the trajectory: which tools were chosen and in what order, how many turns the goal took, what the run cost against the meter, which safety boundaries were approached. The transcript is the deliverable; the Route says judge trajectories, not just final answers, and this is why.

The Half That Measures

Stage 05 carries the number that frames this whole stage: in late-2025 surveys, roughly nine in ten teams running agents had observability, and only about half had evals. The gap is easy to explain. Dashboards arrive with the platform; golden sets have to be built, case by collected case. But observability tells you what happened; evals tell you whether it was any good – and a team with traces and no grades has a flight recorder and no instructor. Chapter 12's cascade names the consequence without mercy: you can only be as reliable as you can measure. If the vocabulary still feels new, notice that it is only new in costume:

A Number You Can Argue With

The Route's fluency test for this stage is disarmingly practical: a teammate asks did that change help, and you answer from your eval set in five minutes, not from a hunch. That is the whole prize. The green bar is gone, and what replaces it is honestly better: a number you can argue with, attached to evidence you can read. The previous dive ended by calling autonomy trust extended one notch at a time, on evidence. This piece is where the notches come from.