The Eval and the
Runbook: Forces 07–08

Forces 05 and 06 bring artifacts to production and route AI task payloads through the running system. Forces 07 and 08 start where the monitoring dashboard used to be enough. Force 07 adds the fourth layer to the test pyramid that nobody has built yet – AI evaluation. Force 08 asks what SRE looks like when the failure mode is "confidently wrong" rather than "unavailable" – and when the alert that should fire does not, because the system is behaving exactly as designed.

The traditional test pyramid was built for a world where software behaved deterministically. Given the same input, the same output emerged. Testing meant specifying inputs, asserting outputs, and confirming the gap between the two was zero. This is still true for the deterministic parts of your system. It is not sufficient for the AI parts.

A language model given the same input will produce semantically similar but textually different outputs across runs. Its response can be factually incorrect, contextually appropriate, confidently delivered, and completely undetected by every existing test in your suite. None of your current quality tooling was designed for this class of failure. Most teams are shipping AI features with a testing gap large enough to affect real users for days before anyone notices.

The Four-Layer Test Suite

Layers 1 and 2 are widely adopted and well-understood. Layer 3 – data quality testing – is implemented by teams with mature data engineering practices and largely absent everywhere else. Layer 4 – AI evaluation – is understood conceptually, has good tooling, and has almost no production adoption. Most teams that ship AI features have never run a single automated AI evaluation test in CI.

Sizing the Functional Matrix: Orthogonal Arrays

Before the new layer, one correction to the oldest one. Layer 1 has a sizing problem that predates AI entirely. A feature with four configuration parameters, each taking three values, has 81 possible combinations. Nobody writes 81 test cases, so most teams sample by instinct, and coverage becomes whatever the last engineer felt like that afternoon. There has been a standard answer since the 1980s: orthogonal array testing, borrowed from Taguchi's design of experiments, derives the minimal balanced set of cases that still covers every pairwise interaction. For the same four parameters and three values, the array needs nine tests. Nine, not 81, and every pair of parameter values still appears together at least once.

The empirical backing is strong. NIST's software fault studies found that a majority of examined failures were triggered by a single parameter value, and up to 97 percent by the interaction of at most two, which is why pairwise coverage catches most of what exhaustive coverage would at a fraction of the count. Modern practice generalises the classical arrays into covering arrays (NIST's ACTS tool builds them for arbitrary parameter counts and strengths), but the discipline is unchanged: the test matrix is derived from the input structure, not improvised from habit.

The reason it belongs in this chapter: an orthogonal array is a computable standard, which makes it a natural scope for an agent. Hand a model the parameter table and the coverage strength, and it derives the matrix and generates the cases, plus the boundary cases the array deliberately leaves out. Test case selection stops being taste and becomes a deliverable a reviewer can check against a published method. That is the same move Chapter 6 argues for every phase: scoped, not constrained.

What Nothing Is Telling You

The last three rows are where your users are living right now if you have shipped an AI feature without Layer 4. The system is healthy by every metric your monitoring tracks. Users are receiving incorrect, outdated, or hallucinated responses. The symptom – bad AI output – and the cause – a data pipeline failure, a stale embedding index, a retrieval precision threshold that was never defined – are invisible to each other.

The AI Evaluation Layer in Practice

The primitives for AI evaluation exist and are production-ready. Ragas and DeepEval provide the core metrics. LangSmith provides the tracking infrastructure and golden dataset management. The tooling is not the gap – the adoption decision is.

Site Reliability Engineering was built on a foundational assumption: a healthy system is one where latency is within bounds, error rate is below threshold, and throughput is within capacity. Define those bounds, alert on breach, escalate, fix. The runbook captures what to do when any of those three numbers goes wrong.

AI systems introduce a failure mode that none of those three numbers captures: the system is fast, reliable, and operating at normal throughput – and it is giving users wrong answers. There is no alert for "confident incorrectness." There is no SLO for "degraded reasoning quality." The runbook has no entry for "the model started hallucinating entity names in user-visible summaries three days ago and nobody knew."

New SLO Vocabulary for AI Systems

The answer is not more dashboards. It is defining what "healthy" means for an AI system – the equivalent of latency and error rate SLOs, applied to AI behaviour. These SLOs are not standard; every organisation must define them from the characteristics of their domain. But the categories are consistent:

Autonomous Action Rollback: The Hard Constraint

Platform Engineering for Agent-Facing Reliability

Force 08 extends platform engineering into new territory. The 2026 platform team is building infrastructure not just for human-facing services, but for agent-facing reliability – the primitives that make AI systems observable, safe, and operable at 3am.

The Cascade: Each Force Is the Ceiling for the Next

These eight forces are not independent. They form a cascade – each force upstream determines the quality ceiling for every force downstream.

This is why the forces series starts with requirements and DDD rather than tools and infrastructure. A team with excellent AI observability tooling and a shallow domain model will observe excellent-quality failures with great clarity. The hard problems stay hard from the top down.