Technical Excellence

Every Error Has an Address

Part VI · Excellence Across the Nine Phases 07 · Monitor

In July 2020 I wrote a design checklist with thirty-five numbered items, one for each decision a team had to settle before it built anything. Item twenty-seven was about errors, and its recipe fits on a single line: "Issue Code and Debug Id with HTTP Code, Module, Sub Module, Class Level."

Four years before that line, in March 2016, the IETF had published Problem Details for HTTP APIs, RFC 7807, by Mark Nottingham and Erik Wilde. It is a standard shape for an error response, the very thing item twenty-seven and its neighbour on the list, number thirty-one, set out to design from scratch. My checklist does not mention it.

// the crux

The code is what you count. The request id is what you find.

// in one breath
  • A 2020 checklist item that built each error's address out of class names, and why that address could not survive a refactor.
  • The split a four-year-older standard had already drawn, and the one string both readers of an error share.
  • Why a catalogue of error codes turns out to be the first monitoring plan.

The instinct behind item twenty-seven was sound, and I still hold it. When an error cannot be traced, debugging turns into archaeology, which is how I put it when I first wrote the principle down as one of five stepping stones. The recipe was the problem. It answered one question well and two others badly.

item twenty-seven

An Address Built Out of Class Names

Read it as the engineer on call and the recipe looks generous. A debug ID that carries the module, the sub-module and the class tells you which file to open before you have read a single log line.

The trouble starts at the next refactor. Rename a class, move it into another module, split a service in two, and every debug ID issued before the change describes a layout that no longer exists. Every ticket, runbook and support note that quoted one now points somewhere that has gone.

It also tells the wrong reader too much. A class name in a public response is a small map of the inside of the system, handed to anyone who triggers an error on purpose, and no help at all to the person who only wanted to pay.

And it cannot tell two failures apart. Two customers who hit the same class on the same night receive the same ID, so the question an investigation actually starts with, which of these happened to this person, has no answer in it. My notes from that year do mention correlation IDs, on another page and in another list. Item twenty-seven never connects the two.

march 2016

The Split the Standard Had Already Drawn

Problem Details gives an error response five members, and two of them do the job item twenty-seven tried to squeeze into one string. The type identifies the kind of problem, and its short human title is not supposed to change from one occurrence to the next. The instance identifies this particular occurrence. One name that stays put, and one identifier that is never reused.

My 2020 recipe fused the two halves, then built the result out of the one thing guaranteed to move, the layout of the code. RFC 9457 replaced the original standard in July 2023, kept the same five members and added a registry of common problem types. The split survived the revision untouched.

The standards I work to now take that split as given. Every error carries a code from one versioned catalogue, shaped as context, kind and number: PAY-RULE-001 for a broken payment rule. Every request carries a request id, accepted or generated at the edge, returned on every response and written on every log line. The request id is the debug ID now. The class level I built into the address in 2020 sits on my list of things a response must never contain, next to stack traces.

// one error, both halves
{
  "type":      "https://example.com/problems/payment-rule",
  "title":     "Payment rule violated",
  "status":    422,
  "detail":    "The transfer is above the daily limit for this account.",
  "instance":  "/transfers/7f3c9a2e",
  "code":      "PAY-RULE-001",
  "requestId": "4bf92f35-7a1c-4e0b-9d3c-5e2a1f6b8c40",
  "retryable": false
}
Illustrative. The first five members are Problem Details. The last three are extensions from my standards, which also add a list of field errors, empty here.

The code is also what a calling program branches on, never the message, which is written for people and may be reworded or translated. That is why the retry flag sits beside it: the caller should not have to guess whether trying again can possibly help.

the second reader

Two Readers, One Error

Every error is read twice, by two people who will never meet. One is on the other side of the screen and needs a sentence they can act on, with nothing to decode. I have written about that person before, standing at a till while a card is declined, and I will not retell it here. The other opens the logs hours later with no context at all.

Item twenty-seven served the second reader, and only halfway. The rule I use now serves both with a single string. An error screen shows the message and the request id together. The person can read that id out to support, and the engineer can paste the same string into a log search and land on every line the request wrote, in every service it passed through. Those lines carry no secrets and no personal data, which is a rule of its own.

That id is the only thing the two readers share, and it is enough.

// the part worth keeping

A good error address can be read aloud by the person it failed, and pasted into a search by the engineer who fixes it.

counting at zero

The Catalogue Is Also the Monitoring Plan

This is where errors meet the Monitor phase, and where it shows whether monitoring was designed or bolted on after the first bad night.

In my standards the catalogue is the only place an HTTP status may come from, and a status typed straight into the code is one of the things I push back on in review. That reads like tidiness. It pays off on a dashboard. When every failure leaves through one catalogue, the catalogue is a complete and finite list of everything that can go wrong, and a finite list can be counted.

So the code goes into the metrics as a label, and the request id never does. A request id is unique by design, so as a label it would create a new time series for every single request, which is exactly the kind of label Prometheus's own guidance warns against. Counting belongs to the code. Finding belongs to the request id, on the log line, where being unique is the whole point.

Counting has one trap, and my standards check for it before anything is called done: an error-rate panel that reads "no data" when nothing failed. A labelled error counter does not exist until its first increment, so on a quiet day the ratio has nothing to divide and the panel goes blank. Blank looks exactly like an exporter that has stopped reporting. It also looks like a query for a metric the application never exposed, so my standards check every dashboard query against the metric names the application actually exposes. Good Swimmers Drown Quietly makes the wider case that silence and health look the same from outside; this is the mechanical half of it.

Prometheus's instrumentation guidance names the fix: for any series you know may exist, export a zero in advance. A catalogue is precisely that list. Register every code at zero when the service starts, and a quiet day reads 0 instead of nothing.

the rewrite

Item Twenty-Seven, Rewritten

If I were writing that checklist again, item twenty-seven would keep its intent and lose its recipe.

// item twenty-seven, today
Issue code
From one versioned catalogue, named for what went wrong: context, kind, number.
Debug ID
The request id. Accepted or generated at the edge, returned on every response, written on every log line.
HTTP status
Only from the catalogue.
Validation
Four places, each with its own status and family of codes: the UI for usability and never trusted, the request (400), the domain (422 for a broken rule, 409 for a conflict), the database (409).
Class and module
On the log line, for the engineer. Never in the response.
Counters
One per code, registered at zero when the service starts.
The response members come from Problem Details, RFC 7807 (March 2016) and RFC 9457 (July 2023). The rules are from the standards I work to now; the two highlighted rows are what the 2020 item was missing.

The catalogue carries a version too, for the reason The Stack Aged. The Categories Didn't. gives: in my notes from those years, the error catalogue is on the list of things that need a version and an expiry date.

The class level did not disappear. It moved to the log line, where its only reader is the engineer who needs it, and where a refactor changes nothing that a customer, a ticket or a dashboard has ever seen.

Every error a system will ever return gets an address before the first one happens, whether anyone designs it or not. The only choice is whether that address is built from things that stay still, or assembled out of whatever the code happened to be called that year.
// carry forward

Monitoring can only count what the design agreed to name. Part VI continues on either side of this phase: the release nobody remembers the next morning, and the gap between shipped and delivered.

// continue exploring