Technical Excellence

The Stack Aged.
The Categories Didn't.

Part VI · Excellence Across the Nine Phases 09 · Change

In July 2020 I saved a copy of a technology stack template from a company wiki. Forty numbered rows, each one a category with the tools we had chosen for it. Row four, main language: Java 11. Row sixteen, fault tolerance: resilience testing, chaos engineering, and two tools, Hystrix and Chaos Monkey.

Reading it again six years later, row sixteen is the one that matters. Netflix had put Hystrix into maintenance mode on 19 November 2018. When that page was saved, the tool in its fault-tolerance row had not been in active development for nearly twenty months.

// the crux

The category is a decision. The tool is a lease.

// in one breath
  • A forty-row template from 2020, and the row that had quietly expired before anyone wrote it down.
  • Why every category still needs an answer when so many of the answers have moved.
  • The one field a decision record needs that the classic format leaves out.

Nothing about that was an outage. Netflix called Hystrix stable enough for the applications already running on it and pointed new projects at other libraries. It kept working, and because it kept working, nothing on the page ever asked anyone to look at row sixteen again.

forty rows

Six Years Later, the Page Splits in Two

Read the template today and it divides cleanly. The tools have moved, some a little and some entirely. The categories have not moved at all.

Four of the forty rows, exactly as written in July 2020
Sr. NoCategoryExample(s)
4Main LanguageJava 11
16Fault ToleranceResilience Testing · Chaos Engineering · Tools: Hystrix, Chaos Monkey
Hystrix: maintenance mode since November 2018
30Unit TestingJUnit 5 · Spring Test
37DB Versioning and MigrationFlyway · Liquibase

Every one of those categories still needs an answer on any backend team I would join tomorrow. Services still call other services, and those calls still fail. Schemas still change and still need migrations that run the same way twice. Code still needs tests that run before anyone trusts it. The questions held. The answers are what moved.

The ecosystem around the tool said as much out loud. When Spring moved its Hystrix integration into maintenance mode on 12 December 2018, it kept the category exactly where it was and named a successor for the tool, Resilience4j. The need stayed. The lease changed hands.

the receipt

It Is Still on My Uses Page

I would like to call that a 2020 problem. Open my uses page today, and its Spring and JVM paragraph still describes my stack with "Netflix OSS from the microservices trenches (Hystrix, Eureka, Zuul, Feign)".

I am leaving that line exactly where it is while this essay is live, because it makes the argument better than any example I could build. The line is accurate about where I have been. What it cannot tell you is when, and a stack description with no date on it goes on sounding current long after it stopped being true.

the rule

Each Resource Has a Version and Expiry Date

Somewhere in my notes from those years sits a rule I filed under cross-cutting concerns, written in title case: "Each Resource Has a Version and Expiry Date." I listed what it covered, too. APIs, the error catalogue, configurations, domains, schemas, documents, database scripts. I have kept writing that rule down since. My engineering philosophy gives every component an expiry date, the libraries included, and Are You Ready for Microservices? asks for a version and an expiry on every API, schema, configuration and database script.

The stack template is not on that list, and it carries no dates at all. That is the self-audit. I think it is also the most common gap in how teams handle change: we version the things we build and forget to version the things we chose. For APIs the lifecycle is well understood, from versioning through deprecation to retirement, and I have written about what that lifecycle owes the developers on the other side of it. A choice of library gets no such lifecycle. It gets a row in a table.

This is what the Change phase is really about. By the time a system reaches it, the earlier phases have worked, people depend on the thing, and every choice made along the way has started a clock. The lifecycle field guide names what belongs in this phase: semantic versioning, decision records, the rollback path written first. The part that gets skipped is the clock on the choices themselves.

// the part worth keeping

A stack page records what you chose. Without a date beside it, it can never tell you when to choose again.

the method

Decide the Category. Lease the Tool.

The fix is small, and most of it is bookkeeping.

Decide by category. A category answers a need that will outlive every vendor on the list: calls between services fail, schemas change, code needs tests. That decision is usually right for years, and it deserves the most care.

Treat the tool as a lease. Hystrix was a reasonable answer to row sixteen when the category was first filled in. The lease simply ran on with no end date written beside it, so it expired in the dark.

Give each category one decision record, with a date the classic format leaves out.

// one record per category
Title
The category, not the tool. Fault tolerance for calls between services.
Context
The forces in play when the choice was made.
Decision
The tool, stated plainly.
Status
Proposed, accepted, deprecated or superseded.
Consequences
What the choice makes easier, and what it makes harder.
Review by
A date. On that day somebody asks whether the lease still makes sense, even if nothing is broken.
The first five fields are Michael Nygard's, from his 2011 post "Documenting Architecture Decisions". The sixth is my addition.

Nygard's format already knows that decisions age: deprecated and superseded are right there in the status field. But both are words you write after the fact, once someone has noticed. A review-by date is written before, on the day of the decision, by the person who still remembers why it was made. Mastering Trade-Offs already asks for one sentence on what would make you reopen a call. The review-by date is the day somebody reads that sentence again. Why a decision deserves writing down at all is its own argument, and it is made elsewhere in this library. This is only the date.

row sixteen

Row Sixteen, Rewritten

If I were writing that template again, row sixteen would keep its category exactly as it was: fault tolerance, resilience testing, chaos engineering. The tools column would carry the current answer, and one more thing next to it, a review-by date.

The page would still age. Every stack page does. The difference is that it would say so on its own, before anyone had to find out from a status notice on a repository that no longer merges pull requests.

The next stack will age too, probably faster than the last one. Write the categories as if they will last, because they will. Write every tool with a date beside it, because it will not.
// carry forward

Change is where earlier choices come due. The rest of Part VI walks back through the phases that made those choices, from the requirement that was never a requirement to the pipeline you can trust at three in the morning.

// continue exploring