Who Pays When It Varies

In a charter I wrote for a backend team in 2020, one line on my own list of responsibilities holds a whole position on how engineering should be organised: "Tech Stack, Development Process and SDLC Independence but Cloud Operational Consistency and Dependency."

On one side of that "but", a team chooses its own tools and its own way of working. On the other, the way its software runs in the cloud is the same for every team. The same notes turn the sentence into two lists, and at no point do they say why an item sits on the side it does.

Two Lists, and One Item on Both

A microservices brief in the same notes names the split outright, as the distinction between independent and dependent decision making, and lists both sides.

Read the documents against each other and one item turns up on both sides. The brief puts deployment on the dependent list, next to monitoring. A post I drafted that year says a team must be independent in its "Deployment Architecture and Strategy". The charter's own line separates the development process, which is independent, from cloud operations, which are not, and deployment is the step that joins the two.

Three documents from the same hand, and deployment sits in three places. Nothing in any of them says where the line comes from, and without that, a row nobody has listed yet has no side to go to.

Who Pays When It Varies

The reason I would write beside those lists now is one question. When two teams do this differently, who pays for the difference?

Start with the first list. If one team writes Kotlin and the team next to it writes Go, the difference is paid for by the team that chose: in its hiring, its builds and its own learning. The same goes for a testing strategy, a merge rule or a pipeline, and for which managed database a service uses, because the team that picked it is the team that lives with it. Variety on that list costs only the people who wanted it, which makes them the right people to decide.

The second list behaves differently. When one team invents its own log format, the bill goes elsewhere: to the engineer from another team following one failed request across several services, to the pipeline that has to parse every format anyone ever chose, and to the dashboard that cannot add two services together because they named the same metric two ways. When a team shapes its errors its own way, every client that calls it pays. Every Error Has an Address makes the case for one error catalogue; this is the reason it has to be one.

That is the rule the lists were missing. A decision whose cost stays inside a team belongs to that team. A decision whose cost crosses the boundary gets made once, for everyone, because the people who would pay for it were never in the room where a team made it.

The same question settles deployment. How a team builds and ships its artifact is paid for by that team, in its own lead time. Where the artifact runs, how it reports its health, and what the engineer on call sees when it fails are paid for by whoever carries the pager that night. Deployment turned up on both lists because the line runs through the middle of it. It also explains the charter's pairing of cloud services with cloud operations: which service a team uses is its choice, and how that service is run is everyone's.

The Starter Every Service Copied

A list on its own makes nothing consistent. The same charter also gave me the job of "Providing A Base Project As Baseline for Backend Development of APIs". I built it, and new services started from it.

It carried the parts every service would otherwise have written its own way: a validation wrapper, a proxy wrapper for how services call each other, a cache wrapper and a Kafka wrapper, with the common modules and a parent POM around them. A team that began from the base project had the consistent version of the expensive things on its first day, and kept every decision on the first list for itself.

The industry has names for this kind of starter. At Netflix, the common platform and tooling that central teams build for every development team is the paved road, and each team keeps the freedom to leave it. Spotify described its golden paths in August 2020 as "the opinionated and supported path" to building something, written after a culture of autonomous teams had left it with what it called rumour-driven development. My base project was the same idea on a much smaller scale: consistency that arrives as the easiest thing to do, instead of as a meeting a team has to attend.

What Must Not Drift

There is an obvious objection, and I have made it myself. In When Architecture Becomes a Leadership Problem I told teams to resist a shared library for everything in common, because every shared dependency is a coupling point. A base project full of wrappers could look like exactly that.

It was not one, because nothing depended on it. A new service copied the base project when it was created and owned its copy from then on, so no release of mine could ever break a team's build.

A copy has a price of its own. Every team that owns its copy is free to change it, and over time some will. So what has to stay the same across services is the contract the code produces rather than the code itself: the fields on a log line, the shape of an error body, the names of the metrics, the header that carries a trace from one service to the next. A contract can be checked wherever services meet, at the edge, in the pipeline that reads the logs, or in a test against the agreed shape, whatever language produced it.

The trace header is the cleanest case. The W3C made Trace Context a Recommendation in February 2020, a standard for how that context is sent and modified between services. It is a format, and a service written in any language can honour it without importing a line of anyone else's code.

Six Years Later, One Level Down

The standards I work to now carry the same split, one level down. Their default for a back end is a single deployable with one module per bounded context, and their rule for everything the modules share reads: "Cross-cutting concerns (errors, logging, correlation, metrics, security, transactions) are implemented once, in a shared kernel, and used by every module. Each module decides its own internals."

Inside one deployable that kernel can be code, because everything that uses it ships together and no copy of it can wander off on its own. Between deployables the same concerns travel as a contract. The list itself barely moved in six years. Errors, logging, metrics and transactions are on both. Correlation joined it, and so did security, which the brief had kept on a line of its own. What changed is the unit on each side of the line: a team and its services then, a module and its internals now.

The question underneath did not change at all.