I spent a career designing for the load I had not met yet. That was the whole art of it: the features are what users touch, but the non-functional requirements, the throughput and the failure modes and the capacity you would need at three in the morning on the worst day, are what decide whether the thing survives once it is live. I got good at anticipating load I could not see. Nobody warned me that one day the load would be the data, and that unlike every other dependency I had ever designed around, it would keep changing its mind after I shipped.
ML system design is system design where your hardest dependency is not a service you call, but data you cannot fully specify, that shifts under you, and that your own predictions feed back into. The model is the small part. The living system around it is the design.
- Why ML system design is ordinary system design with exactly one dependency promoted to first-class citizen.
- The three ways data is not like your other dependencies, and the feedback loop that can quietly poison itself.
- The same non-functional requirements you have always run, in units that decay, and the whole living system drawn as one loop.
Stage 06 is a fork: go wide, into serving and infrastructure, or go deep, toward the machine learning itself. The metal under the model is the wide door, how you serve and pay for inference. This is the deep door, how you design the system that produces a model in the first place. And the good news, if you have designed systems before, is that you already own most of the discipline.
System Design, One Dependency Promoted
Draw an ordinary system and its dependencies sit at the edges: a database, a queue, a payment gateway, a third-party API. You validate what comes across each boundary, you handle its failures, and then you trust it and get on with the logic. An ML system has the same shape, the same services, the same boundaries, with one dependency dragged from the edge into the centre of the design: the data. In an ordinary system, data is an input you check and forget. In an ML system, the data is the specification, the training material, and the thing your own outputs will feed back into. Everything strange about designing these systems follows from that single promotion.
Why Data Is Not Like Your Other Dependencies
Treat the data like a normal dependency and the system will betray you in ways your instincts do not cover, because a normal dependency and a learning dependency behave nothing alike.
The second column is why a model can get worse while sitting perfectly still: the code did not change, the servers did not change, but the world the model learned moved, and the model did not. That is drift, and it is the failure mode with no stack trace, the one an operations engineer learns to fear most, because everything is green and the answers are quietly getting worse. Designing an ML system is, more than anything, designing for drift you cannot yet see, which is the exact skill you built designing for load you had not met yet.
The Loop That Can Poison Itself
Here is the part with no equivalent in ordinary systems. Your model's predictions influence the world, the world generates the next batch of data, and that data trains the next model. The system feeds on its own output. Left unwatched, that loop teaches the model its own biases as if they were facts: recommend the same items, and the only clicks you ever see are on the items you recommended, so the next model learns that those were the only things anyone ever wanted. Every other system I designed reconciled toward a state I had written down. This is the first one that optimises toward its own past and calls the result learning. Naming that loop, and deciding where to break it, is the load-bearing act of the design.
The Same Non-Functional Requirements, New Units
Once the data is a first-class dependency, the rest of ML system design is the non-functional-requirements discipline you already run, in new units. The four lenses I always designed by, throughput, load, capacity, scale, all still apply; they just measure different things.
That fourth row hides the classic trap: the features you compute when training and the features you compute when serving must be the same, or the model meets a world at serving time it never saw in training, and quietly underperforms. It is the ML version of a config that drifts between staging and production, and it is caught the same way, by making the two paths share one source of truth. And the model's own quality is not a vibe you argue about; it is measured on an eval set and gated like any other release. You can only be as reliable as you can measure holds here too.
The Living System, End to End
Step back and the model is a single box in a system that never stops moving. The design is the loop around it.
Every stage of that loop is an ordinary engineering problem you can reason about: a data pipeline, a feature store with one source of truth, a training job, an eval gate, a serving path, and monitoring that watches for drift and trips the loop again. What makes it ML system design is only that the loop is closed and the thing inside it learns. Point your existing rigour at each stage and the mystique falls away.
The Deep Door
This is the deep end of the route, and you do not have to swim here to be an excellent AI engineer; the Route is honest that for most people, wide pays sooner than deep. But if you do come this way, come as what you already are. ML system design is not a new discipline waiting on the far side of a maths textbook. It is the whole-system thinking you have practised for years, pointed at a dependency that learns, decays, and feeds on itself. I spent a career designing for the load I had not met yet. This is the same job, with the load renamed to data, and the data given a mind of its own.
The system learns from a signal. The next dive is about what happens when that signal is a reward. The Reward Is the Whole Problem is the go-deep companion: reinforcement learning, read through the one thing every architect has seen go wrong, a perverse incentive. Or step back onto the Route at Stage 06, the fork.