Short engagement, measurable exit. That was the rule for the six months I spent back in Lahore in 2018, building trip planning and the loyalty platform for Careem, the ride-hailing platform Uber later acquired: every feature we shipped had a number attached before we wrote it.
Writing a number down is the easy half. The harder half is deciding, before the code exists, what counts as the feature working, and how you will know the feature is what did it. Most of this essay is that second half, because it is the part a claimed result has to survive.
- Why the number has to exist before the first line of code, and what it has to name.
- The question every claimed lift has to survive, and the plan that answers it.
- What happens when the number turns into the goal, and what the discipline costs up front.
Write the Number Down First
A number chosen after launch is easy to pick and hard to trust. By then the feature exists, the team is proud of it, and there are a dozen measures to choose from, at least one of which will have gone up. Chosen before, the same number is a prediction the feature can fail.
Barry O'Reilly's template for hypothesis-driven development fits the whole discipline into two sentences.
We believe [this capability] will result in [this outcome]. We will know we have succeeded when [we see a measurable signal].
The second sentence is the one teams leave out, and it is the only one that can show anybody was wrong.
I have argued on this site that KPIs and OKRs should be built around value, not activity. A feature's number is the smallest version of that rule. It names what the feature is for, in terms a user would recognise, instead of what the team did.
A number chosen after launch can only agree with you.
What the Number Has to Name
Four things make a feature's number worth writing down.
The experimentation literature has a name for the agreed measure: the overall evaluation criterion. Ron Kohavi, Diane Tang and Ya Xu define it as a quantitative measure of what an experiment is for, and count agreeing on it among the things a team needs before its experiments are worth running. An earlier paper by Kohavi and his colleagues is blunter: agree on it before the experiment starts. It can be refined between experiments. It should not move once the data is in.
Decide How You Will Know It Was the Feature
Suppose the number moves. Retention is up the month after launch. The feature shipped in the same month as a marketing push, a price change and a public holiday, and every one of those moves retention too. A before-and-after comparison cannot pull them apart, and nobody in the room wants to be the one who says so.
The cleanest answer is a control group: some users get the feature and a comparable group does not, assigned at random, over the same weeks. Whatever else happened that month happened to both groups, so a difference between them can be put down to the feature with a stated confidence, not a hope. In their 2009 survey for web products, Kohavi and his colleagues call controlled experiments the best scientific design for establishing causality.
When I disagreed with a CEO about what our users wanted, I stopped arguing and let the users decide. The test settled what the argument could not.
Some changes cannot be tested that way. A platform migration, or a new price for everyone, rarely has a clean control group. Then the plan gets written down in advance instead: what else could move the number in that window, what you will compare against, and what result would make you conclude the feature did nothing. Writing that before launch is uncomfortable. It is also the version that survives the question.
When the Number Becomes the Goal
Attach a number to a feature and you have also created an incentive. The economist Charles Goodhart made the point about monetary policy in 1975, and in 1997 the anthropologist Marilyn Strathern gave it the wording most people quote: when a measure becomes a target, it ceases to be a good measure.
It is one of the core ten in my atlas of mental models, and a feature's number is where most teams first meet it.
Rewards schemes sit right on top of this, because the incentive is the product. Tiers and points exist to change behaviour, and people are very good at changing exactly the behaviour a points table pays for.
The guardrail is the defence: pair the number you want to move with the one it must not break, and read them together.
The Same Trade, in the Tests
On that team I also introduced test-driven development and continuous integration, to bring production defects down. It is the same bargain as the number: you pay before the code exists, and you collect later.
The clearest public picture of that bargain I know is a 2008 case study of four industrial teams, one at IBM and three at Microsoft, by Nagappan and his colleagues. Against similar projects that did not use test-driven development, their pre-release defect density was 40 to 90 percent lower, and their managers estimated the initial work took 15 to 35 percent longer.
The comparison projects were not assigned at random, so this is what four teams observed, not proof that the tests caused the drop. After everything above, claiming more would be odd.
A feature's number works the same way. The hour spent writing it down, with its baseline, window and guardrail, is paid before anyone knows whether the feature will work. The saving arrives later, as an argument nobody has to have.