Three Kinds of User

Fifty megabytes, a hundred and fifty, five hundred. That is roughly how much memory one signed-in session takes on a third-party risk platform I look after, depending on who is signed in. For a long time we could not say it. We knew the platform served about two hundred people at a regulated financial-services client. We did not know what two hundred people cost.

The question mattered because a bigger client was a real possibility, and nobody could tell me what three or four hundred users would need. We could not change the vendor's code. We could only watch it run. So we taught a model to watch it with us, and the useful part turned out to be everything we had to build before the model could see anything.

// the crux

The model read the numbers. Choosing which numbers to keep was ours.

// in one breath
  • Why the model never got the keys to the cloud console, and what it got instead.
  • Two hundred users, three kinds, and the vocabulary that changed every capacity meeting.
  • A tester that reads the vendor's manual, writes its own handover note, and waits for a person to say yes.

What the Model Was Allowed to Read

We run the platform in two clouds: one active, one passive standby, because the client wanted its data to stay in Germany and chose two providers over two regions. Both clouds have dashboards of their own. We did not hand those to Claude.

Instead we built a small pipeline we could reason about. The audit and activity logs from both clouds flow into an InfluxDB time-series database, an engine turns them into daily numbers, and Grafana draws them on instances we run ourselves. Claude gets read access to that database and nothing else. Every morning it turns the day's numbers into a report: what grew, what shrank, which errors appeared, how the two clouds compare. The report goes to a mailing list for everyone who runs the platform.

That boundary was a design decision, and I would make it again. A model with a console can do things. A model with a database of agreed numbers can only describe them, and a description is something a person can check.

Two Hundred Users, Three Kinds

The platform keeps each user on one server for the length of a session, and a session can stay open for a week. People dip in and out at different hours. So the average user told us nothing. A user who left one form open for five days and a user who ran a heavy report every morning were both "a user".

With the daily numbers in one place, we sorted sessions by how much memory they held and found three clusters. We called them profile one, profile two and profile three, at roughly fifty, a hundred and fifty and five hundred megabytes. Then we counted how many of each were signed in at peak.

The names did more than the numbers. A capacity meeting that used to start with "how many users?" now starts with "how many profile threes?", and the answer prices memory, CPU and storage directly. When I ask what four hundred users will cost, I can now ask it properly: four hundred of which kind? The model drafts the trend and the forecast. A person reads it and decides what to provision.

A Tester That Reads the Manual

The vendor ships a release every quarter, and before it reaches the client we test the eight to ten scenarios the business cannot live without. There is no API to test against, so the testing happens where a user works: in the browser, signed in as test users with different rights.

For that I built a test assistant on a graph-based retrieval layer, the approach Microsoft Research described in 2024 as GraphRAG, stored in Neo4j. Its knowledge is the vendor's full user guide, our walkthrough recordings converted to text, and the critical scenarios written out step by step. A chat interface sits on top, and three people in our QA team use it.

A release starts with the release notes. The tester asks which fixes touch the critical scenarios, and the assistant walks those scenarios one step at a time, driving a headless browser, and stops after each step to wait for the person who started the run. At the end it writes a report, as a file and into the database beside the performance results. The recording side of this, and the trouble with element IDs that change on every build, is in A Weak Claim to Being Valuable.

The Handover Note

One limit shaped the design more than any other. A session holds about 256,000 tokens of context, and a long test run fills it. When it fills, the run cannot just carry on.

So I wrote a handover procedure. At the end of a session the assistant writes a short handover file: where the run stopped, what passed, what is still open. The next session starts by reading it, and the history sits in a Postgres database so the assistant can look further back when it needs to.

It is the oldest discipline in operations, asked of a machine. A shift that ends without a handover note leaves the next shift guessing. The model needed the same note a person would, for the same reason.

Why a Person Stays in the Loop

Every part of this is still a pilot, and I say so to anyone who asks. The alert analyses are scored against a rubric before anyone trusts them. The test assistant waits for a person after every step. The capacity forecast is a report, never an order.

That is a choice about this domain, not about the tools. On a governance and compliance platform for a financial client, a wrong answer can cost money or a finding from an auditor. Until the evidence says otherwise, a person decides. I am more sceptical than many people about fully autonomous agents. I may be wrong about the future. I am not willing to be wrong about this platform.

The next step I have proposed is narrow on purpose: when the tester finds a failure in a critical scenario, the assistant opens a ticket and assigns it to the person who owns that area. A person still reads the ticket. The machine only saves the walk to the ticketing system.

// the part worth keeping

Every useful answer it gave us came from something we had already chosen to measure or to write down.

A model watching a vendor's platform sounds like the clever part. The clever part was deciding to count sessions by kind, and writing the manual into something a machine could read. Whatever model sits on top next year will need the same numbers and the same notes.

// carry forward

The first-read pilot on our alerts, scored line by line against a rubric, is told in A Weak Claim to Being Valuable. The arithmetic behind any capacity plan is in The Capacity Model Fits on One Page.

// continue exploring