There was no API-level access to test against. On one platform I worked with, that was the whole problem: I could not reach the code, so the only seam I could trust was the one a real user touches. The screen. And the screen did not hold still. The vendor's front end renumbered its own element ids whenever it liked, so a script that passed on Monday could fail on Tuesday for no better reason than a framework renaming its own components. I have written that story already, as a story about finding buttons. So when agents that operate screens arrived, I watched the demos with a very particular flinch. They had taken my Tuesday morning and made it the product.
A computer-use agent is an integration with no contract: it works a screen the way a person does, through pixels and clicks. It is the fallback where no API exists and the wrong choice where one does – and every page it reads is untrusted input.
- The ladder every integration climbs down, from API to tool server to the page's own structure to raw pixels, and the rule for where to stop.
- What the screenshot loop really costs you in time and tokens, and why it breaks the way your old UI tests did.
- Why every pixel on the screen can carry instructions, and the four lines the big vendors' own guidance tells you to draw first.
Computer use gets explained two ways, and both fail the working engineer. The demo version shows an agent booking a flight and moves on. The warning version lists everything that could go wrong and never says when you should use one at all. This piece takes the engineer's road: read the agent as an integration with no contract, ask when that is the right trade, and draw the boundaries before anyone hands it a mouse.
Climb Down the Ladder Only as Far as You Must
Every integration you have built had a best available seam, and you took the highest one that held. An API, when the system offers one, because a written contract is the cheapest thing you can lean on. Failing that, a tool server: the MCP move, where someone wraps the system in a contract and the model discovers it at runtime. Failing that, the page's own structure. Failing even that, the pixels.
The API
Typed inputs, real errors, a version. Cheapest to lean on and cheapest to test. If it exists, stop here.
The tool server
A server wraps the system and the model discovers what is on offer. You inherit someone else's contract.
The page's structure
A browser already knows every button's name and role, because assistive technology needs it. An agent can read that instead of squinting.
The pixels
A screenshot, a guess about where the button is, a click. It works on anything a person can see, which is exactly why it is last.
The rule is the one you already follow: stop at the first rung that holds. Microsoft's Playwright MCP is built on the third rung; its README says it uses the accessibility tree, not pixel-based input, and needs no vision model. Anthropic's documentation says the same thing from the other side: for tasks that stay inside webpages, its browser tool is the closer fit, because it reads and acts on the page itself and needs no full desktop.
Computer use is the bottom rung. It is for the desktop application with no automation hooks, the legacy portal, the vendor screen that is the only door in. Everywhere above that rung it is the wrong choice, for reasons the next two sections make plain.
The Loop on the Wire
There is no protocol to learn here, only a loop, and the three big vendors describe it almost identically. Your code looks at the screen. The model says what to do. Your code does it. You look again.
Notice who holds the mouse. The model proposes and your code disposes, which is the same split payments settled on when it separated authorization from clearing. It is the one gate this design gives you for free: every action passes through code you wrote before it touches anything. OpenAI's guide tells you to use it. Set step, time or cost limits, support cancellation, and check the actual outcome rather than trusting that the click landed.
Why It Is Slow, Costly and Brittle
An API call is one round trip. A computer-use task is dozens, and every one carries an image. Latency and cost grow with the number of steps, and the context grows too: Anthropic's documentation warns that long loops accumulate screenshots as tokens, and that past twenty images in a request, stricter size limits apply. The agent is spending your budget to look at the same screen again.
And it breaks the way my scripts broke. Pixels move. A button changes its label, a banner pushes the form down, a dialog appears that nobody planned for. What saved my scripts was never one clever locator. It was five or six alternative ways to find each element, plus one fallback guaranteed to match, and a check at the end that I had landed where I meant to. A model guessing from pixels is one locator with no fallback, so you build the fallback around it: verify the outcome, not the click. That is the same discipline as an eval that grades the journey, applied to a hand that can misplace itself.
Every Pixel Is Untrusted Input
With a tool server, the confused deputy arrives through a tool result. With a screen, the whole surface is the attack. Any web page, any email, any document the agent opens can carry text addressed to it, hidden in white on white or tucked into an image. Anthropic's documentation says plainly that instructions on webpages or in images might override yours. It runs classifiers over the screenshots to catch this, and still says the precautions remain important with the classifiers in place. OpenAI's guide puts it in five words: treat screen content as untrusted.
The honest part is that this will not be patched away. The UK's National Cyber Security Centre wrote in December 2025 that prompt injection will never be mitigated the way SQL injection was, because a language model draws no line between data and instructions; its advice is to reduce the risk and the impact. That is Chapter 17's argument and the Instagram case again: capability is not authority. Give an agent a screen and you have given it every capability a person at that desk has, on the word of whatever text it last read.
Four Lines to Draw Before You Hand It a Mouse
Put the three vendors' guidance side by side and four lines recur. They are trust boundaries, in the same sense as the three primitives of a tool server: each limits one thing the agent can get wrong.
The box
A dedicated virtual machine or container with minimal privileges, an isolated browser, a sandbox. Never your own desktop.
The allowlist
Internet access cut to named domains, and no sensitive data such as login details inside the box.
The gate
A person confirms anything with real-world consequences: a purchase, data sent out, a destructive change. Google's API flags such actions for confirmation.
The meter
Step, time and cost limits, a way to cancel, and a check of the real outcome at the end.
None of these is new. They are the controls you already put around a test run, and the next section is the proof.
Same Discipline, New Hand
If you have ever kept a UI test suite alive, you already know how to run this agent:
The Hand of Last Resort
Stage 03 of the Route is called Give It Hands, and a tool server is the hand you want: boring, typed, testable. This is the hand for when nothing else fits. Prefer the contract. When there is none, work the screen inside a box, with a gate, a meter and a way to check. The skill is what it always was in those UI tests: knowing where the seam is, and how to find it again tomorrow when the vendor moves it.
Hands come before autonomy. Next, the loop that decides when to use them: The Loop Around the Model. Or step back onto the Route at Stage 03.