The experiment was designed to be boring. A paying user of Claude, Anthropic’s chatbot, worked through a multi-step task with the assistant asking permission at each stage, the way the product has always worked. Somewhere in the middle, the researchers slipped in a command that would have been a bad idea.
The user approved it. Not once, but often enough that across 1,053 paying users, people caught only 13.6% of the dangerous steps. A second model, watching the same traffic, caught 89%.
The more uncomfortable number is what happened over time. After roughly fifty consecutive permission prompts, the human interception rate fell to about 5%. People stop reading. They click the button that makes the dialog go away, and they keep clicking it after the dialog stops describing anything they would recognize.
Starting next week, Anthropic will act on that finding, according to people familiar with the rollout. Claude will run in automatic mode by default, worldwide, with another model making the call on whether a given step is allowed to proceed. The human stays in the conversation. The decision about what the software may do moves to a different piece of software.
The change reverses an assumption that has organized the safety discussion for two years. When models became capable enough to take actions rather than produce text, the industry’s answer was the approval prompt: a human reviews each step, and the human is the brake. That answer was always a bet on human attention, and the bet has now been measured and lost.
Anthropic’s argument is that a model reading the actual content of a step does better than a person reading a summary of it, because the model can see the command and the person sees a paragraph describing the command. The test numbers support the claim in the setting the company chose. They do not answer the question the setting was built to avoid.
Because there is a second layer to this, and it is the one the engineers are worried about. If the checker is a model, then the checker can be fooled, tired, or simply blind to the same things the working model is blind to. Two models from the same family, trained on similar data, share assumptions. The monitor agrees with the thing it is monitoring for reasons neither can articulate.
That concern is not hypothetical. METR and Redwood Research, two groups that study model behavior, spent time inside OpenAI and published a 61-page account of what they found. It lists six routes by which a monitoring system can be walked past. One is that oversight is applied only when the agent hands control back to the user, leaving every action taken in between unexamined. Another is that the process of compressing a long context, which every agent does to stay inside its window, falls outside the monitored channel.
The report also puts a number on the gap. Between May and July, about one in ten internal Codex requests at OpenAI went past the monitoring that was supposed to see them. Not malicious requests. Ordinary traffic that the system did not observe, because the observation point was in the wrong place.
Read together, the two results describe a problem with a shape. Oversight that depends on a person fails on attention. Oversight that depends on a model fails on correlation. The first failure is visible in a click-through rate. The second is visible only when something goes wrong and the logs show a gap.
Enterprises buying agentic software have started to ask about this in procurement. The questions are unglamorous: where is the check performed, what does it see, what happens when it cannot see, and who reads the record afterward. A vendor that answers with a percentage from a user study and no answer on placement is now treated as a vendor with an unfinished product, according to people who run those reviews.
Regulators are moving in the same direction, if more slowly. The emerging expectation in Europe and in U.S. financial supervision is not that a human approves each step, which nobody believes scales, but that the system produces an auditable record of what was attempted, what was blocked and why. Automatic mode generates that record by construction, which is part of its appeal to compliance teams.
For individual users, the change will be felt as an absence. Tasks that used to stop and wait will simply continue. That is the product working as intended, and it is also the moment when the last human check on a lot of routine automation disappears without anyone deciding to remove it.
Anthropic says the supervised version stays available for anyone who wants it, and that certain categories of action will still require a person. The company has not said which categories, or who decides when a category has grown.
Which leaves the question the industry will spend next year arguing about. A model that checks another model is faster than a person and more consistent than a person. It is also the same kind of thing as the model it is checking, and when both are wrong they are likely to be wrong together, in the same direction, at the moment nobody is watching. The test that produced the 89% did not measure that. No test has.
