A browser-based game that simulates the permission prompts AI coding agents generate has produced a result that should make every compliance team uncomfortable: human reviewers miss roughly one in three malicious requests. For credential-exfiltration attempts specifically — the kind where an agent asks to read your AWS keys or Kubernetes config — the miss rate is 35 per cent.
The game, built by Belgian developer Alex Wauters, collected data from more than 40,000 runs and 409,000 commands. Players have 60 seconds to approve or deny as many simulated agent requests as possible, losing points for approving dangerous commands or denying safe ones. It is not a perfect simulation of a developer’s workday — the malicious-request density is far higher than real life. But the pattern it reveals is consistent with what Anthropic sees in production.
The Approval-Fatigue Problem
Anthropic’s own telemetry from Claude Code shows users approve approximately 93 per cent of all permission prompts. The company noted in a May post that “the more approvals a user sees, the less attention they pay to each, becoming over time much less diligent in their supervision.”
This is not a criticism of lazy developers. It is a recognition of how human attention works under cognitive load. When a coding agent generates dozens or hundreds of permission prompts during a multi-hour task, each individual prompt receives less scrutiny than the last. The human becomes a throughput bottleneck, not a safety gate.
Wauters saw this firsthand. “I’ve seen people go for --dangerously-skip-permissions as a result because they did not want to find out they stopped their multi-hour agent flows 5 minutes in,” he told The Register. The alternative — manually approving every action — is “a draining activity that invites disaster.”
The data bears this out. The most frequently missed command was npm run analyze, approved nearly 65 per cent of the time despite being able to execute whatever is defined in a project’s package.json. The game tells players what the script contains in a history log above the prompt. Two-thirds approved it anyway, indicating the history log may not be read closely.
What Gets Missed Most
The failure pattern is not random. Scope violations — an agent asking to cat Kubernetes config files or AWS credential lists — were missed 35 per cent of the time. curl requests to unknown APIs and typosquatted packages were missed almost as often. What humans caught reliably were obviously destructive commands: rm -rf on the root directory, recursive chmod 777, crontab injections, git config hijacks.
In other words, humans are good at catching the commands that look dangerous. They are bad at catching the commands that look routine but contain hidden risk. That is precisely the category where AI agents are most dangerous — because the harmful action is embedded in a familiar wrapper.
The AI Governance Institute analysis frames it bluntly: a 35 per cent miss rate on credential-exfiltration attempts is “a material control failure, not a residual risk.” Compliance programs that list human-in-the-loop review as the primary control for agentic AI risk may be overstating their actual protection.
Why This Matters Beyond Coding Agents
The finding has implications that extend well beyond developer tools. Human-in-the-loop oversight is the default safety mechanism in most AI governance frameworks. The EU AI Act requires human oversight for high-risk AI systems. The NIST AI Risk Management Framework treats human review as a core control. China’s agentic AI regulation explicitly mandates human-in-the-loop for autonomous AI agents.
But what if the human in the loop is not actually reviewing? What if the control that regulators, auditors, and compliance teams count on as a meaningful checkpoint is, in practice, a 93-per-cent rubber stamp?
This is the structural problem. The governance frameworks assume a human who reads, considers, and decides. The data describes a human who clicks “approve” because the 47th prompt of the afternoon looks roughly like the 46th.
Anthropic’s response — auto mode — replaces the human with a classifier that caught 89 per cent of dangerous actions in the same controlled study where humans caught 13.6 per cent. It is a tacit admission that the human-in-the-loop model was not working at the scale agentic AI operates.
The Layered-Defence Alternative
Wauters and the researchers agree on the direction: layered defences, not single-point human review. Sandboxing, devcontainers, model-based classifiers, and workflow hooks — each catching what the others miss.
“If the conclusion to draw from Wauters’ data is that humans in the loop are being fatigued into letting malicious commands slip through, and the other end of the spectrum is mass approving everything, then something’s gotta give,” The Register notes. The answer is not to remove humans entirely. It is to stop treating them as the primary safety mechanism for a task they are not equipped to perform at the required frequency and accuracy.
The AI Governance Institute recommends that organisations audit all agentic AI deployments where human review is classified as the primary control, assess whether approval-fatigue thresholds have been set or monitored, and isolate credential-access requests behind a separate, higher-friction approval step rather than mixing them into the general permission flow.
The Uncomfortable Implication
The deeper question is whether human-in-the-loop was ever the right model for AI agent governance — or whether it was a convenient assumption that let everyone move fast without solving the oversight problem.
Regulators wrote it into frameworks because it sounds responsible. Enterprises adopted it because it is cheap. Vendors implemented it because a permission prompt is easier than a classifier. And for a while, it looked like oversight was happening.
The data says it wasn’t. A 13.6 per cent catch rate is not oversight. It is a lottery ticket with a UI.
The question for regulators, enterprises, and vendors now is whether to fix the human review model — lower prompt volumes, higher-friction approvals, better context — or to accept that human review at agent speed is not viable and build something better. Anthropic has chosen the latter. Whether that is the right call depends on whether a 7 per cent classifier miss rate is acceptable when the thing being missed is your AWS credentials.
