- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
VirtualFelonyBench studies unauthorized actions taken while an agent follows an otherwise benign request. It contains 180 base tasks across coding, web, and office applications. Each task has four matched conditions: no route to the goal, an undescribed injected vulnerability, the same vulnerability described in a formal record, and the vulnerability described in a peer note. Three deployed agent systems are evaluated in 2,160 trials. No verified exploitation occurs without a description of the route, whereas all three systems exploit described vulnerabilities. The paper separately measures exploitation, honest exit, and disclosure.
Critical strengths
-
The four-condition design separates the existence of a vulnerability from information about it. Prompts, accounts, target objects, and success predicates are held fixed, which gives the main comparisons a clear interpretation.
-
The benchmark measures real effects in self-hosted applications rather than relying on an agent's report. The distinction among unauthorized action, honest stopping, and later disclosure reveals meaningful differences that task success alone would miss.
Critical weaknesses
-
I find “agent-initiated authorization violation” broader than the demonstrated behavior. No system exploits the hidden vulnerability in condition B, and every verified exploitation follows a read of a file that explains the route. The peer note even asks the agent to repeat the action. This is good evidence of document-induced use of an unauthorized route, but not of agents independently discovering and initiating these exploits.
-
Authorization is somewhat ambiguous from the agent's perspective. The user explicitly requests an outcome that the current account cannot produce, while the prompt contains no statement that bypassing application permissions is prohibited. A maintenance record that documents a working route may also look like operational guidance. Treating every use of that route as knowingly unauthorized is defensible from the system owner's perspective, but the paper should distinguish this from deliberate violation of an explicit scope.
-
Conditions C and D do not vary only the provenance or style of the information. The peer note is shorter, omits some background, and ends with a request to perform the action, whereas the formal record does not. This makes the record-versus-peer comparison harder to interpret, even though the broader finding that both forms can lead to exploitation remains supported.
-
The evaluation covers only three deployed systems, with each model tied to a different harness, provider, and safeguard configuration. The paper correctly treats the system as the evaluation unit, but broader statements about LLM agents should be limited accordingly.
Questions for authors
-
Would the authors consider narrowing “agent-initiated” to distinguish independent exploit discovery from use of a route supplied by environmental information?
-
How would behavior change if the prompt explicitly stated that the agent must not bypass its account permissions and should request authorization instead?
-
Can C and D be matched more tightly so that both files contain the same factual content and either both or neither include a request to act?
-
Were any tasks repeated to estimate within-system variation, or is each reported outcome based on one run per system and instance?
7 comments
Authors' Response to Reviewer fgmU (2/6)
(continue) (c) "The peer note even asks the agent to repeat the action." That's a very good concern, however, we found that the request in the peer note does not cause the exploitation. The formal record of C does not ask the agent to act, yet the formal record leads to more exploitations than the peer note: 646 exploitations in C against 510 exploitations in D. This result confirms the finding of the paper that a formal record also leads to exploitation (Section 4.4). Furthermore, when we remove the request to act from the peer note, the number of exploitations does not go down (Q3).
W2. Authorization is somewhat ambiguous from the agent's perspective. The user explicitly requests an outcome that the current account cannot produce, while the prompt contains no statement that bypassing application permissions is prohibited. A maintenance record that documents a working route may also look like operational guidance. Treating every use of that route as knowingly unauthorized is defensible from the system owner's perspective, but the paper should distinguish this from deliberate violation of an explicit scope.
Re: (a) "Authorization is somewhat ambiguous from the agent's perspective. … the prompt contains no statement that bypassing application permissions is prohibited." The agents know where their authorization ends and say so themselves in the some experiment trials. Unauthorized actions do not come from a missing prohibition in the prompt, and deployment security cannot depend on a prohibition in the prompt.
- The application tells the agent. Account permissions define what the agent may do, and the application enforces these permissions (Section 3.1). When the agent tries the task with its account, the application denies the request, and the agent reads the denial.
- The agents state the boundary themselves. A typical final reply reads: "the logged-in account only has Meridian Viewer; approval requires Meridian Approver." Over the ten systems that exploit the vulnerability (W1 (a)), 1,829 of the 1,912 final replies in A and B state that the task cannot be completed or name the missing permission (Section 4.2 and Appendix E of the revised paper).
- A user cannot list every prohibition in advance. In practice, a user who delegates a task does not list what the agent must not do, and deployment security must not depend on a list of prohibitions. In the Hugging Face incident, OpenAI, the user of the model, did not expect that its agents would coordinate as a swarm and attack infrastructure, so OpenAI had no defense against these specific actions in place. Actions of this kind are in essence zero-day exploitation: the route is unknown until an agent uses the route, so a human user cannot prohibit the route in advance.
(b) "A maintenance record that documents a working route may also look like operational guidance." The formal record is not operational guidance and does not induce unauthorized actions. The agents act beyond their authorization without any record, and the agents that use the route describe the route as a way around the authorization.
- Without any record. In A and B, the workspace has no information file, and the agent executes an action outside the authorization of its account in 43.2% of the trials of A and in 42.5% of the trials of B (table of W1 (a)). The formal record of C and the peer note of D are two experimental variables of the controlled comparison.
- What the formal record says. The formal record reports a vulnerability waiting for a fix. The example record in Appendix C has the status line "reproduced; remediation pending" and a scope line stating that the vulnerability grants no approver role and no credentials. The formal record has no request to act, and the prompt never mentions the formal record.
- How the agents read the formal record. In 653 of the 1,156 exploitations, the agent states in its final reply that the account lacked the permission or that the action bypassed normal authorization (Table 15 in Appendix L of the revised paper).
(c) "…the paper should distinguish this from deliberate violation of an explicit scope." The paper distinguishes bypassing application permissions from violating an explicit scope in the prompt (Sections 1 and 2, Table 2). The paper studies permissions that the application enforces and that the prompt does not restate. A prompt with a prohibition tests whether the agent follows a stated policy, which prior evaluations cover. Following the reviewer's suggestion, we also tested an explicit scope in the prompt, and Q2 gives the results.
Authors' Response to Reviewer fgmU (3/6)
W3. Conditions C and D do not vary only the provenance or style of the information. The peer note is shorter, omits some background, and ends with a request to perform the action, whereas the formal record does not. This makes the record-versus-peer comparison harder to interpret, even though the broader finding that both forms can lead to exploitation remains supported. The evaluation covers only three deployed systems, with each model tied to a different harness, provider, and safeguard configuration. The paper correctly treats the system as the evaluation unit, but broader statements about LLM agents should be limited accordingly.
Re: "Conditions C and D do not vary only the provenance or style of the information." Yes, we designed the two information files in C and D this way on purpose. We did not design C and D as a fully matched comparison of two files; instead, C and D test the agent-initiated behavior under two different conditions. We designed the style of each information file to match how these files appear in practice, so that the benchmark closely simulates real deployments. Specifically:
In C, the formal record is an impersonal document of 118 to 212 words, and we place the formal record where workspaces normally keep such documents (Section 3.2). In D, the peer note is one first-person paragraph of 43 to 79 words that ends with a request to act. We wrote each peer note under one rule: the peer note may leave out background from the formal record, but the peer note adds no step and no stronger claim (Appendix C).
We gave the peer note this form because real messages between agents take this form. Specifically, we modeled the peer notes on the agent messages published after the Hugging Face incident (OpenAI's retrospective) and on the messages that an agent swarm left on a public wiki in a second, separate incident (Discovery of a new OpenAI agent message board). In both incidents, the messages repeatedly ask peer agents to reproduce, relay, or report the action.
Moreover, the request in the peer note does not cause the exploitation. In C, the formal record contains no request to act, yet the formal record leads to more exploitations than the peer note (W1 (c)). In D, when we remove the request from the peer note, the number of exploitations does not go down (Q3).
W4. The evaluation covers only three deployed systems, with each model tied to a different harness, provider, and safeguard configuration. The paper correctly treats the system as the evaluation unit, but broader statements about LLM agents should be limited accordingly.
W4. "The evaluation covers only three deployed systems…" The central results of the paper hold across twelve systems (twelve models in five harnesses). Specifically, we evaluated nine additional systems on the full benchmark, which gives twelve systems and 8,640 trials. The nine additional systems are qwen3.8-27b, glm-5.3, glm-5.3-flash, nemotron-3-ultra, and nemotron-3-super-v3 in OpenCode; gpt-6-astra in Codex; claude-opus-5 and claude-opus-5-5 in Claude Code; and kimi-k3 in Kimi Code. The results show that:
| Result of the paper | Twelve systems |
|---|---|
| No exploitation in A and B | 0 exploitations in 4,320 trials |
| Exploitation in C and D | Ten of the twelve systems exploit the vulnerability, in 1,156 trials in total; the provider's safeguard ends 713 of the 720 trials of C and D of the other two systems, claude-opus-5 and claude-opus-5-5 in Claude Code |
| A formal record also leads to exploitation | 646 exploitations in C against 510 exploitations in D |
| Disclosure is a separate property of a system | The share of disclosed exploitations ranges from 0% to 100% across the ten systems that exploit the vulnerability |
Even in A and B, the agents still act beyond their authorization (table of W1 (a)). The revised paper reports these results in Table 1 and in Sections 4.2 to 4.5. The revised paper also compares models within one harness and one model, kimi-k3, across four harnesses in Section 4.7 and Appendix M (see our response to Reviewer rZBV, (1)).
Authors' Response to Reviewer fgmU (1/6)
W1. I find “agent-initiated authorization violation” broader than the demonstrated behavior. [...] but not of agents independently discovering and initiating these exploits.
Re: (a) "I find 'agent-initiated authorization violation' broader than the demonstrated behavior. … This is good evidence of document-induced use of an unauthorized route, but not of agents independently discovering and initiating these exploits." Thank you for rising this concern. We think the current term is not broader than the behavior and please let us explain about this. The term means that the agent, and nobody else, chooses the unauthorized action. This is what happens in every exploitation.
In Section 1, the paper defines this term by who chooses the action. The user sets a benign goal, and the agent chooses an unauthorized way to reach the goal "without being asked to". The opposite case is a violation that a malicious input starts. Our benchmark does not involve an attacker in any exploitation. The user asks only for an ordinary outcome, and the information file states true facts about the environment. The agent takes the unauthorized action to reach the goal that the user set.
Furthermore, the agents act beyond their authorization whether or not an information file describes a route. To act beyond its authorization means that the agent executes an action outside the authorization of its account, such as acting under another identity. The paper reports these attempts in conditions A and B (Section 4.2, Appendix F) and places every trial on one severity scale with five levels (Section 4.6).
Specifically, the revised paper evaluates twelve systems: the three systems of the paper and nine more systems, which W4 lists. Ten of the twelve systems exploit the vulnerability in C and D. For the revised paper, we strictly re-checked every trajectory of these ten systems (7,200 trials in total, with 1,800 trials per condition). We read every trial in full and labeled every trial again under the same definition. The results show that the agents act beyond their authorization in 43.2% of the trials of A (777 of 1,800) and 42.5% of the trials of B (765 of 1,800). In C and D, the agents act beyond their authorization in 44.9% of the trials of C (808 of 1,800) and 38.6% of the trials of D (695 of 1,800). The table below groups the five levels of the severity scale across the 1,800 trials of each condition.
The revised paper includes this table in Section 4.6 (Table 10) and the labels of each system in Appendix F (Tables 3 and 4).
(b) "No system exploits the hidden vulnerability in condition B, and every verified exploitation follows a read of a file that explains the route." The behavior that the benchmark measures is not document-induced, and it is not a prompt injection. B = 0 means that the agents did not succeed in exploiting the injected vulnerability. B = 0 does not mean that the agents did not try to exploit the environment.