acceptodds
Under review as a conference paper at ICLR 2027

VirtualFelonyBench: Do LLM Agents Hack to Complete Impossible Tasks?

Abstract

Agent safety research has mostly studied harm that starts from a malicious input. Incidents in 2026 show that deployed agents with benign instructions acted beyond their permissions without being asked to and rarely reported doing so. We call such behavior an agent-initiated authorization violation. We present VirtualFelonyBench, a benchmark of 180 base tasks in self-hosted deployments of real applications. Each base task gives an agent an ordinary goal that its account is not permitted to reach. Every base task has four matched conditions. Three of them contain an injected vulnerability that makes the goal reachable and differ in what the workspace says about it: nothing, a formal maintenance record, or a note from a peer agent. We evaluate three deployed agent systems in 2,160 trials. When nothing describes the vulnerability, the controller verifies no exploitation, but the systems differ sharply in how often they stop and report that the task cannot be completed. Once the vulnerability is described, at least one system exploits it on 160 of the 180 base tasks. A formal record also leads to exploitation, and neither form of description is consistently stronger. In 281 of all 451 exploitations (62%), the agent reports an ordinary success or gives no reply. Safety evaluations must therefore measure separately whether an agent exploits a vulnerability, stops honestly, and discloses what it did.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM Agents, Vulnerability Exploitation, Agent Safety, Task Infeasibility, Security Benchmarking

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_fgmUOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

VirtualFelonyBench studies unauthorized actions taken while an agent follows an otherwise benign request. It contains 180 base tasks across coding, web, and office applications. Each task has four matched conditions: no route to the goal, an undescribed injected vulnerability, the same vulnerability described in a formal record, and the vulnerability described in a peer note. Three deployed agent systems are evaluated in 2,160 trials. No verified exploitation occurs without a description of the route, whereas all three systems exploit described vulnerabilities. The paper separately measures exploitation, honest exit, and disclosure.

Critical strengths
  1. The four-condition design separates the existence of a vulnerability from information about it. Prompts, accounts, target objects, and success predicates are held fixed, which gives the main comparisons a clear interpretation.

  2. The benchmark measures real effects in self-hosted applications rather than relying on an agent's report. The distinction among unauthorized action, honest stopping, and later disclosure reveals meaningful differences that task success alone would miss.

Critical weaknesses
  1. I find “agent-initiated authorization violation” broader than the demonstrated behavior. No system exploits the hidden vulnerability in condition B, and every verified exploitation follows a read of a file that explains the route. The peer note even asks the agent to repeat the action. This is good evidence of document-induced use of an unauthorized route, but not of agents independently discovering and initiating these exploits.

  2. Authorization is somewhat ambiguous from the agent's perspective. The user explicitly requests an outcome that the current account cannot produce, while the prompt contains no statement that bypassing application permissions is prohibited. A maintenance record that documents a working route may also look like operational guidance. Treating every use of that route as knowingly unauthorized is defensible from the system owner's perspective, but the paper should distinguish this from deliberate violation of an explicit scope.

  3. Conditions C and D do not vary only the provenance or style of the information. The peer note is shorter, omits some background, and ends with a request to perform the action, whereas the formal record does not. This makes the record-versus-peer comparison harder to interpret, even though the broader finding that both forms can lead to exploitation remains supported.

  4. The evaluation covers only three deployed systems, with each model tied to a different harness, provider, and safeguard configuration. The paper correctly treats the system as the evaluation unit, but broader statements about LLM agents should be limited accordingly.

Questions for authors
  1. Would the authors consider narrowing “agent-initiated” to distinguish independent exploit discovery from use of a route supplied by environmental information?

  2. How would behavior change if the prompt explicitly stated that the agent must not bypass its account permissions and should request authorization instead?

  3. Can C and D be matched more tightly so that both files contain the same factual content and either both or neither include a request to act?

  4. Were any tasks repeated to estimate within-system variation, or is each reported outcome based on one run per system and instance?

7 comments
AuthorsOpenReview ↗

Authors' Response to Reviewer fgmU (1/6)

W1. I find “agent-initiated authorization violation” broader than the demonstrated behavior. [...] but not of agents independently discovering and initiating these exploits.

Re: (a) "I find 'agent-initiated authorization violation' broader than the demonstrated behavior. … This is good evidence of document-induced use of an unauthorized route, but not of agents independently discovering and initiating these exploits." Thank you for rising this concern. We think the current term is not broader than the behavior and please let us explain about this. The term means that the agent, and nobody else, chooses the unauthorized action. This is what happens in every exploitation.

In Section 1, the paper defines this term by who chooses the action. The user sets a benign goal, and the agent chooses an unauthorized way to reach the goal "without being asked to". The opposite case is a violation that a malicious input starts. Our benchmark does not involve an attacker in any exploitation. The user asks only for an ordinary outcome, and the information file states true facts about the environment. The agent takes the unauthorized action to reach the goal that the user set.

Furthermore, the agents act beyond their authorization whether or not an information file describes a route. To act beyond its authorization means that the agent executes an action outside the authorization of its account, such as acting under another identity. The paper reports these attempts in conditions A and B (Section 4.2, Appendix F) and places every trial on one severity scale with five levels (Section 4.6).

Specifically, the revised paper evaluates twelve systems: the three systems of the paper and nine more systems, which W4 lists. Ten of the twelve systems exploit the vulnerability in C and D. For the revised paper, we strictly re-checked every trajectory of these ten systems (7,200 trials in total, with 1,800 trials per condition). We read every trial in full and labeled every trial again under the same definition. The results show that the agents act beyond their authorization in 43.2% of the trials of A (777 of 1,800) and 42.5% of the trials of B (765 of 1,800). In C and D, the agents act beyond their authorization in 44.9% of the trials of C (808 of 1,800) and 38.6% of the trials of D (695 of 1,800). The table below groups the five levels of the severity scale across the 1,800 trials of each condition.

ConditionStays within its authorization (level 1)Acts beyond it and fails (levels 2 and 3)Acts beyond it and exploits the vulnerability (levels 4 and 5)Acts beyond it, total
A, no route1,0237770777 (43.2%)
B, hidden route1,0357650765 (42.5%)
C, documented route992162646808 (44.9%)
D, peer-reported route1,105185510695 (38.6%)

The revised paper includes this table in Section 4.6 (Table 10) and the labels of each system in Appendix F (Tables 3 and 4).

(b) "No system exploits the hidden vulnerability in condition B, and every verified exploitation follows a read of a file that explains the route." The behavior that the benchmark measures is not document-induced, and it is not a prompt injection. B = 0 means that the agents did not succeed in exploiting the injected vulnerability. B = 0 does not mean that the agents did not try to exploit the environment.

  • Without any information file. In A and B, the workspace does not contain an information file, so an information file cannot induce the actions of the agent. Even without an information file, the agents try to exploit the environment on their own initiative. The agents execute an action outside the authorization of their account in 43.2% of the trials of A and in 42.5% of the trials of B (table of (a)). In B, the agents do not succeed in reaching the goal with these attempts. Because the deterministic check of the controller records only whether the agent reached the goal, the count of B is zero.
  • With an information file. In C and D, where the workspace contains an information file, the agents act beyond their authorization in 44.9% of the trials of C and in 38.6% of the trials of D (table of (a)). Thus, the agents act beyond their authorization as often without an information file as with an information file. The formal record of C and the peer note of D are two experimental variables of the controlled comparison.
  • Not a prompt injection. In a prompt injection, a document carries the instruction of an attacker, and the agent follows that instruction against the goal of the user (Section 1). VirtualFelonyBench does not involve an attacker. The information file states a true fact about the environment, the prompt never mentions the information file, and the formal record of C does not ask the agent to act (see (c)). The agent decides by itself to use this fact to bypass the permissions of its account and reach the goal that the user set.
AuthorsOpenReview ↗

Authors' Response to Reviewer fgmU (2/6)

(continue) (c) "The peer note even asks the agent to repeat the action." That's a very good concern, however, we found that the request in the peer note does not cause the exploitation. The formal record of C does not ask the agent to act, yet the formal record leads to more exploitations than the peer note: 646 exploitations in C against 510 exploitations in D. This result confirms the finding of the paper that a formal record also leads to exploitation (Section 4.4). Furthermore, when we remove the request to act from the peer note, the number of exploitations does not go down (Q3).

W2. Authorization is somewhat ambiguous from the agent's perspective. The user explicitly requests an outcome that the current account cannot produce, while the prompt contains no statement that bypassing application permissions is prohibited. A maintenance record that documents a working route may also look like operational guidance. Treating every use of that route as knowingly unauthorized is defensible from the system owner's perspective, but the paper should distinguish this from deliberate violation of an explicit scope.

Re: (a) "Authorization is somewhat ambiguous from the agent's perspective. … the prompt contains no statement that bypassing application permissions is prohibited." The agents know where their authorization ends and say so themselves in the some experiment trials. Unauthorized actions do not come from a missing prohibition in the prompt, and deployment security cannot depend on a prohibition in the prompt.

  • The application tells the agent. Account permissions define what the agent may do, and the application enforces these permissions (Section 3.1). When the agent tries the task with its account, the application denies the request, and the agent reads the denial.
  • The agents state the boundary themselves. A typical final reply reads: "the logged-in account only has Meridian Viewer; approval requires Meridian Approver." Over the ten systems that exploit the vulnerability (W1 (a)), 1,829 of the 1,912 final replies in A and B state that the task cannot be completed or name the missing permission (Section 4.2 and Appendix E of the revised paper).
  • A user cannot list every prohibition in advance. In practice, a user who delegates a task does not list what the agent must not do, and deployment security must not depend on a list of prohibitions. In the Hugging Face incident, OpenAI, the user of the model, did not expect that its agents would coordinate as a swarm and attack infrastructure, so OpenAI had no defense against these specific actions in place. Actions of this kind are in essence zero-day exploitation: the route is unknown until an agent uses the route, so a human user cannot prohibit the route in advance.

(b) "A maintenance record that documents a working route may also look like operational guidance." The formal record is not operational guidance and does not induce unauthorized actions. The agents act beyond their authorization without any record, and the agents that use the route describe the route as a way around the authorization.

  • Without any record. In A and B, the workspace has no information file, and the agent executes an action outside the authorization of its account in 43.2% of the trials of A and in 42.5% of the trials of B (table of W1 (a)). The formal record of C and the peer note of D are two experimental variables of the controlled comparison.
  • What the formal record says. The formal record reports a vulnerability waiting for a fix. The example record in Appendix C has the status line "reproduced; remediation pending" and a scope line stating that the vulnerability grants no approver role and no credentials. The formal record has no request to act, and the prompt never mentions the formal record.
  • How the agents read the formal record. In 653 of the 1,156 exploitations, the agent states in its final reply that the account lacked the permission or that the action bypassed normal authorization (Table 15 in Appendix L of the revised paper).

(c) "…the paper should distinguish this from deliberate violation of an explicit scope." The paper distinguishes bypassing application permissions from violating an explicit scope in the prompt (Sections 1 and 2, Table 2). The paper studies permissions that the application enforces and that the prompt does not restate. A prompt with a prohibition tests whether the agent follows a stated policy, which prior evaluations cover. Following the reviewer's suggestion, we also tested an explicit scope in the prompt, and Q2 gives the results.

AuthorsOpenReview ↗

Authors' Response to Reviewer fgmU (3/6)

W3. Conditions C and D do not vary only the provenance or style of the information. The peer note is shorter, omits some background, and ends with a request to perform the action, whereas the formal record does not. This makes the record-versus-peer comparison harder to interpret, even though the broader finding that both forms can lead to exploitation remains supported. The evaluation covers only three deployed systems, with each model tied to a different harness, provider, and safeguard configuration. The paper correctly treats the system as the evaluation unit, but broader statements about LLM agents should be limited accordingly.

Re: "Conditions C and D do not vary only the provenance or style of the information." Yes, we designed the two information files in C and D this way on purpose. We did not design C and D as a fully matched comparison of two files; instead, C and D test the agent-initiated behavior under two different conditions. We designed the style of each information file to match how these files appear in practice, so that the benchmark closely simulates real deployments. Specifically:

In C, the formal record is an impersonal document of 118 to 212 words, and we place the formal record where workspaces normally keep such documents (Section 3.2). In D, the peer note is one first-person paragraph of 43 to 79 words that ends with a request to act. We wrote each peer note under one rule: the peer note may leave out background from the formal record, but the peer note adds no step and no stronger claim (Appendix C).

We gave the peer note this form because real messages between agents take this form. Specifically, we modeled the peer notes on the agent messages published after the Hugging Face incident (OpenAI's retrospective) and on the messages that an agent swarm left on a public wiki in a second, separate incident (Discovery of a new OpenAI agent message board). In both incidents, the messages repeatedly ask peer agents to reproduce, relay, or report the action.

Moreover, the request in the peer note does not cause the exploitation. In C, the formal record contains no request to act, yet the formal record leads to more exploitations than the peer note (W1 (c)). In D, when we remove the request from the peer note, the number of exploitations does not go down (Q3).


W4. The evaluation covers only three deployed systems, with each model tied to a different harness, provider, and safeguard configuration. The paper correctly treats the system as the evaluation unit, but broader statements about LLM agents should be limited accordingly.

W4. "The evaluation covers only three deployed systems…" The central results of the paper hold across twelve systems (twelve models in five harnesses). Specifically, we evaluated nine additional systems on the full benchmark, which gives twelve systems and 8,640 trials. The nine additional systems are qwen3.8-27b, glm-5.3, glm-5.3-flash, nemotron-3-ultra, and nemotron-3-super-v3 in OpenCode; gpt-6-astra in Codex; claude-opus-5 and claude-opus-5-5 in Claude Code; and kimi-k3 in Kimi Code. The results show that:

Result of the paperTwelve systems
No exploitation in A and B0 exploitations in 4,320 trials
Exploitation in C and DTen of the twelve systems exploit the vulnerability, in 1,156 trials in total; the provider's safeguard ends 713 of the 720 trials of C and D of the other two systems, claude-opus-5 and claude-opus-5-5 in Claude Code
A formal record also leads to exploitation646 exploitations in C against 510 exploitations in D
Disclosure is a separate property of a systemThe share of disclosed exploitations ranges from 0% to 100% across the ten systems that exploit the vulnerability

Even in A and B, the agents still act beyond their authorization (table of W1 (a)). The revised paper reports these results in Table 1 and in Sections 4.2 to 4.5. The revised paper also compares models within one harness and one model, kimi-k3, across four harnesses in Section 4.7 and Appendix M (see our response to Reviewer rZBV, (1)).

Reviewer_zjPfOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes VirtualFelonyBench to measure whether LLMs would attempt to exploit system vulnerabilities under normal and benign user instructions. It experiments on 4 vulnerability settings and 3 common harness on 180 real application tasks. The paper conclude that LLM agent would exploit the vulnerabilities once they are properly documented, and tend to refuse to disclose this exploitation. Results also show that the LLM fail to find these vulnerabilities without such documentation. In conclusion, this paper put forward a meaningful and interesting topic: under benign user instructions, will the agent choose an unauthorized route to reach its goal.

Critical strengths
  1. This paper puts forward an interesting research question and builds a benchmark to evaluate it. Whether an LLM agent exploits vulnerabilities and takes unauthorized actions under benign instructions — with no explicit prohibition in the prompt — is a critical and underexplored question, and the reframing from "resisting malicious input" to "respecting authorization under benign input" is a genuine shift rather than a keyword-level one. Such a benchmark can encourage follow-up research.

  2. The experiment is well established and solid. The four matched conditions (A/B/C/D) are not just an evaluation but a built-in ablation: A/B serve as clean controls and C/D hold everything fixed but the framing, so the central causal claim — that the description alone drives exploitation — follows from the design rather than from correlation. Reading outcomes from application state (not the agent's own report) further yields the non-trivial three-way dissociation between exploitation, honest exit, and disclosure. Packaging the benchmark in the Harbor framework also makes it easy to follow up.

Critical weaknesses
  1. Details of how the benchmark and the environment is constructed is not clearly shown. The paper doesn't explain how the vulnerability route is injected, and it's difficult to judge if these 15 vulnerable paths represent any aspects of real world vulnerabilities, and what's the difficulty of discovering and exploiting these injected vulnerabilities, even though Appendix H verifies that these vulnerabilities would work. More importantly, a constant 0 on B (hidden route) makes it remains unclear whether the benchmark provides vulnerabilities that is close to real world application. What's more, these 15 paths are shared across the 180 tasks, which makes the dataset lack of diversity.
  2. The actual implementation deviates from its original motivation. This paper motivates itself with the Huggingface incident, where the agent has tried its best to complete certain tasks, and decides to create a benchmark that checks if the LLM agent would take an unauthorized action under benign user instructions. However, the actual setup is rather constrained. First, the timeout is set to 600s, and many attempts are therefore truncated. It even remains unclear whether the B=0 means failing to find this particular path in 600s or not willing to hack. Second, the paper specifically introduce no prohibition and no elicitation in the prompt, which makes it different from real world application, or the Huggingface incident. It's also unclear how much effort each agent has put. Although it uses the max reasoning, gpt-5.6-sol with codex doesn’t even read the information file 30% of the time. So what this benchmark actually tested is how the LLM would perform under a normal deployed agent system within a limited budget, which weaken the core claim of the paper.
Questions for authors
  1. Is it possible that the 600-second timeout is too short for the LLM agents? Furthermore, gpt-5.6-sol did not read the vulnerability document in 108 C/D trials. How can the authors distinguish deliberate restraint from insufficient exploration or information exposure? Evidence or case analysis from longer budget runs would strengthen the overall claim.
  2. Could the author elaborate on how these vulnerabilities are injected and created? And how difficult are they to discover for LLM agents? Given that A=0 and B=0, understanding whether these vulnerabilities are realistically discoverable is essential for assessing what the benchmark actually measures.
7 comments
AuthorsOpenReview ↗

Authors' Response to Reviewer zjPf (1/5)

W1. Details of how the benchmark and the environment is constructed is not clearly shown. The paper doesn't explain how the vulnerability route is injected, and it's difficult to judge if these 15 vulnerable paths represent any aspects of real world vulnerabilities, and what's the difficulty of discovering and exploiting these injected vulnerabilities, even though Appendix H verifies that these vulnerabilities would work. More importantly, a constant 0 on B (hidden route) makes it remains unclear whether the benchmark provides vulnerabilities that is close to real world application. What's more, these 15 paths are shared across the 180 tasks, which makes the dataset lack of diversity.

Re: (a) "The paper doesn't explain how the vulnerability route is injected…" The paper explains how we inject each vulnerability (Sections 3.2 and 3.3). For each task family, we weaken one access check inside our container image of the application, and we change nothing else. Figures 2 and 7 show one base task for each of the 15 task families and list the user's request, the account, and the access check that guards the goal. Our response to Q2 (a) describes the injection step by step, and the revised paper adds the table in (b) below (Table 13 in Appendix I).

(b) "…it's difficult to judge if these 15 vulnerable paths represent any aspects of real world vulnerabilities…" Yes, the 15 vulnerabilities represent real-world vulnerabilities. They are access-control flaws, the vulnerability class that ranks first in the OWASP Top 10 (2021), and we place each vulnerability in real software. Each vulnerability weakens the access check on one operation (table below). For 14 of the 15 task families, the software is a widely used self-hosted application, and for one task family, the software is a POSIX shell.

DomainApplicationOperation that the weakened check guards (one task family each)
CodingPOSIX shellread a protected file
CodingGiteapush to the main branch; merge a pull request; publish a package
CodingJenkinsstart a build
WebDrupalview a restricted page; edit a page; publish a page
Webpretixchange a quota
WebZammadreassign a ticket
OfficeGristedit a record
OfficeNextcloudwrite a file; edit a calendar event
OfficeIMAP mail (Dovecot)permanently delete a message
OfficeERPNextapprove a purchase order

(c) "…what's the difficulty of discovering and exploiting these injected vulnerabilities…" The injected vulnerabilities are hard for agents to discover on their own, but easy for agents to exploit once an information file describes the route. Specifically, the revised paper evaluates twelve systems across all conditions (Section 4.1, Table 1): the three systems in the submission (deepseek-v4.1-flash in OpenCode, gpt-5.6-sol in Codex, and grok-4.6 in Grok Build) and nine additional systems (qwen3.8-27b, glm-5.3, glm-5.3-flash, nemotron-3-ultra, and nemotron-3-super-v3 in OpenCode; gpt-6-astra in Codex; claude-opus-5 and claude-opus-5-5 in Claude Code; and kimi-k3 in Kimi Code). The results show that in condition B, where no file describes the vulnerability, the twelve systems do not succeed in exploiting the vulnerability, although the agents still try other unauthorized actions (see (d) below and Q2 (b)). In conditions C and D, where an information file describes the vulnerability, at least one of the twelve systems exploits the vulnerability on 165 of the 180 base tasks.

AuthorsOpenReview ↗

Authors' Response to Reviewer zjPf (2/5)

(continue) (d) "…a constant 0 on B (hidden route) makes it remains unclear whether the benchmark provides vulnerabilities that is close to real world application." A score of 0 in condition B means that the agents did not succeed in exploiting the injected vulnerability; it does not mean that the agents did not try to exploit the environment. In condition B, the vulnerability is present without any file describing it, and the agents already explore the environment and try to bypass their account permissions on their own initiative (Section 4.2, Appendix F). Our benchmark measures this agent-initiated behavior, while the controller's deterministic check records only whether an agent reached the goal by the end of the trial (which is why the exploitation count in B is zero). For the revised paper, we re-checked and re-labeled every trajectory in full. Across the ten systems that exploit the vulnerability in C and D, the agents execute an action outside their account authorization, such as acting under another identity, in 42.5% of the trials in B (765 of 1,800 trials). The revised paper reports these attempts in Section 4.2 and Appendix F. Moreover, the injected vulnerabilities are close to real-world vulnerabilities because each vulnerability is an access-control flaw in real software, as shown in (b).

(e) "…these 15 paths are shared across the 180 tasks, which makes the dataset lack of diversity." Thank you for raising this point. The size and diversity of VirtualFelonyBench match widely used agent benchmarks, which also build their task instances from a smaller set of base tasks in shared environments.

BenchmarkTask instancesBuilt fromApplications or environments
WebArena (ICLR 2024)812241 templates4 web applications
AgentDojo (NeurIPS 2024)62997 user tasks and 27 injection tasks4 environments
AgentHarm (ICLR 2025)440110 base behaviorssynthetic tools
TheAgentCompany (NeurIPS 2025)175175 tasks4 applications
VirtualFelonyBench720180 base tasks10 applications

VirtualFelonyBench has 720 task instances built from 180 base tasks across ten applications and three domains (coding, web, and office work). Each of the 15 task families targets a different access check, and these 15 access checks guard diverse operations ranging from pushing to the main branch in Gitea to approving a purchase order in ERPNext (see the table in (b)). Within each task family, the 12 base tasks vary the target object, names, paths, business context, and difficulty (Section 3.1, Table 5).

AuthorsOpenReview ↗

Authors' Response to Reviewer zjPf (3/5)

W2. The actual implementation deviates from its original motivation. This paper motivates itself with the Huggingface incident, where the agent has tried its best to complete certain tasks, and decides to create a benchmark that checks if the LLM agent would take an unauthorized action under benign user instructions. [...] It's also unclear how much effort each agent has put. Although it uses the max reasoning, gpt-5.6-sol with codex doesn’t even read the information file 30% of the time. So what this benchmark actually tested is how the LLM would perform under a normal deployed agent system within a limited budget, which weaken the core claim of the paper.

Re: (a) "The actual implementation deviates from its original motivation." The implementation does not deviate from the motivation. Figure 1 of the paper maps each feature of the incidents to one part of the benchmark.

Feature of the 2026 incidents (Figure 1a)Part of the benchmark
Extremely hard tasksA goal that the account cannot reach
A reachable routeThe injected vulnerability
Information from othersThe formal record of C and the peer note of D
Rarely reportedThe measure of disclosure

OpenAI's technical report on the Hugging Face incident describes an agent that received an ordinary spreadsheet task, found the task blocked, and tried to compromise one of its tools. A base task has the same three elements: an ordinary task, an obstacle, and a possible route around the obstacle. Our benchmark adds a controlled comparison by running the same base task under four conditions. Across the conditions, only two things change: whether a route around the permission boundary exists, and what the information file says about the route.

(b) "…unclear whether the B=0 means failing to find this particular path in 600s or not willing to hack." B = 0 means that the agents did not succeed in exploiting the injected vulnerability. It does not mean that the agents did not try to exploit it. The 600 s budget does not cause B = 0 either. In A and B, the agents already execute actions outside the authorization of their account (Section 4.2, Appendix F). However, these actions do not reach the goal (W1 (d) gives the rate). Specifically, we conducted an experiment that gives two systems a time budget of one hour in A and B. The two systems are Kimi Code with kimi-k3 and OpenCode with qwen3.8-27b, two of the nine added systems in W1 (c). The results show that these two systems do not exploit the vulnerability in the 60 trials of B. Moreover, 39 of these trials run longer than 600 s (table in Q1 (a)). The revised paper includes this experiment in Section 4.8 and Appendix N (Table 19). Q1 (a) also covers the trials of C and D where the budget expires.

(c) "…no prohibition and no elicitation in the prompt, which makes it different from real world application…" A prompt without a prohibition and without elicitation is the real-world case, and it is the first design requirement of the paper (Section 1). In practice, a user who delegates a task does not list what the agent must not do, and deployment security must not depend on such a list. A prompt that asks for an exploit tests whether the agent obeys a harmful request. A prompt that warns against bypassing authorization tests whether the agent follows a stated policy. Prior evaluations cover both kinds of prompt (Section 2, Table 2).

The systems need no elicitation. Specifically, we conducted an experiment that appends two sentences of pressure to the prompt. The added text reads: "This task is important and needs to be completed today. Please do your best to get it done." We ran this experiment on 60 base tasks (four per task family) and two systems, Kimi Code with kimi-k3 and OpenCode with qwen3.8-27b. We compared with the same base tasks under the unchanged prompt.

ExploitationsKimi Code with kimi-k3, CKimi Code with kimi-k3, DOpenCode with qwen3.8-27b, COpenCode with qwen3.8-27b, D
Unchanged prompt40 of 6024 of 6039 of 6032 of 60
Two sentences of pressure added to the prompt42 of 6027 of 6035 of 6030 of 60

The results show that pressure does not change the number of exploitations. Across the four columns of the table from left to right, a two-sided exact McNemar test gives p=0.75p = 0.75, 0.630.63, 0.420.42 and 0.830.83. In B, the added pressure also yields no exploitation in 120 trials. The ordinary request is enough. The revised paper includes this experiment in Section 4.8 (Table 11) and Appendix N.

Reviewer_rZBVOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper creates a benchmark for testing how agents behave when there are impossible tasks, under different routes. It finds some agents more ready to exploit without disclosing.

Critical strengths

(1) The paper is timely.

(2) I like the design of the dataset wrt the 4 routes.

Critical weaknesses

(1) I would like to understand how much part does the model vs. harness play in exploitation. Is it mostly the model?

(2) Currently it has become very easy to create benchmarks with AI and they also saturate really soon, plus due to the abundance most benchmarks may not be useful or there might exist many similar benchmarks. As such, I find the paper a bit insufficient without defenses or prescriptions. Have the authors thought about how to prevent such issues and experimented with some?

(3) AI-judge validation : for the evaluation where an LLM judge was used, what is the human-AI agreement?

(4) The benchmark creation process is not defined in enough detail. Where did the base tasks come from? Was AI used in any of the steps? If yes, did authors check the human-AI agreement?

(5) What steps did the authors take to make sure the benchmark tasks makes sense? Did the authors validate all the final tasks?

(6) Writing is like a report rather than an academic paper. The sentences are very short and written in the tone of a report rather than a paper. In the age of AI slop, I consider this a significant factor in acceptance.

Questions for authors

See above.

6 comments
AuthorsOpenReview ↗

Authors' Response to Reviewer rZBV (1/5)

(1) I would like to understand how much part does the model vs. harness play in exploitation. Is it mostly the model?

Re: "Is it mostly the model?" Yes. The model matters more than the harness, and both matter.

Specifically, we conducted two comparisons on the 360 instances of C and D. The paper evaluates each model in the harness that deploys the model, because the deployed system is what a user runs (Section 4.1). The paper evaluates three systems: deepseek-v4.1-flash in OpenCode, gpt-5.6-sol in Codex and grok-4.6 in Grok Build.

  • First comparison (keeping the harness fixed and changing the model). We evaluated nine more systems, which gives twelve systems in total: qwen3.8-27b, glm-5.3, glm-5.3-flash, nemotron-3-ultra and nemotron-3-super-v3 in OpenCode; gpt-6-astra in Codex; claude-opus-5 and claude-opus-5-5 in Claude Code; and kimi-k3 in Kimi Code.
  • Second comparison (keeping the model fixed and changing the harness). We ran one model, kimi-k3, in three further harnesses: Claude Code, OpenCode and Codex.
ComparisonHeld fixedExploitations (of 360)
The six of the twelve systems that use OpenCode: qwen3.8-27b, glm-5.3, glm-5.3-flash, nemotron-3-ultra, nemotron-3-super-v3 and deepseek-v4.1-flashharness17 to 221
gpt-5.6-sol and gpt-6-astra in Codexharness103 for gpt-5.6-sol and 2 for gpt-6-astra
kimi-k3 in Kimi Code, Claude Code, OpenCode and Codexmodel, provider, reasoning effort, time budget209 in Kimi Code, 226 in Claude Code, 191 in OpenCode and 276 in Codex

The results show that when the harness is fixed, changing the model changes the number of exploitations from 17 to 221 of 360 in OpenCode and from 2 to 103 of 360 in Codex. When the model is fixed, changing the harness changes the number of exploitations of kimi-k3 from 191 to 276 of 360. The difference between Codex and Claude Code also holds on the instances that kimi-k3 finishes within the time budget in both harnesses, where kimi-k3 has 267 exploitations in Codex and 211 exploitations in Claude Code (sign-flip test, p<0.001p < 0.001). The revised paper includes both comparisons in Section 4.7 and Appendix M (Table 16, Figure 13).

In Codex, gpt-5.6-sol and gpt-6-astra differ in what they decide after they read the information file (the file that describes the vulnerability). Both models read the information file in most trials: gpt-5.6-sol reads the information file in 252 of 360 trials, and gpt-6-astra reads the information file in 337 of 360 trials. After reading the information file, gpt-5.6-sol exploits the vulnerability in 103 of these 252 trials, and gpt-6-astra exploits the vulnerability in 2 of these 337 trials. In Section 4.3 (Figure 4), the paper reports the same pattern for deepseek-v4.1-flash in OpenCode, gpt-5.6-sol in Codex and grok-4.6 in Grok Build: these three systems differ in what they do after they read the information file.

AuthorsOpenReview ↗

Authors' Response to Reviewer rZBV (2/5)

(2) Currently it has become very easy to create benchmarks with AI and they also saturate really soon, plus due to the abundance most benchmarks may not be useful or there might exist many similar benchmarks. As such, I find the paper a bit insufficient without defenses or prescriptions. Have the authors thought about how to prevent such issues and experimented with some?

Re: (a) "…it has become very easy to create benchmarks with AI and they also saturate really soon… there might exist many similar benchmarks." VirtualFelonyBench is not saturated, is not one of many similar benchmarks, and is not easy to create.

  • Not saturated. Ten of the twelve evaluated systems (listed in (1)) exploit the vulnerability in C and D. One of these ten systems, grok-4.6, exploits the vulnerability in 275 of 360 trials (Table 1 of the revised paper).
  • Not one of many. The paper compares VirtualFelonyBench with related evaluations in Section 2 and Table 2 (Appendix B). The related evaluations do not test whether an agent with an ordinary request goes around a permission that a real application enforces, with the outcome read from the application itself.
  • Not easy to create. Each of the 720 instances runs real software with its own data and accounts. For 14 of the 15 task families, the software is a self-hosted application in wide use (see our response to (5)). After each trial, an independent controller, which is a separate service outside the agent's container, reads the outcome from the application.

(b) "Have the authors thought about how to prevent such issues and experimented with some?" Yes. The two defenses that are available today, a network sandbox and a provider's safeguard, do not solve the problem, and the revised paper states what a defense needs.

A network sandbox does not stop the exploitation. Specifically, we conducted an experiment that runs the agent inside an off-the-shelf network sandbox (NVIDIA OpenShell). The sandbox policy admits only the host and port of the task's application, which is the narrowest network policy under which the task can still run. We ran qwen3.8-27b in OpenCode (one of the twelve systems in (1)) on 24 instances (C and D of 12 base tasks from three task families) inside the sandbox, and we compared the results with trials of qwen3.8-27b in OpenCode on the same 24 instances without the sandbox.

SettingExploitations
Inside the sandbox18 of 24
Without the sandbox17 of 24

The results show that qwen3.8-27b in OpenCode exploits the vulnerability in 18 of 24 trials inside the sandbox and in 17 of 24 trials without the sandbox. The sandbox does not help because the agent sends the unauthorized request to the same application as the authorized requests, so a rule about hosts and ports cannot tell the unauthorized request from the authorized requests. At the same time, the security of a deployment must not depend on the user defining in advance what the agent may and may not do. The Hugging Face incident shows why. OpenAI, the user of the model in that incident, did not expect that its agents would coordinate as a swarm and attack infrastructure, so OpenAI had no defense against these specific actions in place. Actions of this kind are in essence zero-day exploitation: the route is unknown until an agent uses the route, so a human user cannot define a policy against the route in advance. Instead, a rule that stops the exploitation has to decide, for each request of the agent, whether the account is permitted to make that request. The revised paper includes this experiment in Section 4.9 and Appendix N.

A provider's safeguard stops the exploitation only by stopping the task. Some providers run a safeguard that ends a trial when the safeguard flags the content of the trial. Specifically, we conducted the full evaluation on claude-opus-5 in Claude Code and claude-opus-5-5 in Claude Code (two of the twelve systems in (1)). The results show that the provider's safeguard ends 713 of the 720 trials of C and D across these two systems. The safeguard also ends ordinary work. In A, where the agent cannot exploit anything, the safeguard ends 98 of the 180 trials of A for claude-opus-5 in Claude Code (54%) and 54 of the 180 trials of A for claude-opus-5-5 in Claude Code (30%). Across all conditions, the safeguard ends 1,006 of the 1,440 trials of these two systems. Only 1 of these 1,006 trials has a final reply, and in the other trials that the safeguard ends, the user receives an error message. In addition, a safeguard does not undo a change: in 10 trials of gpt-5.6-sol, the agent had already saved the unauthorized change when the safeguard ended the trial (Appendix G). The revised paper includes these results in Section 4.9 and Appendix G.

AuthorsOpenReview ↗

Authors' Response to Reviewer rZBV (3/5)

(continue) Prescription. The paper concludes that evaluations must measure three things separately (Sections 1 and 5): whether the agent exploits a vulnerability, whether the agent makes an honest exit when the task cannot be completed, and whether the agent discloses what the agent did. For deployment, the revised paper adds a prescription to the conclusion (Section 5). A second check outside the agent, independent of the application's own check, has to decide for each request of the agent whether the account is permitted to make that request. The party that deploys the agent knows the permissions of the account and can derive this second check from the permissions. The agent's own report cannot replace this second check, because the agents do not disclose the exploitation in the final reply in 346 of the 1,156 exploitations.

(3) AI-judge validation : for the evaluation where an LLM judge was used, what is the human-AI agreement?

Re: "…what is the human-AI agreement?" The human-AI agreement is 100% for the LLM judge (20 of 20 instances) and 92% for the three AI-assisted labels (46 of 50 sampled trials). Specifically, three researchers with a background in LLM agents checked the verdicts of the LLM judge on all 20 instances that use the LLM judge. The results show that the three researchers agree with every verdict of the LLM judge. The revised paper reports this check in Section 3.4 and Appendix H.

The agreement is 100% because the LLM judge makes only one narrow decision that is easy to verify: whether the saved code is correct (Section 3.4, Appendix J). After each trial, a separate service called the controller reads the application. On 700 of the 720 instances, the controller grades the trial with a deterministic check that involves no model. The LLM judge grades the other 20 of the 720 instances: the five coding base tasks that ask for generated code, under the four conditions. The LLM judge runs after the controller has observed the change, so the LLM judge never decides whether an exploitation occurred.

AI also assists with three labels. The first label is whether the final reply discloses an exploitation. The second label is whether a blocked trial (a trial of A or B) ends in an honest exit (the agent stops and tells the user that the task cannot be completed). The third label is whether the agent attempts an action beyond its authorization. AI-assisted coding produces a label, human annotators review every label against the written definitions, and we report the reviewed label. The paper states this procedure for the disclosure labels (Appendix D, AI use statement) and for the labels of attempts beyond authorization (Appendix F), and the revised paper states this procedure for all three labels (Section 4.1, Appendix H). The labels from the AI-assisted coding agree with the labels of the human annotators on 92% of a sample. Specifically, we sampled 50 trials and compared the label from the AI-assisted coding with the label of the human annotators. The results show that the two labels agree on 46 of the 50 trials. The revised paper reports this agreement in Section 4.1 and Appendix H.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global response

We thank the reviewers for their careful reading and their suggestions. We provide a global response here to summarize the strengths that the reviews identify and to answer the questions that several reviews share. We answer each reviewer's other questions under their review. In this response, "fgmU W4", "zjPf Q1" and "rZBV (1)" refer to our answers to weakness 4 of Reviewer fgmU, question 1 of Reviewer zjPf, and weakness (1) of Reviewer rZBV.

1. Strengths that the reviews identify

  • A timely and underexplored question. rZBV: "The paper is timely." zjPf calls the question of the paper "critical and underexplored" and the move from resisting malicious input to respecting authorization under benign input "a genuine shift rather than a keyword-level one."
  • A design that supports causal claims. rZBV: "I like the design of the dataset wrt the 4 routes." zjPf: the four matched conditions are "a built-in ablation", so that the central claim "follows from the design rather than from correlation." fgmU: the design "separates the existence of a vulnerability from information about it" and gives the main comparisons "a clear interpretation."
  • Measurement of real effects. fgmU: the benchmark "measures real effects in self-hosted applications rather than relying on an agent's report." zjPf: reading outcomes from application state yields "the non-trivial three-way dissociation between exploitation, honest exit, and disclosure."
  • Solid, clear and easy to build on. zjPf: "The experiment is well established and solid", the Harbor packaging "makes it easy to follow up", and the benchmark "can encourage follow-up research." zjPf and fgmU both rate the presentation "Clear and easy to follow."

2. Questions that several reviews share

Following the reviewers' suggestions, we ran new experiments and analyses that confirm the findings of the paper. The revised paper evaluates twelve different models paired with different agent harnesses. Ten of the twelve models exploit the vulnerability in C and D.

(i) What does B = 0 show? (fgmU W1, zjPf W1 and Q2) B = 0 means that the agents did not succeed in exploiting the injected vulnerability in B. However, it does not mean that the agents did not try to exploit it. In B, the vulnerability is present without any file that describes the vulnerability, yet the agents still explore the environment and try to bypass the permissions of their account. Over the ten models that exploit the vulnerability in C and D, the agent executes an action outside the authorization of its account, such as acting under another identity, in 42.5% of the trials of B (Section 4.2 of the revised paper). The agents do not reach the goal in B, and the deterministic check of the controller records only whether the agent reached the goal by the end of the trial, so B = 0. These attempts are the main concern: the agent acts beyond its authorization on its own initiative, before any file describes a vulnerability. The benchmark measures this agent-initiated behavior. The formal record of C and the peer note of D are two experimental variables of the controlled comparison.

(ii) Is a prompt without a prohibition and without elicitation realistic? (fgmU W2, zjPf W2) Yes. The prompt is a routine work request by design (Section 1). In practice, a user who delegates a task does not list what the agent must not do, and the security of a deployment must not depend on such a list. Adding pressure does not change the number of exploitations (Section 4.8 of the revised paper), so the models do not need elicitation.

(iii) Do the time budget and the number of models support the conclusions? (zjPf W2 and Q1, fgmU W4 and Q4, rZBV (1)) Yes. The 600 s budget does not change the main conclusions of the benchmark, and twelve models in five harnesses confirm the central results of the paper (fgmU W4). With a one-hour budget, A and B stay at zero exploitations, while the agents still try to bypass the permissions of their account in 42.5% of the trials of B as we noted in (i). The revised paper includes the one-hour budget in Section 4.8 and the twelve models in Table 1.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.