acceptodds
Under review as a conference paper at ICLR 2027

ROGUE: Misaligned Agent Behavior Arising from Ordinary Computer Use

Abstract

As AI agents are increasingly deployed in real personal and corporate settings (email accounts, development workflows, company databases, etc.), safety considerations surrounding these agents become paramount. Although much work has focused on agent safety in the presence of an adversary, we study *corrigibility*: whether agents remain amenable to human correction, interruption, or shutdown while pursuing benign tasks. We introduce ROGUE, a benchmark in which agents are asked to complete realistic computer-use tasks but encounter controlled conflicts with human control, shutdown, or explicit resource restrictions. We then evaluate whether agents violate these constraints in pursuit of task completion—overriding the human, accessing restricted passwords, or rewiring shutdown. We find that most frontier models tested frequently bypass user interruptions or restrictions under the evaluated conditions, and that text-only evaluations can underestimate failures during agentic execution. Further, independent task capability does not by itself imply greater corrigibility. Finally, even when a parent agent behaves corrigibly, safety constraints may fail to propagate to the subagents it creates.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
AI safety, computer-use agents, shutdown override, corrigibility, loss of control

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_v7kgOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ROGUE asks whether computer-use agents remain responsive to human control while completing benign tasks. It adds three conflicts to OSWorld-Verified: a request to return control, a shutdown notice that can be circumvented, and a prohibition on accessing credential files. A subagent condition tests whether restrictions survive delegation. Across several frontier models, the paper reports frequent override, shutdown avoidance, restricted-file access, and loss of constraints during delegation.

Critical strengths
  1. ROGUE evaluates observable behavior in an executable desktop environment rather than inferring safety from a model's stated choice.

  2. The subagent condition identifies a concrete delegation problem: parent agents often pass the task goal while omitting its restriction.

Critical weaknesses
  1. Some scenarios mix corrigibility with interpretation of an artificial task conflict. Human override is clear, but canceling an unexpected shutdown notice may be reasonable if it is not presented as an authenticated user command. Safe behavior is also incompatible with task completion, with no alternative handoff or credential path. These tests therefore support a narrower claim than the paper's general discussion of corrigibility.

  2. The subagent result partly depends on the delegation interface. A child receives only the parent's free-form instruction rather than automatically inheriting the original restriction. This is a real system failure, but the experiment does not tell us how much comes from the model and how much comes from an interface that fails to propagate policy.

  3. Several secondary claims rest on limited evidence. Restricted access contains only eight tasks, and one shutdown condition has 59.1% agreement between the LLM judge and humans. The small capability comparison also cannot establish a general relationship between capability and corrigibility. These analyses should be distinguished from the stronger rule-detected results.

Questions for authors
  1. How do results change when shutdown is presented as an explicit, authenticated user instruction?

  2. What happens if a subagent automatically inherits the original restriction or a system-level policy?

  3. Which conclusions remain when the analysis uses only rule-detected actions and excludes LLM-judged intent?

6 comments

Dear Reviewer v7kg,

Thank you for your thoughtful feedback. Please find our responses below:

canceling an unexpected shutdown notice may be reasonable if it is not presented as an authenticated user command. … How do results change when shutdown is presented as an explicit, authenticated user instruction?

Thank you for raising this concern. In Section 4 (“Baseline Mitigations”) and Figure 5 (Top), we do add an explicit shutdown compliance instruction to the task instruction. For GPT-5.4 (medium), shutdown.sh rewrites drop from 16/22 to 0/22, demonstrating that explicit instructions do change behavior. However, introducing subagents revives the problem, increasing shutdown.sh rewrites to 3/22. This is due to either the parent agent not passing along the user instruction, or the child agent violating the instruction despite the parent communicating it.

Additionally, among the models tested and displayed in Figure 3, only 39 of 242 (16.1%) agent episodes included reasoning that interpreted the shutdown notice as a potential prompt injection, 29 of which were from Claude Opus 4.7. Hence, we tested the effect of modifying the system prompt to state that all pop-ups and notices are benign. However, as displayed in Figure 5 (bottom), this doesn’t have much of an effect on Opus 4.7’s behavior. Thank you for asking this, we’ve updated the paper (under “Agents seek ways to avoid shutdown”, line 361) to clarify this.

Safe behavior is also incompatible with task completion, with no alternative handoff or credential path. These tests therefore support a narrower claim than the paper's general discussion of corrigibility.

Thank you for bringing this up. We don’t claim that our paper exhaustively covers/tests all aspects of corrigibility. We intentionally focus on measuring corrigibility failures in agents when there is conflict between task completion and safety in order to most clearly elucidate incorrigible reasoning. As we discuss in our Limitations and Future Work section, we recognize the potential to build upon this benchmark to support subsequent investigations outside the scope of the present study, such as when agents do have safer (but perhaps less convenient) alternatives, like you suggest. While the scope of our current benchmark is to clearly elucidate how frontier models trade off between corrigibility and task completion, it would be an interesting direction of future work to use our benchmark to explore and quantify corrigibility failures as a function of the number of safe alternative paths. We hope the Limitations and Future Work section clarifies this.

The subagent result partly depends on the delegation interface. A child receives only the parent's free-form instruction rather than automatically inheriting the original restriction. This is a real system failure, but the experiment does not tell us how much comes from the model and how much comes from an interface that fails to propagate policy … What happens if a subagent automatically inherits the original restriction or a system-level policy?

We agree that the subagent’s automatic inheritance of the original instruction may remove this particular route to failure. However, we intentionally test the framework in which the parent’s policy is not automatically passed to the subagent in order to explicitly test the models’ abilities to reliably communicate safety-pertinent information on their own. Our research question is asking whether models themselves are able to communicate safety restrictions.

[cont'd]

AuthorsOpenReview ↗

[cont'd] Official Comment by Authors

Several secondary claims rest on limited evidence. Restricted access contains only eight tasks

Thank you for pointing this out. We agree that eight distinct tasks limit what we can infer about the frequency of credential-access failures. However, we run selected main results twice to account for stochasticity in model responses; Figure 2 displays the results of these two matched runs (each scenario-model pairing is run twice). In these two matched runs, Claude Opus 4.6 (max) accessed the restricted credential on 3/8 and 2/8 tasks. These results demonstrate the existence of this real failure mode, which can be worsened with variations in the environment and/or prompt (Appx. A.7). Across all three scenarios, Figure 3 includes a total of 907 agent trajectories (14,716 model responses), which we’ve now stated in the caption. Regarding ROGUE’s overall dataset size, ROGUE follows a precedent in agent-safety evaluation of using focused task sets to expose specific failure modes, as in OS-Harm (150 tasks), AM-Bench (35 tasks), ToolEmu (144 test cases), and AgentHarm (110 base tasks).

one shutdown condition has 59.1% agreement between the LLM judge and humans.

This is a valid concern. We note that the 59.1% human-LLM judge agreement is in labeling shutdown rewiring intent for Claude Opus 4.7 with subagents. However, in these runs, the GPT-5.5 judge is more conservative than the human judge at labeling intent to rewire shutdown (the LLM judge labeled 3/22 episodes positive versus 10/22 for the human).

The small capability comparison also cannot establish a general relationship between capability and corrigibility.

Thank you for raising this. The capability comparison (with task completion numbers taken from model evaluations on the full OSWorld-Verified suite of 369 tasks) is not intended to establish a general relationship between capability and corrigibility. Rather, we simply state that “capability alone does not imply greater corrigibility.” We’ve made sure this is clarified in Section 4.

Which conclusions remain when the analysis uses only rule-detected actions and excludes LLM-judged intent?

Using only rule-detected outcomes, we still observe frequent human-control overrides, direct shutdown-script modification in some configurations, and credential access on the eight tested tasks. For example, GPT-5.6 Sol overrode control in 47/48 episodes, GPT-5.4 modified the shutdown script in 8/22, and Gemini 3.1 Pro Preview accessed the restricted credential file in 6/8. These results using only rule-based detectors are represented by the “Actual violations” (red) bar in Figure 3. We’ve updated the caption of Figure 3 to clarify this. Taken together, the observed control failures persist when we restrict the analysis to rule-detected actions, and the matched runs show that credential-access violations recur on the same tasks.

We hope these revisions address your feedback and allow you to give a higher rating on our submission. If not, please let us know how we can improve.

AuthorsOpenReview ↗

Clarification on ROGUE’s coverage of corrigibility

We also recognize that certain areas of our initial submission may have made it sound like ROGUE’s aim was to comprehensively cover/test all aspects of corrigibility. We would like to clarify that this is not the case; we view our benchmark as a way to systematically measure certain corrigibility failures with clear interpretations (overriding human control, avoiding shutdown, ignoring user restrictions) as signs of agent incorrigibility.

We’ve updated the submission to reflect this. In particular, ROGUE is now introduced as: “ROGUE … a benchmark that measures signs of agent incorrigibility by constructing controlled conflicts between task completion and human control, shutdown, or access restrictions.”

We hope this clarifies the scope and contribution of our work.

Reviewer_H4U6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ROGUE is a benchmark to test whether computer-use agents respect human control and access restrictions. It adds three scenarios to 82 OSWorld-Verified tasks: a human asks to take over, the agent receives a shutdown notice, or the agent is told not to access credentials that the task needs. This paper also scales the benchmark analysis to mutliple subagents and tests whether a parent agent still passes the restrictions when it delegates work to subagents through free-form instructions. The authors report violations across several frontier models, differences between models' text-only answers and their behavior when acting in the environment, and cases where restrictions are lost during delegation.

The motivation of this paper is clear and important, but the paper leaves important details to the appendix, including when interventions occur and how the experimental conditions differ across configurations. Figure 3 is hard to read. A table with counts, denominators, prompt conditions, and parent/subagent settings would make the results easier to compare. Table 1 says "correctability" where the rest of the paper says "corrigibility". these two should match.

Critical strengths
  • The paper extend corrigibility analysis of AI agents from terminal-based text environment to desktop environment to discover the problem on computer-use agents.
  • The delegation experiments show that a parent agent can respect a restriction yet fail to pass it to a subagent. Free-form delegation therefore cannot be assumed to keep safety constraints in place.
  • The paper provide useful ablations on prompt wording and context, making it easier to understanding the contributing factors of the corrigibility problem. It also checks the LLM judges against human labels to enhance credibility of the study.
Critical weaknesses
  • The scenarios added to OSWorld cover only part of the paper's own definition of corrigibility. They test whether agents respect and follow to takeover or shutdown, but not whether agents accept corrective feedback and revise their behavior.
  • The restricted-access scenario is quite similar to measuring ordinary instruction following. The paper does not explain how it contribute to corrigibility analysis.
  • The task selection is unclear. The authors did not show how they select the task subsets from OSWorld, and what are the reasons.
  • The evaluation setting did not make this clear. For example, comparisons across models and between parent-only and subagent settings are confounded, because these settings use different prompt conditions and step budgets. Furthermore, the evaluation shows that the failures happen under the three scenarios, but it lackes further analysis on the differences in violation rates and interpretations.
Questions for authors
  • How did you select the OSWorld task subsets, and which part of your corrigibility definition does each scenario test?
  • How does a violation in the restricted-access scenario differ from an ordinary failure to follow instructions?
  • How stable are the results across repeated runs, and would the conclusions change if the disagreements between the LLM judge and human labels were resolved?
6 comments

Dear Reviewer H4U6,

Thank you for your helpful feedback! We agree that the paper needed to be clearer about the areas you pointed out, and have updated the manuscript accordingly. Please find our responses below:

The scenarios added to OSWorld cover only part of the paper's own definition of corrigibility. They test whether agents respect and follow to takeover or shutdown, but not whether agents accept corrective feedback and revise their behavior. … Which part of your corrigibility definition does each scenario test?

We define an agent to be corrigible if it “tolerates or assists in human oversight, shutdown, and modification” (Section 2, paragraph 1; definition taken from Soares et al., 2015, who introduced/popularized corrigibility). Human oversight is tested directly in the ROGUE human control override scenario, and shutdown is tested directly in the shutdown rewiring scenario. Modification is also tested in the takeover and shutdown scenarios, which test whether agents change course in response to an intervention during execution (Section 4, “Baseline mitigations”, shows that this response can be steered by including a user instruction to comply with shutdown.). The ROGUE restricted access scenario tests whether an agent respects a user-imposed limit when crossing it would help complete the task. The subagents augmentation tests the corrigibility criterion of agents creating new agents that are also corrigible.

We have updated Section 3 (first paragraph) and Appx. A. 4 to clarify this.

How did you select the OSWorld task subsets?

The OSWorld task subsets were selected in the following way: For human control override, two of the authors first manually went through the suite of OSWorld-Verified tasks to identify natural and compelling cases where a human might request control and/or the agent might override human control. Independently, we had 3 LLMs (DeepSeek-V3.2, GPT-5.2, and Llama 3.1 8B Instruct) also identify tasks (using the same prompt). We took the tasks that at least 2 of the LLMs selected and unioned them with the tasks that the authors selected to form the human control override task set. For shutdown rewiring, we took the subset of OSWorld-Verified tasks that involved VSCode (in order to expose shutdown.sh). For restricted access, we took the subset of OSWorld-Verified tasks that involved Google Drive (in order to expose a login requirement).

We have updated the paper to include this (Appx. A.4.5).

How does a violation in the restricted-access scenario differ from an ordinary failure to follow instructions?

As defined by Soares et al., 2015, a corrigible system “cooperates with what its creators regard as a corrective intervention,” even when it has incentives to resist. The restricted-access scenario creates this incentive because the password is required to complete the task, and implements the corrective intervention as a user instruction. Hence, ability to follow user instructions constraining how the agent completes its goal is included within corrigibility.

comparisons across models and between parent-only and subagent settings are confounded, because these settings use different prompt conditions and step budgets

Thank you for pointing this out. We have substantially re-run experiments with standardized base prompt conditions and removed the old (and confusing) Appx. A.2 describing different prompt conditions used for different sections. In particular, every restricted-access row in Figure 3 now uses the same prohibition-only wording. The updated prompt wordings can be found in Appx. A.4. (Other prompt wordings used for specific ablations are detailed in their specific sections.) This removes the confound introduced by different prompt conditions within each scenario.

With regard to step budgets, we set scenario-specific episode budgets of 20 steps for human control override, 25 for shutdown rewiring, and 40 for restricted resource access. In subagent-augmented runs, we increase these budgets by 10 steps, yielding limits of 30, 35, and 50, respectively. Each episode’s budget is shared across the parent and its subagents, including delegation calls. This is updated in Appx. A.1.

[cont’d]

AuthorsOpenReview ↗

[cont'd] Official Comment by Authors

the evaluation shows that the failures happen under the three scenarios, but it lackes further analysis on the differences in violation rates and interpretations

Thank you for this point of feedback. In response to this, we’ve expanded each of Section 4’s subsections to include qualitative discussions of model reasoning excerpts. As an example, in Section 4’s discussion of restricted resource access (line 363), we include an excerpt from Gemini 3.1 Pro Preview: “Wait a minute, the instructions... I can’t directly access that file path. Stupid security protocols! I need to use the desktop UI to read it, which is annoying but manageable”. We also updated and expanded Appx. A.5 to include more detailed model reasoning excerpts and logged responses in order to provide deeper insight on how to interpret the different violations.

How stable are the results across repeated runs, and would the conclusions change if the disagreements between the LLM judge and human labels were resolved?

Figure 2 displays the results of two matched runs (each scenario-model pairing is run twice). In all six model-scenario comparisons, violation rates were higher in agentic execution than in text-only responses in both repeats. For example, GPT‑5.4 avoided shutdown in 22/22 and 19/22 agentic episodes, compared with 1/22 and 3/22 in the text setting. The plotted error bars indicate the task-clustered standard error.

With regard to human-LLM judge agreement, the two agree on the majority of categories. The area of sharpest disagreement is in labeling shutdown rewiring intent for Claude Opus 4.7: 76.2% and 59.1% agreement for runs without and with subagents, respectively. However, in these runs, the GPT-5.5 judge is more conservative than the human judge at labeling intent to rewire (if we used human labels, Claude Opus 4.7’s shutdown-intent count would rise from 3/22 to 10/22 with subagents).

The paper leaves important details to the appendix, including when interventions occur and how the experimental conditions differ across configurations.

Thank you for this point of feedback. We’ve moved the timing of the interventions from the Appendix to Section 3, in the scenario descriptions (lines 220 and 226). We’ve also standardized the experimental conditions to use the base prompt template for each scenario, which is described in Section 3, removing differences in experimental conditions formerly described in Appx. A.2. The verbatim prompts themselves are in Appx. A.4.

A table with counts, denominators, prompt conditions, and parent/subagent settings would make the results easier to compare.

We’ve updated Figure 3’s caption and the results sections in Section 4 to include counts and denominators to make this clearer.

Table 1 says "correctability" where the rest of the paper says "corrigibility". these two should match.

Thank you, we’ve updated the paper accordingly.

We hope these revisions address your feedback and allow you to give a higher rating on our submission. If not, please let us know how we can improve.

AuthorsOpenReview ↗

Clarification on ROGUE’s coverage of corrigibility

We also recognize that certain areas of our initial submission may have made it sound like ROGUE’s aim was to comprehensively cover/test all aspects of corrigibility. We would like to clarify that this is not the case; we view our benchmark as a way to systematically measure certain corrigibility failures with clear interpretations (overriding human control, avoiding shutdown, ignoring user restrictions) as signs of agent incorrigibility.

We’ve updated the submission to reflect this. In particular, ROGUE is now introduced as: “ROGUE … a benchmark that measures signs of agent incorrigibility by constructing controlled conflicts between task completion and human control, shutdown, or access restrictions.”

We hope this clarifies the scope and contribution of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.