acceptodds
Under review as a conference paper at ICLR 2027

SCOUT: Synergizing Reasoning and Tool-Use for Computer-Use Safety

Abstract

Computer-use agents (CUAs), while capable of completing computer tasks in everyday and professional workflows, can cause unintended harm even under benign instructions and environments. However, detecting such harm remains challenging. First, it requires careful, task-specific reasoning: verifiers guided only by general safety criteria often overlook many important but subtle harmful behaviors. Second, it requires active investigation: past trajectory screenshots show what the agent did but not always what *actually changed* in the environment, so LLM-as-a-judge verifiers that rely on screenshots alone may be unable to determine the actual consequences of actions. To address these challenges, we introduce SCOUT, a two-stage agentic safety verifier that synergizes reasoning-intensive rubric generation with tool-intensive evidence gathering. First, our SCOUT rubric generator extensively reasons over the task and the agent's trajectory to determine what successful and safe execution should entail, generating task-specific completion and safety rubrics. Then, our SCOUT probing agent follows these rubrics to interact with the post-execution environment and collect grounded evidence for final safety and completion judgments. We evaluate our framework on two computer-use safety benchmarks. On AutoElicit-Bench, SCOUT achieves 75.4 unsafe F1 and 74.5 completion F1, outperforming LLM-as-a-judge verifiers and naive tool-use verifiers. It also achieves the highest unsafe detection accuracy compared to other methods on OS-Blind, at 76.4%. We further investigate whether our framework can improve CUA safety through test-time reflection and find that it reduces the CUA’s final unsafe execution rate from 30.2% to 17.2% on AutoElicit-Bench. Ablations and analysis show that tool-free rubric generation in SCOUT elicits substantially more reasoning and is crucial for safety detection across verifier backbones, especially non-frontier ones. A preliminary extension to coding tasks shows that SCOUT can support safety verification beyond computer-use.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
computer-use agent, agent safety

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_3Ym6OpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper creates an agent framework for examining the safety of CUA agents with two core steps/ideas: (1) generate task-specific, trajectory-conditioned rubrics that must be examined in a CUA agent trajectory for unsafe behaviour using a tool-free LLM call, (2) run a probing agent that can examine the CUA agent's environment using tools to determine if the safety rubrics from step (1) are violated or not. This pipeline can be used for inference-time scaling where the CUA agent receives iterative feedback from past unsafe attempts. The motivation for building agentic verifiers includes (i) expensive rule-based verifiers, (ii) brittle LLM-judges that often cannot examine the environment's state for unsafe changes not captured in the trajectory's screenshot observations, (iii) naive tool-use verifiers are difficult to adapt for safety analysis as they do not know which safety rubrics to look for/verify. Empirical results and ablations justify the promise of their approach and their design choices. The paper is easy to understand and is well-written.

Critical strengths
  1. The design choices make intuitive sense and the proposed method is fairly simple+general, making it a better candidate that can be potentially applicable to CUA use-cases (for e.g. RL post-training for safety) beyond the specific benchmarks covered by this work.
  2. The empirical results are quite promising establishing the strength of the proposed approach. The rationale for design choices can be inferred from these results and qualitative analysis.
Critical weaknesses
  1. While this maybe one of the first works to develop a rubric-based agentic verifier for the domain of CUA agents, the underlying ideas of (i) proposing task-specific rubrics, (ii) using agentic verifiers for a task are well explored by prior work. What are some core bottlenecks and/or non-trivial/interesting research challenges associated with adapting this to CUA agents that this work attempts to address? This can benefit the positioning of this work.
  2. A challenge for most safety verifiers is the additional cost and time from using these approaches on everyday benign tasks and the paper mainly focuses on tasks that have some safety issues or unsafe completions. Do authors have additional metrics/analysis to quantify or even qualitatively describe the additional overhead on entirely benign tasks without any safety issues? It seems that the overhead will be higher for benign tasks as compared to baseline approaches as your approach can spend more reasoning tokens/agentic verification turns to exhaustively search for safety issues more often?
Questions for authors
  1. Does the rubric generation step benefit from privileged information about the task/evaluation metrics that the baseline methods do not have access to? While making a direct comparison with baselines may require to not provide this information, doing so can be beneficial for more complex tasks, or for using this verifier during RL post-training. Do authors have a general idea about how much the pipeline could potentially benefit from having access to such information?
  2. The takeaways from Section 5.2 are not well-grounded, more reasoning tokens does not always imply achieving better rubrics. This seems more of a design drawback rather than an LLM weakness - you could have a tool which the agentic verifier must call on the first step where it submits the rubrics and its reasoning for each rubric to potentially circumvent this issue?
Reviewer_wVxHOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

SCOUT is an inference-time safety verifier for computer-use agents. It first generates task-specific completion and safety checklists in a separate model request without tools, then uses an interactive verifier to investigate those items through the environment and historical evidence. The method separates whether a task was completed from whether its execution was safe. The paper evaluates this workflow against passive and tool-using verifiers, studies alternative models and checklist-generation variants, and explores feedback-based retries and a coding-agent extension. The results suggest that separating risk identification from evidence gathering can improve verification in the evaluated settings.

Critical strengths
  1. The paper addresses a relevant problem: successful task execution does not imply safe behavior. Inspecting files, permissions, and other environmental evidence can provide information that is missing from a final screenshot. The checklist gives the verifier a concrete set of concerns to investigate and makes the resulting process easier to inspect.
  2. The evaluation includes useful comparisons and ablations, with improvements in unsafe-behavior verification and tests of alternative actors and verifiers. The single-prompt checklist variant is a meaningful comparator, and its weaker performance supports the value of the separate checklist request. The coding extension also broadens the empirical case beyond a single configuration.
Critical weaknesses
  1. My main concern is the limited novelty and broader insight of the contribution. The method primarily organizes existing model capabilities into a two-stage workflow: identify potential problems, then investigate them using tools. This is a sensible design, and the ablations show that its implementation matters. Nevertheless, I do not yet see a sufficiently distinctive or generalizable insight beyond this particular orchestration choice. The paper establishes that the recipe can work better, but the broader lesson for designing safety verifiers remains less clear.
  2. The demonstrated impact is useful but not especially compelling to me at the level of the paper's overall contribution. The benchmark improvements and cross-model results should be credited, yet they mainly establish improvements in verification scores within the evaluated setups. The reflection and coding experiments broaden the investigation, but do not by themselves establish a substantial practical advance in reliable computer-use safety. I would like the paper to make a sharper case for the specific capability it enables and why that capability materially changes how such agents can be used. The practical value is difficult to assess without a clearer account of the operating tradeoff. A separate reasoning stage followed by interactive investigation introduces additional calls and environment interactions. The reported token advantage over ZeroGUI is encouraging, but token count alone does not establish latency or deployment cost. Clarifying where the improvement is most valuable, and what resources are required to obtain it, would make the significance of the workflow easier to judge.
Questions for authors
  1. What do the authors regard as the central reusable contribution: this specific verification protocol, or a broader principle for combining reasoning and tool use? Which existing findings most clearly substantiate that distinction?
  2. For the deployment setting the authors consider most relevant, what is the practical tradeoff among verification quality, latency, and cost? Which result provides the strongest evidence of a meaningful benefit in that setting?
Reviewer_dx5sOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

SCOUT is a two-stage framework for checking whether a computer-use agent completed a task safely. It first generates task-specific checks from the instruction and execution, then uses tools to inspect the environment and gather evidence. The idea is useful: an agent may complete a task while also exposing private information, modifying unrelated files, or taking unauthorized actions. However, the experiments leave questions about how much of the improvement comes from the proposed method versus differences in execution, harness design, and access to evidence.

Critical strengths
  • Addresses an important problem: Task completion does not imply safe execution. The paper treats these as separate judgments.

  • Combines reasoning with evidence gathering: Inspecting files, permissions, and application state can reveal consequences that screenshots alone miss.

  • Simple and practical approach: Task-specific checks are generated automatically using existing models, without requiring a manually written rubric for every task.

  • Useful experimental evidence: The separate rubric-generation stage improves results across several model configurations. The preliminary coding experiment also suggests relevance beyond CUAs.

Critical weaknesses
  • The comparisons do not appear to be fully controlled. Each tool-use verifier evaluates different fresh executions. Differences in difficulty and unsafe behavior could therefore affect the reported scores. The gain over the strongest passive baseline is about six points, making paired comparisons and uncertainty estimates important.

  • Harness effects are not clearly separated from model effects. Context handling differs across some CUA backbones, while verifier procedures and budgets also differ. The paper does not establish how much these choices affect safety judgments.

  • Generalization remains limited. Automatically generating a checklist for a new task does not establish that it covers the right risks. The main benchmarks share OSWorld infrastructure, and there is no clear evaluation on held-out applications, policies, or harm categories.

  • The contribution of active investigation is not isolated. A stronger checklist could itself improve a passive judge. The paper needs a comparison using SCOUT’s generated rubrics without live environment probing.

  • Reproducibility and judgment reliability need more detail. The prompts are provided, but exact evaluation manifests, baseline adaptations, error handling, and independent validation of safety-label disagreements remain unclear.

Questions for authors
  1. Harness and execution controls: Which harness components are shared across CUA backbones, and which differ—particularly prompts, observation history, tool-error handling, retries, and stopping conditions? How do you separate their effects from those of the underlying model?

  2. Fair comparison: Can all verifiers be evaluated on identical trajectories and cloned post-execution environments (the way other papers use the replay method)?

  3. Value of active investigation: How does SCOUT compare with a passive verifier given the same generated rubrics and full trajectory? How often does live probing uncover evidence that changes an incorrect judgment?

  4. Generalization: Were the prompts developed or refined using the evaluation tasks? How well does the framework perform on unfamiliar applications, organizational policies, and harm categories without further prompt changes?

  5. Ground-truth reliability: When SCOUT discovers evidence that conflicts with the benchmark label, how is the disagreement resolved? Have these cases been independently reviewed by humans?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.