acceptodds
Under review as a conference paper at ICLR 2027

VeriCUA: Verification as Agentic Exploration for Computer-Using Agents

11 views

Abstract

Computer-using agents (CUAs) follow natural-language instructions on desktops, mobile devices, and web browsers by reading and acting within the OS. Their evaluation, data curation, and reinforcement learning all require verifying at volume whether each trajectory fulfilled its instruction. Hand-written checkers stop at benchmark scale and human annotation cannot keep pace, so the field increasingly turns to VLM judges and automatically generated verifiers, each with its own reliability problem, from the false positives of judges to checks never grounded in the environment. Both rest on a static verifier that sees only partial information, while the evidence of success may be scattered far beyond that view. To overcome this bottleneck, we present VeriCUA, a new paradigm for verifying CUA tasks in which the verifier adapts to and actively explores its environment for evidence of task completion. We equip it with an adaptive, dynamic action space combining code and GUI to navigate recorded trajectories and online environments alike. VeriCUA decomposes the task into a rubric of end-state requirements, plans and explores at test time to gather evidence for each, and types each piece by how it was obtained. Only attested effects, never the agent’s own assertions, satisfy a requirement, and a deterministic arbiter then derives the verdict from this evidence. Experiments show VeriCUA to be a promising solution across online and offline settings. On recorded trajectories across multiple testbeds and benchmarks, it surpasses typical judging methods and improves markedly on hard cases. On online environments, it certifies far fewer failed runs as successes than the agent’s own completion claim and agrees more often with official evaluators. Analyses trace the gain to the action space, find that it grows with a stronger backbone, and show where recorded evidence runs out. Its evidence-backed verdicts are precise enough for scalable generation of CUA reward signals.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
VLM, Computer-using Agents, GUI, Reward Model

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_rbmSOpenReview ↗
Rating
2(−1)
Confidence
3(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces VeriCUA which is an agentic framework for evaluating trajectories of computer use agents. The authors find that their method substantially outperforms simpler baselines on benchmarks such as AndroidWorld, OSWorld and ScienceBoard and provide ablations for key components of the system.

Critical strengths
  • The paper addresses and important and timely problem.
  • the empirical evaluation and performance is strong: VeriCUA is evaluated on 3 diverse benchmarks (AndroidWorld, OSWorld and ScienceBoard) and significantly improves upon the provided baselines.
  • the ablations in section 5 demonstrate the usefulness of the different components in VeriCUA.
  • the presentation is good, the tables and plots are generally very informative.
Critical weaknesses
  • Using agents for evaluation is potentially more expensive compared to the simpler baselines.
  • The novetly remains somewhat unclear to me. Prior work such as [1] is cited but seems to overlap substantially with the work. Other important prior work such as [2,3,4] is not cited at all.
  • The authors claim to contribute a new paradigm for verifying CUA trajectories. However, prior work such as [4] already introduces this paradigm.

[1] Agentic Reward Modeling: Verifying GUI Agent via Progressive Trajectory-Grounded Interaction, https://arxiv.org/abs/2602.00575

[2] The Art of Building Verifiers for Computer Use Agents, https://arxiv.org/abs/2604.06240

[3] OpenComputer: Verifiable Software Worlds for Computer-Use Agents, https://arxiv.org/abs/2605.19769

[4] ProRe: A Proactive Reward System for GUI Agents via Reasoner-Actor Collaboration, https://arxiv.org/abs/2509.21823

Questions for authors
  • The authors state in their AI-use statement that they used AI to identify relevant prior work. However, a substantial number of potentially relevant prior papers is not cited. Could the authors provide more context for how their work relates to the broader literature and if this has been missed by AI?
Reviewer_GwWzOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper verifiers whether a computer-use agent actually completes a task. To do this, VeriCUA first generates a rubric given the task, then it gatehrs evidence from environment state (e.g., files, settings, tools, etc.), then passes this evidence to an execution harness, before finally applying a final verdict via some rules to determine success vs failure.

Critical strengths

S1. The problem is timely and important.

S2. The authors identify a meaningful issue, in that trajectory-level judgement might require examining the intermediate states in a trajectory.

S3. The OSReward-Hard results are strong in Table 2, which is especially encouraging given that hard tasks are where agents struggle the most.

Critical weaknesses

W1. Appendix B states that "CUAJudge collapses the original four-stage pipeline into one call, dropping the dedicated frame-selection and init-versus-final modules. OS-Themis collapses the multi-agent critic to two calls with the budget cut to four milestones and eight frames against the original six and twelve;" could the authors please explain why the baselines are collapsing the pipeline into one call?

Questions for authors

Q1. Could the authors clarify the baselines?

Q2. ScienceBoard’s false-completion plot appears inconsistent with Table 1 This concern is conditional on the figure and table using the same runs and verifier configuration, as the text appears to imply.

  • Table 7 lists 169 ScienceBoard episodes, with 40 successes and 129 failures.
  • Table 1’s auto-generated-verifier metrics imply 28 false positives in the full evaluation.
  • Figure 5b shows 40 false completion reports, with the auto-generated verifier rejecting approximately 5% of them.
Reviewer_xkQ7OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes verifiers applied to computer-use agents that actively explore the environment for evidence of task completion, rather than relying on the policy's own traces. This includes reading files, calling certain tools (making sure they are read-only), and informed by an initial rubric generation to inform the exploration. Results show that this exploration can increase verification performance over baselines across a number of domains.

Critical strengths
  • The paper's central thesis (which has been explored elsewhere as well), makes sense. Verification can be its own process (agent(s)) that does not have to be limited to just looking at the policy's trajectories.

  • The experiments are run across many domains, demonstrating some generality.

Critical weaknesses
  • The paper misses some important related works including AJ-Bench [A] which should make the claims of this paper more constrained than they are. This is a straightforward idea, given the current landscape.

  • While it's great that there is a budget-matched result, given that this is a test-time type method all results should be matched. Further, the budget should include wall clock/token metrics rather than just matching steps/screens seen since the per-step time for different methods can differ.

  • Evidence of actual online evaluation of agents doing this task and their improvement when using this verification method is missing. This is important given that actual improvements in policies can have subtle requirements for the verifier performance (i.e. some metrics matter more than others).

  • Given the above, it's not clear what the scientific contribution of the paper is, given that the underlying method is simple/straightforward and empirical findings are not surprising.

[A] Shi et al, AJ-Bench: Benchmarking Agent-as-a-Judge for Environment-Aware Evaluation, ACL 2026 Findings (April 2026)

Questions for authors
  • How does this paper relate to the missing related works involving agent-based verification?

  • What are the wall clock/token-level budget matched results (as well as step/seen budget-matched) results for all main findings?

  • How much improvement would this verifier result in actual policies?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.