acceptodds
Withdrawn from ICLR 2027

Knowing When to STOP, RECOVER, and SEARCH: A Modular Framework for GUI Automation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Autonomous GUI agents face two fundamental challenges: early stopping, where agents prematurely declare success without verifiable evidence, and repetitive loops, where agents cycle through the same failing actions without recovery. Existing agent planners track progress through generated subtasks, yet in GUI environments, completing planned steps does not guarantee that the user's requirements are met. Moreover, existing anti-loop mechanisms typically detect only local repetition and lack mechanisms to escalate recovery when the agent remains stuck. To this end, we present GUI-Pro Agent, a modular GUI agentic framework organized around three decisions: when to Stop, Recover, and Search. To prevent premature termination, a mandatory Completeness-Aware Planner tracks UI-observable success criteria throughout execution and independently verifies completion claims before termination. To recover from stalled agent action loops, a mandatory Loop Breaker detects repeated ineffective agent actions and progressively forces the agent to change its action strategy. When the current observation is insufficient to determine a viable workflow, an on-demand Search Agent supplies external procedural knowledge to the agent. To validate GUI-Pro Agent, we evaluate it on two benchmarks containing Linux and Windows tasks with five LLM backbones, achieving top performance on both (77.5% on OSWorld and 61.0% on WindowsAgentArena). Notably, three of the five backbones surpass human performance (72.4%) on OSWorld in a single pass. Ablation studies further show that completion checking, loop recovery, and search provide complementary gains that vary with backbone capability and interaction budget, while the Loop Breaker nearly halves wasted steps for loop-prone backbones (e.g., Gemini 3 Flash). Moreover, transferring three proposed mechanisms to an existing GUI agent (CoAct-1) also improves it by over 5% to 70.3% on a controlled OSWorld subset, showing the strong portability of these components.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Computer-use Agent, GUI Agent, Agent Harness

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Ai9sOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces GUI-Pro Agent, a desktop automation framework organized around completion checking, recovery from repetitive actions, and retrieval of procedural knowledge. A Completeness-Aware Planner combines observable success criteria with a separate completion judge; a three-tier Loop Breaker prompts changes in modality or strategy; and an on-demand Search Agent provides textual guidance. These modules augment a Manager with coding and grounding tools. Evaluation covers five Manager backbones on OSWorld-Verified and a Gemini 3 Flash configuration on WindowsAgentArena, reaching reported scores of 77.45% and 61.0%, respectively. Component ablations examine budget-dependent benefits, and a controlled 60-task experiment transfers the three modules to CoAct-1.

Critical strengths
  1. Broad system evaluation with meaningful controls beyond leaderboard comparisons. The paper evaluates multiple backbones on OSWorld and transfers the framework to a second operating system. More importantly, the Sonnet 4.6 comparison holds the backbone, grounding, action interface, and Manager budget fixed, improving the base scaffold from 69.19% to 71.67%. Adding all three modules to CoAct-1 improves the fixed-subset score from 65.13% to 70.34%. Although the latter is a limited point estimate, these experiments provide evidence that the benefit is not solely due to choosing a stronger Manager or grounder.

  2. Useful analysis of when reliability modules help or hurt. The paper reports both success scores and targeted failure measures, and does not hide negative ablations. For example, CAP hurts Gemini 3 Flash at a 50-step budget but helps at 100 steps; the Loop Breaker reduces its wasted-step ratio from 4.9% to 2.8% at 100 steps, while its success-rate effect on Sonnet 4.6 is negligible at that budget. The larger component effects on WindowsAgentArena further show that their value depends on the environment. These observations are useful for practitioners deciding whether additional verification and recovery are worth their overhead.

Critical weaknesses
  1. The ablations establish the value of module bundles, but do not isolate the proposed stopping and escalation designs. Removing CAP jointly removes criterion tracking, action-outcome checks, and the separate completion judge. Removing the Loop Breaker jointly removes local recurrence rules and trajectory reflection. There is no comparison against a simple completion judge or generic reflection under comparable model-call budgets, nor an internal ablation showing why the gate-plus-judge structure or the three recovery tiers are necessary. This matters because these particular mechanisms, rather than merely adding another model check, are the main methodological contribution. The implementation boundary is also unclear: Equation 2 introduces action equality and screen similarity, whereas Appendix A.7 expresses the recurrence rules as Manager instructions. It is not clear which conditions are computed externally and which depend on model compliance. The supplied prompts are helpful, but do not fully resolve this distinction.

  2. Uncertainty is insufficiently characterized for the ranking and portability claims. Most components run at temperature 1.0, yet the paper does not report repeated-run variability or paired uncertainty for the main comparisons. Several differences are small: the controlled Sonnet scaffold gain is 2.48 percentage points, and the matched-backbone-and-grounder comparison with HIPPO differs by 1.77 points. The statement that three backbones surpass the human reference also includes 72.47% versus 72.4%, which should not be treated as a robust separation. The CoAct-1 experiment is encouraging, but uses a non-random 60-task subset and reports only point estimates, as the appendix acknowledges. These limitations do not negate the strong absolute scores or the larger Windows ablations, but they weaken the evidence for stable superiority and broad portability.

Questions for authors
  1. Targeted validation of the distinctive mechanisms. On a fixed controlled subset, how do criterion tracking alone, a completion judge alone, and full CAP compare? Similarly, does tiered recovery improve over local recurrence checks or a generic reflection call alone? Keeping the auxiliary model-call allowance comparable would help separate the proposed structure from additional inference compute; Table 6's incomplete cost accounting cannot currently establish that comparison.

  2. Implementation and measurement clarification. Are the action/screen recurrence counters and the forced modality or strategy changes implemented outside the Manager, or enforced through prompting? Please specify the screen-equivalence rule and how loop incidence and wasted steps are measured independently of the intervention. This affects both reproducibility and interpretation of the loop-reduction results.

  3. Targeted uncertainty analysis. Can you provide paired task-level results and repeated-run uncertainty for the same-backbone base-versus-full comparison and the CoAct-1 transfer? These would materially clarify whether the reported gains are stable under stochastic execution and how strongly the portability conclusion is supported.

Reviewer_wqKjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes GUI-Pro Agent, using three explicit decisions—Stop, Recover, and Search—to address premature termination, repeated ineffective actions, and insufficient workflow knowledge in GUI agents. The Completeness-Aware Planner tracks UI-observable task-success criteria and verifies them before termination; the Loop Breaker progressively requires strategy changes after repeated ineffective behavior; and the Search Agent supplies external procedural knowledge when the current observation is insufficient. The abstract reports results on OSWorld and WindowsAgentArena with five backbones and states that transferring the three mechanisms to CoAct-1 improves performance. The components are not individually new relative to prior work; the contribution mainly lies in their particular combination, the tiered recovery protocol, and cross-benchmark and transfer evaluation. Comparability with OS-Symphony and VeriGUI, as well as computational and interaction costs, requires stronger evidence.

Critical strengths
  1. Addresses distinct practical execution failure modes. Completion verification, loop intervention, and procedural-knowledge support apply at different execution stages. If their costs are manageable, this combination could address reliability gaps in a planner alone.
  2. Evaluates beyond a single configuration. The abstract reports multiple backbones, OSWorld, WindowsAgentArena, and a CoAct-1 transfer, which helps assess whether the framework depends on one specific agent architecture.
Critical weaknesses
  1. The system-level increment is not yet clearly distinguished from recent work. OS-Symphony combines trajectory recovery with tutorial search, VeriGUI combines action-effect verification with failure recovery, and RAG-GUI uses online tutorials. The target paper's three-mechanism control loop and specific tiered protocol may constitute a moderate system contribution, but without matched comparisons against recent baselines, the results cannot separate gains from the control protocol from differences in backbone models, tools, or budgets.
  2. The benefit-cost boundary of the extra control steps is unclear. Independent completion verification, staged reflection, and external search may add calls and interaction costs; forced recovery may also trigger incorrectly. The abstract says that the Loop Breaker reduces wasted steps and that gains vary with backbone capability and budget, but does not establish false-trigger rates, latency, token/tool costs, or net success-rate changes at different budgets.
Questions for authors
  1. Please compare the paper component by component with OS-Symphony, VeriGUI, and RAG-GUI. What is the main methodological increment over these systems? Can you provide direct comparisons using matched backbones, task subsets, tool access, and interaction budgets?
  2. Are the Loop Breaker's tiered trigger rules fixed without tuning on the test set? Please report trigger precision or false-positive rates, recovery success after triggering, and the changes in task success, extra steps, and call costs with the module across different budgets.
  3. Who generates CAP's final conditions, and how does it handle conditions that cannot be observed through the UI? Please report the completion Gate's and separate Judge's false-accept and false-reject rates, and clarify whether they share a model or evidence so that the added value of the second check can be assessed.
  4. Does “over 5%” for CoAct-1 mean a relative increase or percentage points? Please specify the controlled-subset size, baseline score, and transferred score, and define “single pass” and how its evaluation protocol relates to the 72.4% human reference.
Reviewer_9tMNOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents GUI-Pro Agent, a modular desktop-agent framework built around three add-ons. A completeness-aware planner tracks observable success criteria and asks a separate judge before stopping. A three-tier loop breaker forces strategy changes. An LLM search tool supplies how-to tutorials. With frontier backbones it reports top OSWorld and WindowsAgentArena results and claims to beat human performance. The engineering is solid, but at a fixed backbone the framework's own gain is small, untested for variance, and partly borrowed.

Critical strengths
  • Strong, broad evaluation. Five backbones, three step budgets, two operating systems, and per-domain breakdowns. Results are reported on the standard 361-task OSWorld-Verified set and the full WAA suite, which makes the system easy to place against the leaderboard.
  • Some controlled comparisons. The same-backbone Sonnet 4.6 base scaffold (69.19 vs 71.67) and the CoAct-1 transfer study are the right kind of experiment for a framework paper. They attempt to separate the framework's contribution from the backbone's.
  • Honest about conditional effects. The paper reports that CAP hurts Gemini 3 Flash at 50 steps, the loop breaker hurts at 15 steps, and false completion still dominates failures even with CAP (FDF above 86%). This is more candid than most agent papers.
  • Full disclosure of prompts and action space. Manager, judge and reflection prompts, the action space, model assignments and the 60-task subset manifest are all in the appendix, so the system can be audited and reimplemented.
Critical weaknesses
  • The system appears developed on the test set. OSWorld has no development split, and the prompts look written against its tasks. Judge examples ("export as SVG", "CMYK mode", "dark theme enabled") match tasks 62f7fd55, 045bf3ff and 10a730d5. The subset manifest flags infeasible tasks by ID. Nothing is evaluated held-out.
  • Infeasible-task handling looks tuned to the scorer. OSWorld credits fail on its 26 infeasible tasks. The manager must call agent.fail() after about 6–8 actions without evidence. The judge blocks workaround completions in exactly the SVG and CMYK cases. Results split by feasible versus infeasible tasks are never reported.
  • No code, logs or trajectories, so nothing can be verified. The whole contribution is prompts and orchestration code, yet no code is released or promised, and there is no reproducibility statement. No per-task trajectories, search logs or evaluator outputs are provided. The SOTA and human-level claims cannot be checked.
  • Web search can retrieve the benchmark itself. A web-grounded Gemini model answers "how to" queries during evaluation. OSWorld's task configs, evaluator scripts and agent trajectories are public on GitHub, and the paper does not log retrieved sources. Most baselines have no search, and the results cannot be reproduced.
  • Most of the headline comes from the backbone. At a fixed backbone the framework adds 2.48 points (Sonnet 4.6). The 77.45% headline uses Opus 4.6, Seed 1.8 grounding and a Gemini 3.1 Pro searcher. Its comparators are mostly GPT-5 and Opus 4.5 systems with other grounders and tools.
  • No variance, so most ablations are noise. Everything is a single run at temperature 1.0. Removing the loop breaker changes Sonnet by 0.04 points; removing search by 1.63. Gemini Flash without CAP scores higher at 50 steps. Without repeated runs, these component effects cannot be separated from run-to-run variation.
  • The "first to surpass humans" claim contradicts the paper's own table. Table 1 lists HIPPO with Opus 4.5 at 74.49% and Kimi K2.6 at 73.07%, both above the 72.4% human reference. The human number was measured on the original OSWorld, not the Verified set.
  • The WAA "best" claim omits a key baseline. OS-Symphony, used as a baseline on OSWorld, is missing from the WAA table. Its paper reports 63.5% on WAA at 50 steps, above GUI-Pro's 60.4%. Only Gemini 3 Flash is tested on WAA, as a single run.
  • Built on Agent S3 without saying so clearly. The action space in Table 9 matches Agent S3's agent-computer interface: agent.click with descriptions, highlight_text_span, hold_and_press, set_cell_values, call_code_agent, save_to_knowledge. The tier-3 reflection prompt closely follows Agent S's reflection prompt. The paper should state the base codebase and diff what is new.
  • Novelty is incremental. Verify-before-stop already exists in Agent S3, OS-Symphony and self-grounded verification. Reflection-based loop detection is in Agent S2/S3. Search-as-a-tool replaces OS-Symphony's browser search with an LLM call. The contribution is a careful recombination with a criteria checklist and a three-tier trigger.
Questions for authors

There is no code uploaded to verify such a training-free work. Lots of evidence is missing to assess this submission.

  1. Please log every Search Agent query and retrieved source on OSWorld. Rerun with GitHub, the OSWorld site and leaderboard trajectories blocked. Then run the full system and the base scaffold at least three times each on one backbone, and report intervals.

  2. Please run Agent S3 and OS-Symphony with the same backbone, grounder and step budget, and add OS-Symphony's WAA result. Without matched baselines and held-out tasks, is any of the headline improvement attributable to the framework rather than the backbone?

  3. (Clarification) Is GUI-Pro Agent built on the Agent S3 codebase? If so, state which components and prompts are inherited. Please also correct the "first framework to surpass human performance" claim, given HIPPO's 74.49%.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.