acceptodds
Under review as a conference paper at ICLR 2027

Failure Migration in GUI Agents: Outcome Churn and the Trade-offs of Reflective Recovery

Abstract

Higher aggregate success can conceal task-level regressions and successful states lost during retry. We study these two forms of outcome change and their resource trade-offs through paired OSWorld evaluations, same-configuration reruns, and reflective recovery. Scaling Qwen3.5 from 4B to 9B raises recorded full-split success from 21.4% to 26.3%. Among 342 tasks with valid evaluations for both models, 35 improve and 22 regress; an always-on accessibility scaffold yields 21 improvements and 31 regressions on 339 evaluable pairs. Same-configuration reruns also exhibit substantial outcome churn. We evaluate grounded verification and reflective retry with an always-strong agent and FMART, a self-hosted 4B agent with failure-triggered strong-tier takeover. The always-strong reflective system improves full-split success from 51.5% to 59.1% on 369 tasks; its gain remains significant in a runner-matched paired comparison. With a three-round cap, FMART reaches 57.2%; its retained records contain 0.77 times as many strong execution steps as the always-strong reflective system. Both record approximately USD 0.42 per included task in their cost ledgers, which combine API estimates with nominal local-step charges for the cascade. In two FMART runs, five and seven tasks, respectively, have successful checkpoints followed by zero-score selected endpoints. Each of these tasks contains a successful checkpoint labeled incomplete by the completion checker. Together, these results show why evaluating GUI-agent improvement requires paired outcomes, returned-state success, and separate execution and monetary resource measures.

Primary area:
applications to robotics, autonomy, planning
Keywords:
GUI agents, computer-use agents, failure migration, vision-language models, reflection, adaptive escalation, OSWorld

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_mCRiOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper uses computer use agents or GUI agents to run comparative task level evals on OSWorld. The first experiment is running the same family of model across sizes and going deeper than the aggregate improvement and seeing whether the smaller models success is a subset of the larger models; the results however show that a significant share no overlap. Further reruns show more such swaps showing evals are stochastic. There is a an LLM labelled taxonomy that is given which classifies failure types with most being premature ending or incorrect stuck loops. They also evaluate reflective recovery where GUI agents check task completion, generate feedback from prior attempts and retry. This improves overall success modestly but incorrect completion judgments can also trigger unnecessary retries that destroy previously successful states.

Critical strengths
  1. Rather than relying solely on aggregate success rates, the paper measures individual task improvements and regressions, alongside same-configuration reruns that reveal substantial stochastic variation.
  2. The paper demonstrates that recovery mechanisms can inadvertently discard previously successful states when completion checkers incorrectly judge tasks as incomplete.
Critical weaknesses
  1. The paper primarily demonstrates phenomena that are already well established in existing literature - no novelty or significance of the work can be found.
  2. The experiments rely on a single benchmark, a narrow selection of models and weak ones at that (4B and 9B).
  3. The reflection experiments do not isolate the benefits of reflection from simply allowing additional attempts while the FMART comparisons involve different controllers and execution budgets.
  4. No solution for the problematic phenomena has been provided.
  5. The taxonomy of errors found are synthetic which is fine but the results of the tagging are common knowledge again given that endless looping and premature ending are common llm artifacts.
Questions for authors

What better benchmarking practices or methods do you suggest can over come this? Do you have AI measurement robustness ideas that could make future evals better suited for this? maybe different metrics? General Practices?

Reviewer_XWRcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies:

  1. Whether an improved agent can still solve tasks that the agent solved before. the authors compares model sizes, input settings and repeated runs on OSWorld, and found that the overall success rate can increase, but some previously successful tasks become failures.
  2. The paper also studies whether retries can lose earlier success. Some tasks are already completed, but the checker marks them as unfinished. Later retries then end in failure.
  3. They propose FMART, using a small model first, then switches to a strong model when needed. Compared with the always-strong system with retries, it uses less strong-model execution steps, but the recorded cost is almost same.
Critical strengths
  • The paper reports cases where a successful round is followed by a failed one. It finds this in two FMART runs. This gives some empirical support for the problem discussed, within the tested system.

  • The paper provides task-level tables and describes the record selection rules. Readers can check the main counts and statistics from these tables. This makes part of the analysis easier to verify.

Critical weaknesses
  • Personally, I feel the paper writing is different from what I would expect from a paper. The authors do analyze some specific observations and possible causes, but I did not learn much from the paper.
  • The empirical contribution seems limited. Task-level regressions are already studied in MUSCLE and AgentDevel. Related success-loss cases are also reported in bBoN. This paper adds measurements from a narrow setting, but the new insight beyond these works is not clear enough.
  • Some execution records are incomplete. The continuous-rank check helps, but it does not confirm the actual final state of every run. This limits the conclusions about what the system finally returns to the user.
Questions for authors
  • What is the main new empirical finding beyond the regression analysis in AgentDevel and the success-loss examples in bBoN? What additional understanding comes from the checkpoint-to-endpoint analysis?
  • Can authors provide complete round histories and stopping decisions for the success-loss cases? How many selected endpoints can be confirmed as the actual terminal states?
Reviewer_rNSROpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies task-level outcome changes in GUI agents that aggregate success rates can hide by showing regressions and new wins. Using OSWorld, it evaluates model scaling, an accessibility-tree scaffold, repeated runs, and reflective recovery. It introduces FMART, a local-first cascade that escalates failures to a stronger model. The paper shows that the always-strong reflective system improves success over its single-attempt baseline; most other comparisons are statistically inconclusive.

Critical strengths

The position of the paper where distinction between aggregate improvement, task-level churn, and loss of successful states during retry is useful for evaluating GUI agents.

Critical weaknesses

Lack of novelty and confound contributions: The paper combines a churn measurement, a failure-label audit and a cascade system, and none of them yields a substantive contribution. Prediction churn and negative flips are studied in the cited work and nothing new. The proposed FMART methodology does not beat the always-strong reflective system (57.2% vs 59.1%, p = .46) and costs the same per task.

Questions for authors
  • What is the single main contribution: the churn measurement, the failure audit, or FMART? What should a practitioner do differently after reading the paper?
  • Do the findings hold on another benchmark (e.g., WindowsAgentArena or AndroidWorld), another model family, or another strong model?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.