acceptodds
Withdrawn from ICLR 2027

CUA-Recovery: Benchmarking Error Recovery in Long-Horizon Computer-Use Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

As computer-use agents (CUAs) move to long-horizon workflows spanning multiple applications, undetected errors can silently corrupt persistent state and cause cascading failures in subsequent actions. Reliable error detection and recovery have therefore become critical bottlenecks for building robust CUAs. However, existing benchmarks focus primarily on end-to-end task performance, providing limited insight into whether agents can detect and recover from errors during execution. The few benchmarks that evaluate recovery rely largely on manually injected errors, which may not reflect the types or horizons of failures induced by real agent policies. This mismatch limits both realistic evaluation and systematic progress. In this paper, we introduce CUA-Recovery, a benchmark comprising test and training suites across multiple desktop applications. We construct the benchmark using ReRail, a framework for synthesizing long-horizon tasks and trajectories in a persistent desktop environment. ReRail composes MyPCBench tasks into new long-horizon workflows with executable ground truth, collects verifier-confirmed failures from multiple policies, and turns them into verified recovery trajectories. ReRail yields CUA-Recovery, which comprises a human-verified test suite of 2,250 erroneous states over 100 tasks and a training suite of 8.6K trajectories over 377 tasks. Evaluation on the test suite shows that agents struggle both to recognize inherited errors and to complete the task. Across six agents and six takeover depths, Error Awareness Rate (EAR) reaches at most 44.0% and Pass@3 at most 25.5%. After fine-tuning Qwen3.5-35B-A3B on the training suite, its average EAR rises from 19.9% to 35.2% on held-out recovery states, and its Rubric Score goes from 5.0% to 16.1%. It also gains 15.3 points on MyPCBench and 10.8 points on Odysseys for Pass@3, showing that the training trajectories help both with recovery and with completing tasks from a clean start.

Primary area:
datasets and benchmarks
Keywords:
Robust Computer-Use Agent, Visual Language Model, Error Recovery of CUAs, Long-Horizon Tasks, Real Policy-induced Errors

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_U1swOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper builds CUA-Recovery: 2,250 erroneous states from 619 failed rollouts of six agents on 100 multi-app desktop tasks, plus 8,577 training trajectories. Agents that take over mid-task recover in at most 25.5% of states (Pass@3); fine-tuning Qwen3.5-35B-A3B raises its average EAR from 19.9% to 35.2%. The problem matters, but the headline numbers rest on unreported, sometimes tiny cells, and the protocol is Bu et al.'s at longer horizons.

Critical strengths
  • States come from the agents' own failures on trajectories averaging 60 steps, with takeover up to 25 steps after the root cause (page 6); Bu et al. (2026) stop at depth 5.
  • The judge reads terminal output and tool calls as well as screenshots (page 3), so CLI and GUI routes score alike; Table 11 backs it with human agreement.
  • The training set transfers: Odysseys Pass@3 rises from 24.1 to 34.9 (Table 3).
Critical weaknesses
  • Table 2 gives no n per cell. Table 5 lists 69 GPT 5.5 self-takeover episodes, so at most 69 states over six depths, about a dozen per cell, underlie the headline 25.5%. Report n and a confidence interval per cell.
  • Section 3 says each evaluation runs three times, yet Table 5's episode counts sum to 2,250, the number of states. State the runs per takeover state; if one, the metric is Pass@1.
  • Finding 4 (page 8) compares d=20 with d=25 on different state subsets (273 versus 231 states, Appendix G.2). Rerun it on states eligible at both depths before blaming the context window.
  • EAR is judged on the first three reasoning segments only (Appendix G.1; Section 5.1 omits this), and no human check of the judge (Listing 4) is reported. Scaffolds expose different amounts of reasoning text, a confound the judge cannot separate. Hand-label about 100 takeover runs.
  • The teacher that writes recovery targets is never named (page 6; Table 4 counts 3,350 fallback attempts), nor its share of the targets. F.3 does not document the token matching page 9 claims. Name the teacher; draw the control from its successes.
  • Page 4 calls Figure 3a's 84.3% and 71.2% execution shares Bu et al.'s reported statistics; Appendix C.2 says you labeled 100 trajectories per corpus. Bu et al. report no such numbers.
Questions for authors

Bu et al. (ICML 2026, "Recovering Policy-Induced Errors") already define the root-cause step, the error horizon, takeover at depth d, Error-Awareness Rate, and the comparison against AgentNet and GUI-Reflection. The paper reuses all of it, metric names included, and names them as the closest work (page 4) without saying the protocol is theirs. It adds depth 25, trajectories averaging 60 steps across 17 applications, a judge that reads application state, and a promised data release (Bu et al.'s remains "in preparation" on GitHub). Zhang et al. (ACL 2026, VeriGUI) also train on synthetic failure trajectories and report a Recovery Success Rate; Chung et al. (2026, WebStep) locate where web agents go wrong. Neither is cited.

The abstract's EAR and Rubric averages reproduce from Table 2; the ReRail Pass@3 average computes to 2.1, not Table 14's 1.9. Finding 3's association between awareness and recovery rests on 15.6% versus 12.2% macro averages with no counts and no test; add both.

Prose that reads as AI-written: "have therefore become critical bottlenecks" (page 1), "node density underpins the long-horizon nature" (page 19); the abstract lists eight performance figures.

2 comments

We thank the reviewer for the detailed review. We address each point below.

  • Q1: Report n and a confidence interval per cell.

Answer:

Table 5 uses 69 GPT 5.5 takeover states to estimate the computer for one pass through the evaluation set. For Pass@3, we run three rollouts per state and count the state as successful if at least one rollout completes the task. The number of eligible states varies by depth. For confident interval, the experiment is conducted under same seed for three times, we planned to add multi-seed experiment, but consider the high-cost and the domain tradition, single seed-multi turn experiment is already enough to reveal the problem.

  • Q2: State the runs per takeover state

Answer:

Each takeover state is evaluated with three rollouts. Pass@3 counts a state as successful if at least one of those rollouts completes the task. The self-takeover counts in Table 5 sum to 2,250 because the compute estimate uses one rollout per state, with a representative per-rollout runtime. We will label these rows as single-pass compute estimates and rename the “Episodes” column to “Takeover states” for the self-takeover rows.

  • Q3: Rerun it on states eligible at both depths before blaming the context window.

Answer:

We rerun on the same 231 failure trajectories. Pass@3 is slightly lower at (d=25), from 3.90% (9/231) at (d=20) to 3.03% (7/231) at (d=25). The decline is modest and persists when the set of failures is held fixed. The (d=25) condition tests whether agents can recover after the root-cause screenshot leaves the 20-screenshot history. The application state can also change over the five intervening steps, so we interpret the paired result as a comparison of later takeover on the same failures, with reduced access to visual evidence as one possible contributor.

  • Q4: no human check of the judge

Answer:

The EAR judge agrees with human annotators on 92.2% of 245 takeover rollouts (Cohen’s κ = 0.773), with over 90% agreement for each of the six agents, including all three API-based agents. We will state in Section 5.1 that the judge scores whether the agent explicitly recognizes a problem in the inherited work within its first three post-takeover reasoning segments. The audit validates scoring on the reasoning available to the evaluator. Because scaffolds expose different amounts of reasoning, we interpret EAR as the rate of expressed awareness within that window under each evaluated scaffold.

Table A. Human agreement on error awareness

Evaluated agentRolloutsHuman-awareJudge-awareAgreement (%)Cohen’s κPrecisionRecall
GPT 5.53611991.70.7240.8890.727
Claude Opus 4.849151591.80.8080.8670.867
Kimi-K3336590.90.6730.8000.667
EvoCUA-32B397892.30.7530.7500.857
OpenCUA-72B466993.50.7630.6671.000
Qwen3.5-35B-A3B428895.20.8460.8750.875
  • Q5: Name the teacher; draw the control from its successes.

Answer:

GPT 5.6-Terra supplied 21.6% of the retained recovery targets, and training on Qwen-generated targets alone still improved recovery. On the same 571 held-out takeover states, base-only Recovery SFT improves Pass@3 from 0.4% (2/571) with success-only behavior cloning to 1.9% (11/571). EAR rises from 24.9% to 32.4%, and Rubric Score rises from 11.5% to 12.9%. This comparison shows that CUA-Recovery provides useful recovery supervision under standard SFT even when every recovery target comes from the base model.

Terra generated approximately 1,853 of the 8,577 retained trajectories. Qwen3.5-35B-A3B generated the remaining 6,724. The 3,350 entries in Table 4 count fallback attempts. We will report the teacher identity and retained counts in Section 4.

The success-only control starts from the same Qwen checkpoint and uses successful executions from the same training workflows. We will document in Appendix F.3 how we matched the training token budget for this control.

  • Q6: Page 4 calls Figure 3a's 84.3% and 71.2% execution shares Bu et al.'s reported statistics; Appendix C.2 says you labeled 100 trajectories per corpus. Bu et al. report no such numbers.

Answer:

The execution-error shares in Figure 3a, 84.3% for AgentNet and 71.2% for GUI-Reflection, come from our own annotations. Following Bu et al.’s comparison, we independently sampled and annotated 100 trajectories from each corpus using the protocol in Appendix C.2. A trajectory can receive multiple error labels, so each percentage is calculated over all error labels assigned within that corpus. We will revise the text and caption to state the source and denominator explicitly.

We thank the reviewer for detailed review. We answer the questions below:

We will explicitly credit Bu et al. for the protocol we adopt. This includes root-cause localization, the error horizon, controlled-depth takeover, Error-Awareness Rate, and the comparison with AgentNet and GUI-Reflection. We will also cite VeriGUI for training agents to verify and correct action effects using synthetic failures, and WebStep for locating decisive errors through semantic state tracking.

The correct equally weighted ReRail Pass@3 average is 2.1%. We will correct the 1.9% entry in Table 14 and specify in the caption that the six takeover depths receive equal weight. For Finding 3, the 15.6% and 12.2% figures are macro-averaged completion rates for rollouts with and without expressed awareness. We will report the successful and total rollout counts for each group by agent and test the association while accounting for repeated rollouts from the same source workflow.

The authors wrote the manuscript by themselves. Our disclosed use of AI was limited to polishing and rephrasing author-written text. Inferring authorship from two phrases is unwarranted.

Reviewer_aZKwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper:

  1. proposes a method ReRail for turning failure states into a benchmark.
  2. trains a model for error recovery, and demonstrates that it is useful.
Critical strengths
  1. The paper does a good job of demonstrating that there is a problem with error recovery in existing models.
  2. Results are strong both on error recovery and clean start training.
Critical weaknesses
  1. There is a ton of work in reinforcement learning and imitation learning (e.g. OPD, dagger) that this paper doesn't reference at all. I think it needs to be grounded better in previous work, including baselines that use standard methods from these fields.
Questions for authors

None

1 comment

We thank the reviewer for the detailed review. We address the point below.

  • Q: We should include more baselines

Answer:

CUA-Recovery is positioned as a benchmark paper. We use standard SFT just to test whether its training data are useful. The paper’s contribution is a set of human-verified failure states for evaluation and verified recovery trajectories synthesis for training. In Section 5.3, we fine-tune the same Qwen3.5-35B-A3B checkpoint on either successful executions or recovery trajectories from the same workflows, with a matched token budget. Recovery training raises EAR from 23.6% to 35.2% on held-out takeover states and clean-start Pass@3 from 28.4% to 34.9% on Odysseys. This comparison shows what the recovery data add under a standard training procedure.

We will connect the benchmark to prior work on imitation learning and reinforcement learning. DAgger obtains expert action labels at states visited by the learner, and on-policy distillation provides teacher feedback on learner-generated outputs. Both address a central issue in CUA-Recovery: once an agent makes an error, it can reach states rarely seen in successful demonstrations. We will discuss these connections alongside prior work on online RL for computer-use agents, such as DigiRL.

Long-horizon recovery makes online training costly in this setting. Smaller agents still rarely complete these workflows after fine-tuning, so online RL with completion rewards would collect few successful rollouts. Larger agents are more expensive to run and update. A practical starting point for these experiments would be to warm-start the policy with SFT, then collect fresh multi-application rollouts and either score them with a verifier for RL or obtain token-level teacher scores for on-policy distillation. CUA-Recovery provides the verified trajectories for that first stage and a common set of held-out failures for evaluating the resulting policies.

Reviewer_zHjHOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces CUA-Recovery, a benchmark and training suite for error recovery in long-horizon, multi-application computer-use workflows. Erroneous states are taken from the agents' own failures (619 failures from six agents; 2,250 human-verified takeover states at depths 0–25). The authors build it with ReRail, which composes MyPCBench tasks into executable DAG workflows with gold lineage and frozen verifiers. Fine-tuning Qwen3.5-35B-A3B on 8.6K verified recovery trajectories improves EAR and Rubric Score on held-out states, and also improves clean-start Pass@3 on MyPCBench and Odysseys.

Critical strengths
  • The gold-lineage idea is valuable and well engineered. Each task has executable ground truth at every step (Appendix B.2–B.4). Root causes are therefore located against a reference execution, not guessed by an LLM. The mutation test makes verifier coverage measurable instead of assumed (Eq. 3, Table 6; median R = 0.64, and dropping freezing collapses R to 0). This is the paper's most original and reusable contribution.
  • The setting is more realistic than prior recovery benchmarks. Workflows have persistent state, span multiple applications (67.5%), allow both GUI and CLI, and run to 60 steps on average. Takeover depths reach 25, beyond the 20-screenshot context. Together these let the benchmark probe a failure mode that short-horizon benchmarks cannot reach: the root-cause screenshot is no longer in the agent's context (Finding 4, Table 17).
  • The metrics are interface-fair and separate awareness from recovery. The judge reads terminal output, tool calls and screenshots, so CLI-heavy agents are not penalized (the 45.1%→85.0% GPT 5.5 example, p.3). EAR, Rubric Score and Pass@3 are reported separately, and Table 18 shows that awareness alone is a weak predictor of recovery.
  • The training ablation is well controlled. The success-only baseline is token-matched and drawn from the same workflows (Table 14). It supports the claim that the gains come from recovery supervision rather than more in-domain data.
  • Costs and prompts are documented transparently. Full API and GPU costs are given (Tables 4–5), along with all judge and takeover prompts (Appendix D) and an anonymized code link.
Critical weaknesses
  • Novelty relative to "Recovering Policy-Induced Errors: Benchmarking and Trajectory Synthesis for Robust GUI Agents" is overstated and under-attributed. That work already introduced several elements the submission presents as its own framing: 1. policy-induced errors as the target; 2. the coverage and horizon mismatches of existing training data; 3.the Error-Awareness Rate metric; 4. benchmark construction by root-cause localization, expert repair of the prefix, replay, and takeover at controlled depth; 5. the error-type and error-horizon comparison against AgentNet and GUI-Reflection.

Yet the only mention is one sentence calling it work on "short-horizon GUI tasks". The abstract, the introduction (l.77–78) and the conclusion (l.475) also claim that existing data contain no failures produced by the acting policy. That is inaccurate, because Bu et al. synthesize policy-induced recovery data. The statement "we compare with their reported statistics (Figure 3a)" (l.188–189) does not match Figure 3a, which contains no such comparison, and conflicts with Appendix C.2, which says the baseline corpora were relabeled by the authors. Several appendix sentences (l.1062–1064, l.1157–1159) and method sentences (l.212–213, l.279–280) closely follow the wording of that paper. I ask the authors to: (i) state explicitly which protocol components are inherited; (ii) reposition the contributions around what is genuinely new (persistent multi-app state, GUI+CLI, gold lineage and verifier freezing, depths beyond the context window); and (iii) rewrite the overlapping passages.

  • Cross-agent comparisons are confounded by agent-specific test sets of very different sizes. Each agent is evaluated only on its own failures (self-takeover). The per-agent state counts in Table 5 (69 / 139 / 175 / 613 / 683 / 571) sum to exactly 2,250. So GPT 5.5 is evaluated on about 69 states, roughly 11 per depth. Claude Opus 4.8 has about 23 per depth.

Two problems follow: 1. At these sizes, a 95% interval on a proportion near 20% is roughly ±20–25 points. Differences such as GPT 5.5's EAR 16.2→21.0 or Pass@3 25.5→18.9 between d=20 and d=25 (Finding 4, Table 17) are within noise. 2. The failure sets differ in kind: a strong agent's residual failures are its hardest cases. Statements like "Pass@3 reaches at most 25.5%" or "Claude reaches at most 7.7%" therefore are not comparable across agents.

No confidence intervals, standard deviations or per-cell n are reported in Table 2. The aggregate statistics, such as "planning and perception account for 57.3%", are dominated by the three open-source agents, which contribute about 83% of the states. Please report per-cell n and bootstrap CIs. Please also add a cross-agent condition on a shared state pool, which the paper already does for ReRail in Appendix F.5: every agent takes over the same states.

  • The clean-start transfer to MyPCBench is in-domain, and the depth comparisons are confounded. CUA-Recovery-Train is composed from the same 135 accepted MyPCBench source tasks that are then used to measure clean-start MyPCBench performance (l.357–360, Appendix B.2). The train/test split is by workflow (l.275–276), but source tasks, applications, personas and the database snapshot are shared. The 9.2%→24.5% MyPCBench gain is therefore largely in-domain, and Odysseys is the only real out-of-domain evidence. Please report MyPCBench results on source tasks excluded from all training workflows.

Takeover depth is also confounded in two ways:

- Context truncation: at d=25 the root-cause screenshot leaves the 20-screenshot window.
- Sample selection: the eligible set shrinks from 602 to 231 states and keeps only long, late-surfacing failures (Appendix G.2).

Trends across depth therefore mix three effects: depth, context loss, and a change in population. A fixed-subset analysis, restricted to failures eligible at all depths, would make these claims interpretable.

  • Judge validation is incomplete. Human–LLM agreement is reported only for the Pass@3 judgment, on 80 tasks from two open-source agents (Table 11). Recall is 0.75, so the judge systematically under-credits success, more so on EvoCUA (0.733). The Error-Awareness judge (Listing 4) is the basis of the paper's central metric, but no human agreement is reported for it. The partial Rubric Score judge is not validated either. Please add these validations, including for the API agents, which use the CLI more and whose trajectories are judged mainly from tool output. The judge, GPT 5.6-Terra, also comes from the same provider as the evaluated GPT 5.5. A self-preference check, or a second judge from a different model family, would address this.
  • The source of the training signal and the baselines are under-specified. A teacher model provides fallback continuations (3,350 attempts, Table 4), but the teacher is not named. The share of the 8,577 trajectories that come from the teacher is not reported. Without these numbers, the training result could reflect distillation from a stronger model rather than learning from self-generated recovery. Hints are removed from the input, but the targets r* were generated under hints. The filter "whether r* contains information that comes from hints" (Appendix F.4) is described only in one sentence. Finally, there is no comparison with any existing recovery or reflection training method (GUI-Reflection data, Agent-R-style self-training, or the tree-based synthesis of Bu et al.). The paper therefore shows that the new data help, but not that it beats existing approaches to recovery training.
  • Some definitions are inconsistent or unclear.
    • Pass@3 is defined as success in any one of three runs (l.144, l.227–229, l.1504), but the reported numbers do not fit this definition. Table 18 reports per-rollout final-verifier pass rates for GPT 5.5 of 34.7% (aware runs) and 31.1% (unaware runs), yet its highest Pass@3 in Table 2 is 25.5%. OpenCUA-72B shows the same pattern: 1.5–4.9% per rollout against at most 0.9% Pass@3. Under an "any of three" definition, a single-run pass rate cannot exceed Pass@3 on the same states. The gap may come from using two different judges, the programmatic final verifier for Table 18 and the LLM rubric judge for Pass@3, but this is not stated. Line 1726 also uses "in all three runs", which suggests an all-pass definition. Please clarify the definition and the judge used for each table.
    • Rubric Score is defined as "the score of that rollouts" (l.229); it is unclear which of the three runs it refers to.
    • Table 5 counts one self-takeover episode per state (e.g., 69 for GPT 5.5), which seems inconsistent with Pass@3 requiring three runs per state.
    • Line 1529 says all scaffolds support both CLI and GUI, but Table 12 and l.348–349 state that EvoCUA and OpenCUA are GUI-only.
    • Prompting is asymmetric across agents. Only GPT 5.5 and Claude Opus 4.8 receive the VERIFICATION behavior block, which tells the agent to check whether each step succeeded (Table 9). This block could directly raise EAR for those two agents. In addition, EAR depends on visible reasoning text, and the API models expose different amounts of it.
    • Table 16 repeats the self-takeover rows of Table 2 without adding new information.
    • Bold is used inconsistently in Table 2.
    • Typos: "as followed" (l.366), "ReRail-35B-A3Bhas" (l.1718, l.1725), "Tables 4, and 5" (l.704), and a dangling comma at the start of l.427.
Questions for authors
  • Was each test state run three times? Table 5 suggests a single run, and the per-rollout pass rates in Table 18 exceed Pass@3 in Table 2. Do the two tables use different judges? Please also report per-agent, per-depth test counts and confidence intervals.
  • Are the source tasks of training and test workflows disjoint? If not, how does ReRail-35B-A3B perform on MyPCBench source tasks absent from training?
  • Which teacher model was used, what fraction of CUA-Recovery-Train comes from it, and how do results change without its continuations?
3 comments

We thank the reviewer for the detailed review. We address each point below.

  • Q1: Novelty relative to "Recovering Policy-Induced Errors: Benchmarking and Trajectory Synthesis for Robust GUI Agents" is overstated and under-attributed.

Answer:

The fundamental difference is that our paper focuses on long-horizon tasks while RoTS focuses on short-horizon tasks. The tasks in our benchmark are all cross applications, GUI+CLI hybrid and contain situations that root cause may be out of the context windows. We also examine three parts of recovery: detecting that something went wrong, locating the cause, and repairing its downstream effects. The largest measured gain from recovery training is in detection: Error-Awareness Rate rises from 19.9% to 35.2%, while full recovery remains uncommon.

We also build datasets for training where data has persistent state and executable verification that make these failures testable. We compose 477 workflows, with a median of eight applications per workflow and 55% of dependency edges crossing application boundaries. We evaluate both GUI-only and GUI+CLI agents. We establish an executable reference for each workflow, freeze its verifier, and use mutation testing to screen workflows. A synthesized recovery is retained only if the frozen verifier confirms task completion.

While we borrow some ideas from RoTS, our findings show a lot of difference with their because many attributes will only show in long-horizon tasks. We build on Bu et al.’s focus on policy-induced errors, their coverage and horizon mismatch framing, their Error-Awareness Rate, and their benchmark protocol for root-cause localization, expert prefix repair, replay, and controlled-depth takeover. We will attribute these components where they are used, rewrite the flagged method and appendix passages, and revise the abstract, introduction, and conclusion to compare with the RoTS.

  • Q2: Cross-agent comparisons are confounded by agent-specific test sets of very different sizes.

Answer:

Self-takeover and cross-takeover. Firstly, we need to emphasis the tasks we use for all agents are the same. We agree that self-takeover results are not comparable across agents, because each agent resumes from its own failures. For example, a stronger agent's remaining failures are its hardest cases. But self-takeover answers a different question: can an agent recover from the errors it actually makes? Each agent resumes its own recorded failures without an error warning and gets an attempt to finish the workflow. As a benchmark paper, we use these results to show where recovery breaks down within each agent, and we do not use them to rank agents. Comparisons on identical states are made in the cross-takeover setting. We also want to prove the effectiveness of our training set under the different failure patterns situation by asking ReRail-35B to take over the same failure prefixes produced by three agents of different strength: Kimi-K3, EvoCUA-32B, and Qwen3.5-35B.

In the revision, we will report the self-takeover and cross-takeover in separate tables. The self-takeover table will give per-agent results with per-cell n and bootstrap 95% CIs following your suggestion. The cross-takeover table will hold all comparisons across agents.

The breakdown of error type per model is below:

AgentPlanning (%)Perception (%)Execution (%)Termination (%)Planning + Perception (%)
Claude Opus 4.821.127.88.942.248.9
GPT 5.525.057.512.55.082.5
Kimi-K326.532.47.433.858.8
EvoCUA-32B16.333.721.928.150.0
OpenCUA-72B24.937.622.115.362.6
Qwen3.5-35B17.736.316.429.654.0
All agents, equal weight21.937.614.925.759.5
API-based agents, equal weight24.239.29.627.063.4

We thank the reviewer for the detailed review, the answers of Q3 and Q4 are listed below:

  • Q3: The clean-start transfer to MyPCBench is in-domain, and the depth comparisons are confounded.

Answer:

CUA-Recovery holds out entire composed workflows, and the clean-start gains extend to a separate benchmark. The training and test splits contain distinct composed workflows. Every rollout, recovery trajectory, and takeover state derived from a given workflow stays in the same split at all depths. The splits share an application environment and can reuse MyPCBench source tasks as building blocks. We report MyPCBench as an in-domain clean-start evaluation and use Odysseys to measure transfer beyond that environment. We also reran the depth comparison on the same 231 failure trajectories, and Pass@3 remained slightly lower at (d=25).

Recovery supervision improves clean-start performance under matched training conditions. On MyPCBench, Pass@3 rises from 9.2% for the base model to 13.8% after training on successful executions and to 24.5% after training on recovery trajectories. The two fine-tuned models use the same training workflows and a matched token budget. On Odysseys, across 200 live-web tasks, the same three conditions achieve 24.1%, 28.4%, and 34.9%, respectively. The gain over the success-only control therefore extends to a separate environment. A MyPCBench source-task holdout would address a complementary question about generalization to unseen source tasks within MyPCBench.

The paired depth comparison holds the underlying failures fixed. On the 231 trajectories eligible at both depths, Pass@3 is 3.90% at (d=20) (9/231) and 3.03% at (d=25) (7/231). At (d=25), the root-cause screenshot has left the 20-screenshot history, while the consequences of the error can still be present in the application state. The paired result shows a modest decline when agents take over five steps later on the same failures. Loss of visual evidence is one possible contributor, alongside changes in the application state over those five steps.

  • Q4: Judge validation is incomplete.

Answer:

Our human audit covers all six agents, including all three API-based agents. The EAR judge agrees with humans on 92.2% of 245 rollouts (Cohen’s κ = 0.773), with over 90% agreement for every agent. The completion judge agrees with humans on 90.5% of 4,390 rubric decisions. Across 245 takeover states, humans and the judge each mark 29 as successful under Pass@3, with 22 in common, yielding pooled precision and recall of 0.76 each. In the same-sample check with Opus 5, both judges have identical EAR and Pass@3 bias on GPT 5.5, and Terra has a smaller positive Rubric Score bias (+1.2 vs. +1.7 percentage points). We find no evidence that Terra favors GPT 5.5 in this audit. The full judge results are:

Table A. Human agreement on error awareness

Evaluated agentRolloutsHuman-awareJudge-awareAgreement (%)Cohen’s κPrecisionRecall
GPT 5.53611991.70.7240.8890.727
Claude Opus 4.849151591.80.8080.8670.867
Kimi-K3336590.90.6730.8000.667
EvoCUA-32B397892.30.7530.7500.857
OpenCUA-72B466993.50.7630.6671.000
Qwen3.5-35B-A3B428895.20.8460.8750.875

Table B. Human agreement on rubric scoring and Pass@3

Evaluated agentStatesRolloutsRubric decisionsItem agreement (%)MAE (pp)Bias (pp)Human Pass@3Judge Pass@3PrecisionRecall
GPT 5.53610872393.226.1+1.222.2%19.4%0.860.75
Claude Opus 4.84914776489.149.5+2.226.5%28.6%0.790.85
Kimi-K3339958485.7911.8+2.912.1%12.1%0.750.75
EvoCUA-32B3911774888.509.8+2.45.1%5.1%0.500.50
OpenCUA-72B4613884191.447.6+1.72.2%2.2%0.000.00
Qwen3.5-35B-A3B4212673093.845.7+1.22.4%2.4%1.001.00

Table C. Comparison with a judge from a different model family

Evaluated agent groupEAR rolloutsCompletion statesJudgeEAR bias (pp)Rubric Score bias (pp)Pass@3 bias (pp)
GPT 5.53636GPT 5.6-Terra−5.56+1.2−2.78
GPT 5.53636Opus 5−5.56+1.7−2.78
Other API agents8282GPT 5.6-Terra−1.22+2.5+1.22
Other API agents8282Opus 5−1.22+2.4+1.22
Open-source agents127127GPT 5.6-Terra+3.15+1.70.00
Open-source agents127127Opus 5+4.72+1.70.00

We thank the reviewer for the detailed review, the answers of Q5 and Q6 are listed below:

  • Q5: The source of the training signal and the baselines are under-specified.

Answer:

Recovery supervision improves performance even when every training target comes from the base model. GPT 5.6-Terra is used only when the base model produces no verified continuation. Qwen3.5-35B-A3B generated 78.4% of the retained recovery trajectories, and GPT 5.6-Terra supplied the remaining 21.6% as fallback. Hints guide trajectory generation, and we screen the resulting targets for information carried over from those hints. Both training and evaluation inputs omit the hints. On the same 571 held-out takeover states, Pass@3 is 0.4% with success-only behavior cloning, 1.4% with one-round DAgger-style expert relabeling, and 1.9% with base-only Recovery SFT. These comparisons show that CUA-Recovery supports both expert action supervision on learner-visited states and SFT on verified recovery trajectories. Table 4 counts 3,350 fallback attempts. Of the 8,577 retained recovery trajectories, approximately 1,853 came from Terra and 6,724 from Qwen. We will report the teacher identity and retained counts alongside the generation budget.

Task completion and hint carryover are checked separately. The frozen verifier checks whether the continuation completes the task. The target filter checks whether the continuation carries over information from the generation hint. We will expand Appendix F.4 to describe the filter and give examples of how it handles hint-derived content.

The token-matched control tests the value of recovery trajectories on the same workflows. With the same starting checkpoint, training workflows, and token budget, recovery SFT raises EAR on Qwen’s failure states from 23.6% to 35.2% relative to success-only training. Clean-start Pass@3 also rises from 13.8% to 24.5% on MyPCBench and from 28.4% to 34.9% on Odysseys. The DAgger-style comparison complements this control by testing expert next-action labels on states visited by Qwen, using the same held-out evaluation cases.

  • Q6: Some definitions are inconsistent or unclear.

Answer:

Pass@3. We run three rollouts per takeover state. A state counts as successful if the completion judge finds that at least one rollout satisfies every rubric item. We will revise Appendix G to say “in at least one of three runs.”

Table 18. Table 2 reports state-level Pass@3, while Table 18 reports individual-rollout pass rates grouped by expressed error awareness. We will check Table 18 against the final evaluation records and specify the evaluator and rollout set used for each table.

Rubric Score. If any rollout satisfies every rubric item, we use its score. Otherwise, we average the weighted rubric scores of all three rollouts.

Table 5. The self-takeover rows estimate compute for one rollout per state using a representative runtime. We will label them as single-pass estimates and identify the counts as takeover states.

GUI and CLI. EvoCUA and OpenCUA ran with their official GUI-only harnesses. The other evaluated configurations expose both GUI and CLI, as listed in Table 12. We will align the appendix text with that table.

Prompting and EAR. GPT 5.5 and Claude Opus 4.8 receive the verification block listed in Table 9. EAR measures awareness expressed in the first three post-takeover reasoning segments available to the evaluator. We will clarify that both prompting and the amount of reasoning exposed can affect the measured rate.

Tables and copy edits. We will remove the duplicate self-takeover rows from Table 16, apply boldface consistently to the matched-state comparisons in Table 2, and fix the listed typos.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.