- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper builds CUA-Recovery: 2,250 erroneous states from 619 failed rollouts of six agents on 100 multi-app desktop tasks, plus 8,577 training trajectories. Agents that take over mid-task recover in at most 25.5% of states (Pass@3); fine-tuning Qwen3.5-35B-A3B raises its average EAR from 19.9% to 35.2%. The problem matters, but the headline numbers rest on unreported, sometimes tiny cells, and the protocol is Bu et al.'s at longer horizons.
Critical strengths
- States come from the agents' own failures on trajectories averaging 60 steps, with takeover up to 25 steps after the root cause (page 6); Bu et al. (2026) stop at depth 5.
- The judge reads terminal output and tool calls as well as screenshots (page 3), so CLI and GUI routes score alike; Table 11 backs it with human agreement.
- The training set transfers: Odysseys Pass@3 rises from 24.1 to 34.9 (Table 3).
Critical weaknesses
- Table 2 gives no n per cell. Table 5 lists 69 GPT 5.5 self-takeover episodes, so at most 69 states over six depths, about a dozen per cell, underlie the headline 25.5%. Report n and a confidence interval per cell.
- Section 3 says each evaluation runs three times, yet Table 5's episode counts sum to 2,250, the number of states. State the runs per takeover state; if one, the metric is Pass@1.
- Finding 4 (page 8) compares d=20 with d=25 on different state subsets (273 versus 231 states, Appendix G.2). Rerun it on states eligible at both depths before blaming the context window.
- EAR is judged on the first three reasoning segments only (Appendix G.1; Section 5.1 omits this), and no human check of the judge (Listing 4) is reported. Scaffolds expose different amounts of reasoning text, a confound the judge cannot separate. Hand-label about 100 takeover runs.
- The teacher that writes recovery targets is never named (page 6; Table 4 counts 3,350 fallback attempts), nor its share of the targets. F.3 does not document the token matching page 9 claims. Name the teacher; draw the control from its successes.
- Page 4 calls Figure 3a's 84.3% and 71.2% execution shares Bu et al.'s reported statistics; Appendix C.2 says you labeled 100 trajectories per corpus. Bu et al. report no such numbers.
Questions for authors
Bu et al. (ICML 2026, "Recovering Policy-Induced Errors") already define the root-cause step, the error horizon, takeover at depth d, Error-Awareness Rate, and the comparison against AgentNet and GUI-Reflection. The paper reuses all of it, metric names included, and names them as the closest work (page 4) without saying the protocol is theirs. It adds depth 25, trajectories averaging 60 steps across 17 applications, a judge that reads application state, and a promised data release (Bu et al.'s remains "in preparation" on GitHub). Zhang et al. (ACL 2026, VeriGUI) also train on synthetic failure trajectories and report a Recovery Success Rate; Chung et al. (2026, WebStep) locate where web agents go wrong. Neither is cited.
The abstract's EAR and Rubric averages reproduce from Table 2; the ReRail Pass@3 average computes to 2.1, not Table 14's 1.9. Finding 3's association between awareness and recovery rests on 15.6% versus 12.2% macro averages with no counts and no test; add both.
Prose that reads as AI-written: "have therefore become critical bottlenecks" (page 1), "node density underpins the long-horizon nature" (page 19); the abstract lists eight performance figures.
2 comments
We thank the reviewer for detailed review. We answer the questions below:
We will explicitly credit Bu et al. for the protocol we adopt. This includes root-cause localization, the error horizon, controlled-depth takeover, Error-Awareness Rate, and the comparison with AgentNet and GUI-Reflection. We will also cite VeriGUI for training agents to verify and correct action effects using synthetic failures, and WebStep for locating decisive errors through semantic state tracking.
The correct equally weighted ReRail Pass@3 average is 2.1%. We will correct the 1.9% entry in Table 14 and specify in the caption that the six takeover depths receive equal weight. For Finding 3, the 15.6% and 12.2% figures are macro-averaged completion rates for rollouts with and without expressed awareness. We will report the successful and total rollout counts for each group by agent and test the association while accounting for repeated rollouts from the same source workflow.
The authors wrote the manuscript by themselves. Our disclosed use of AI was limited to polishing and rephrasing author-written text. Inferring authorship from two phrases is unwarranted.
We thank the reviewer for the detailed review. We address each point below.
Answer:
Table 5 uses 69 GPT 5.5 takeover states to estimate the computer for one pass through the evaluation set. For Pass@3, we run three rollouts per state and count the state as successful if at least one rollout completes the task. The number of eligible states varies by depth. For confident interval, the experiment is conducted under same seed for three times, we planned to add multi-seed experiment, but consider the high-cost and the domain tradition, single seed-multi turn experiment is already enough to reveal the problem.
Answer:
Each takeover state is evaluated with three rollouts. Pass@3 counts a state as successful if at least one of those rollouts completes the task. The self-takeover counts in Table 5 sum to 2,250 because the compute estimate uses one rollout per state, with a representative per-rollout runtime. We will label these rows as single-pass compute estimates and rename the “Episodes” column to “Takeover states” for the self-takeover rows.
Answer:
We rerun on the same 231 failure trajectories. Pass@3 is slightly lower at (d=25), from 3.90% (9/231) at (d=20) to 3.03% (7/231) at (d=25). The decline is modest and persists when the set of failures is held fixed. The (d=25) condition tests whether agents can recover after the root-cause screenshot leaves the 20-screenshot history. The application state can also change over the five intervening steps, so we interpret the paired result as a comparison of later takeover on the same failures, with reduced access to visual evidence as one possible contributor.
Answer:
The EAR judge agrees with human annotators on 92.2% of 245 takeover rollouts (Cohen’s κ = 0.773), with over 90% agreement for each of the six agents, including all three API-based agents. We will state in Section 5.1 that the judge scores whether the agent explicitly recognizes a problem in the inherited work within its first three post-takeover reasoning segments. The audit validates scoring on the reasoning available to the evaluator. Because scaffolds expose different amounts of reasoning, we interpret EAR as the rate of expressed awareness within that window under each evaluated scaffold.
Table A. Human agreement on error awareness
Answer:
GPT 5.6-Terra supplied 21.6% of the retained recovery targets, and training on Qwen-generated targets alone still improved recovery. On the same 571 held-out takeover states, base-only Recovery SFT improves Pass@3 from 0.4% (2/571) with success-only behavior cloning to 1.9% (11/571). EAR rises from 24.9% to 32.4%, and Rubric Score rises from 11.5% to 12.9%. This comparison shows that CUA-Recovery provides useful recovery supervision under standard SFT even when every recovery target comes from the base model.
Terra generated approximately 1,853 of the 8,577 retained trajectories. Qwen3.5-35B-A3B generated the remaining 6,724. The 3,350 entries in Table 4 count fallback attempts. We will report the teacher identity and retained counts in Section 4.
The success-only control starts from the same Qwen checkpoint and uses successful executions from the same training workflows. We will document in Appendix F.3 how we matched the training token budget for this control.
Answer:
The execution-error shares in Figure 3a, 84.3% for AgentNet and 71.2% for GUI-Reflection, come from our own annotations. Following Bu et al.’s comparison, we independently sampled and annotated 100 trajectories from each corpus using the protocol in Appendix C.2. A trajectory can receive multiple error labels, so each percentage is calculated over all error labels assigned within that corpus. We will revise the text and caption to state the source and denominator explicitly.