- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates when and how GUI agents should recover after failure (WHEN and HOW). The paper first conducted a counterfactual recovery experiment (counterfactual recovery) on 120 failed OSWorld trajectories, and found that recovering from a later checkpoint often had higher recovery utility than recovering from the earliest critical error (earliest critical error). Based on this observation, ReTIME uses a stateful WHEN Timing Gate to decide whether to continue execution or start recovery, and uses the HOW diagnosis model to classify the current state into a recovery taxonomy consisting of 21 leaf categories, and provides corresponding recovery guidance. The paper also proposes ReTIME-Bench, and increased the success rate of replicating the backbone on OSWorld-Verified from 26.8% to 35.2%.
Critical strengths
- The paper raises a meaningful question: Being able to observe errors does not necessarily mean that immediate recovery should be initiated.
- The paper not only reports the results of checkpoint-level classification, but also conducts end-to-end evaluation. Component ablation also provides some evidence indicating that both timing and diagnosis may affect the final task success rate.
Critical weaknesses
- I think the paper might lack sufficient novelty. Many papers already include runtime stopping and recovery, and there are already relatively similar existing works for failure diagnosis, recovery taxonomy, and corrective guidance. The paper should more clearly distinguish its own contributions. If my understanding is incorrect, please also correct it in the rebuttal.
- The current experiments have not well isolated the contribution of timing itself. Although ReTIME-Bench is isolated from LoRA training tasks, the training data, benchmark, and final evaluation are still within the OSWorld system, and in Table 2, the ReTIME-8B after task adaptation mainly compares with models without the same adaptation.
- I also have doubts about the reliability of the most important 78.3% results of the paper. Each trajectory compares multiple checkpoints, but the recovery utility of each checkpoint is estimated by a relatively small number of recovery rollouts, and as long as the estimated value of any subsequent checkpoint is strictly higher than the root, it will be counted as Post-root Better. Taking the maximum value among multiple noisy estimates may itself make this proportion higher, but the paper does not report confidence intervals or statistical tests.
Questions for authors
- Compared to other recent recovery systems, what are the specific technical contributions of ReTIME? It is hoped for a more precise definition.
- If possible, please add simple timing baselines such as immediate recovery and fixed-delay recovery, and try to use the same backbone. If possible, please provide the calibration set and actual calibration parameters of the Timing Gate, and it would be best to conduct tests in at least one non-OSWorld environment.
- In the main experiment, how many recovery rollouts were actually used for each checkpoint, and what is the variance of these utility estimates? If a non-zero margin relative to root recovery is required, or if there is a statistically reliable improvement, how many trajectories can be considered as Post-root Better? Please explain it more clearly.
7 comments
4. “If possible, please add simple timing baselines such as immediate recovery and fixed-delay recovery, and try to use the same backbone. If possible, please provide the calibration set and actual calibration parameters of the Timing Gate, and it would be best to conduct tests in at least one non-OSWorld environment.”
We added two baselines that use the same Qwen3-VL-8B-Instruct execution backbone and diagnosis-conditioned guidance. One intervenes when the online self-check flags an error. The other waits three decision steps after that signal. Under a 50-step budget, the mean official OSWorld-Verified evaluator scores are 33.81% for immediate recovery, 31.07% for fixed-delay recovery, and 41.28% for ReTIME. ReTIME scores above both simple rules. The timing ablation evaluates the intervention policies as a whole, with both simple rules anchored to the same self-check signal. And the revised ablation table is also attached below:
| Variant | Intervention rule | Recovery guidance | Score |
|---|---|---|---|
| Qwen3-VL-8B-Instruct | — | — | 29.59% |
| Immediate recovery | Self-check, (d=0) | Diagnosis-conditioned | 33.81% |
| Fixed-delay recovery | Self-check, (d=3) | Diagnosis-conditioned | 31.07% |
| Generic guidance | ReTIME Timing Gate | Generic recovery prompt | 26.44% |
| ReTIME-8B | ReTIME Timing Gate | Diagnosis-conditioned | 41.28% |
Appendix B.5 now reports the deployed gate parameters, including the evidence weights, decay, thresholds, persistence requirements, grace period, and recovery reserve. For example, the evidence decay is 0.7088, the base accumulated-evidence threshold is 1.7103, and the recovery reserve is 8 steps. We calibrated these parameters on a separate set of OSWorld recovery trajectories disjoint from ReTIME-Bench and fixed them before evaluation.
Our current evidence for end-to-end gains comes from OSWorld-Verified. We agree that evaluation beyond OSWorld would help establish how broadly these gains transfer. In future work, we plan to evaluate ReTIME on benchmarks such as Windows Agent Arena to test whether its timing and recovery guidance remain effective across operating systems and application workflows.
Response to Authors
Thank you for the detailed response and additional experiments. The newly added timing baselines strengthen the paper to some extent. However, I still believe that the current experiments do not sufficiently isolate the contribution of adaptive recovery timing itself.
I would appreciate further clarification on whether temporal evidence accumulation provides independent performance gains when keeping the failure detection signals and recovery guidance unchanged. Additionally, I would like to better understand the advantages of taxonomy-based recovery guidance over directly generating state-specific recovery guidance, and whether ReTIME introduces unnecessary interventions in tasks that the original backbone would otherwise complete successfully.
Therefore, while I appreciate the improvements made by the authors, I remain inclined to maintain my original rating at this stage. Given the limited time remaining before the discussion deadline, I would be willing to raise my rating if the authors could satisfactorily address any of the concerns raised above.
Thank you for your detailed feedback. To address your concerns about the reliability of the timing study, we clarify how the recovery rollouts were aggregated and report how often the observed gains exceed positive margins in the available records. To clarify ReTIME’s contribution, we explain how it differs from prior recovery systems and how to interpret the comparison in Table 2 given that our diagnoser was adapted for the task. In response to your requests for simpler timing baselines and calibration details, we added immediate and fixed-delay recovery to Table 4 and report the Timing Gate’s operating parameters in Appendix B.5. We also outline how we plan to evaluate transfer beyond OSWorld. We respond to each point below.
We ran three recovery rollouts per checkpoint in the main study and compared their mean utilities. We randomly sampled 120 eligible failed OSWorld trajectories, with 20 from each of six source agents, aiming for balanced coverage across task categories. Within each trajectory, we held the recovery policy, prompt, and total interaction budget fixed. In 94 of 120 trajectories (78.3%), at least one later checkpoint had higher observed mean utility than the earliest critical error. The mean best-over-root gain was 0.17, with source-agent group means ranging from 0.13 to 0.26.
To assess the magnitude of these gains, we applied positive margins to the available archival subset. Among the 94 records with a positive observed gain, 78.7% exceeded 0.05, 69.1% exceeded 0.10, and 31.9% exceeded 0.20.
The approximate pooled 95% Wilson interval for the main study’s observed proportion is 70.1%–84.8%. It describes uncertainty in the trajectory-level proportion under the reported comparison rule. Selecting the largest of several checkpoint means can still inflate the observed gains. The positive-margin counts describe the magnitude of the observed gains, while checkpoint-level variances and the number of statistically reliable improvements remain unquantified.
ReTIME separates failure recognition from the decision to intervene and diagnoses what needs repair at the chosen intervention point. We study the timing decision through controlled comparisons within the same failed GUI trajectory, then implement it online with a stateful Timing Gate and a checkpoint-specific diagnoser.
ReTIME-Bench uses a question-answering format so models can participate without a dedicated recovery module. Each checkpoint asks for a binary CONTINUE/RECOVER decision and a diagnosis from the same recovery taxonomy. This gives all evaluated systems the same checkpoints and answer space. We did not have access to comparably recovery-adapted versions of the baseline models, so Table 2 measures each system in its evaluated configuration. The effect of task adaptation is therefore outside the scope of that table. The end-to-end timing ablation in Table 4 addresses a separate question by comparing intervention rules with diagnosis-conditioned guidance held fixed.