acceptodds
Under review as a conference paper at ICLR 2027

ReTIME: Learning When and How to Recover GUI Agents from Failures

Abstract

We present ReTIME, a framework for deciding when and how to recover GUI agents from failures. Recognizing an error does not establish whether recovery should begin immediately: further interaction may reveal useful evidence, but may also propagate the error and consume the remaining execution budget. In a controlled counterfactual study of 120 failed OSWorld trajectories, 78.3% con- tain a later checkpoint with higher recovery utility than the earliest critical error under the same recovery policy and interaction budget. ReTIME addresses this temporal variation with a stateful stopping policy that accumulates failure evi- dence, accounts for ongoing progress, and reserves a budget for recovery. Once intervention is triggered, a learned diagnosis model identifies the current recov- ery need and selects category-specific guidance for the execution agent. We also introduce ReTIME-Bench, comprising 672 checkpoints from 133 failure trajecto- ries, to evaluate intervention timing and recovery diagnosis separately and jointly. ReTIME-8B achieves 56.31% timing balanced accuracy and 31.55% diagnosis accuracy. On OSWorld-Verified, it achieves 35.2% task success under a 50-step budget, compared with 26.8% for its backbone Qwen3-VL-8B-Instruct.

Primary area:
applications to robotics, autonomy, planning
Keywords:
GUI-agent, Recover, MLLM

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Tgu6OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates when and how GUI agents should recover after failure (WHEN and HOW). The paper first conducted a counterfactual recovery experiment (counterfactual recovery) on 120 failed OSWorld trajectories, and found that recovering from a later checkpoint often had higher recovery utility than recovering from the earliest critical error (earliest critical error). Based on this observation, ReTIME uses a stateful WHEN Timing Gate to decide whether to continue execution or start recovery, and uses the HOW diagnosis model to classify the current state into a recovery taxonomy consisting of 21 leaf categories, and provides corresponding recovery guidance. The paper also proposes ReTIME-Bench, and increased the success rate of replicating the backbone on OSWorld-Verified from 26.8% to 35.2%.

Critical strengths
  1. The paper raises a meaningful question: Being able to observe errors does not necessarily mean that immediate recovery should be initiated.
  2. The paper not only reports the results of checkpoint-level classification, but also conducts end-to-end evaluation. Component ablation also provides some evidence indicating that both timing and diagnosis may affect the final task success rate.
Critical weaknesses
  1. I think the paper might lack sufficient novelty. Many papers already include runtime stopping and recovery, and there are already relatively similar existing works for failure diagnosis, recovery taxonomy, and corrective guidance. The paper should more clearly distinguish its own contributions. If my understanding is incorrect, please also correct it in the rebuttal.
  2. The current experiments have not well isolated the contribution of timing itself. Although ReTIME-Bench is isolated from LoRA training tasks, the training data, benchmark, and final evaluation are still within the OSWorld system, and in Table 2, the ReTIME-8B after task adaptation mainly compares with models without the same adaptation.
  3. I also have doubts about the reliability of the most important 78.3% results of the paper. Each trajectory compares multiple checkpoints, but the recovery utility of each checkpoint is estimated by a relatively small number of recovery rollouts, and as long as the estimated value of any subsequent checkpoint is strictly higher than the root, it will be counted as Post-root Better. Taking the maximum value among multiple noisy estimates may itself make this proportion higher, but the paper does not report confidence intervals or statistical tests.
Questions for authors
  1. Compared to other recent recovery systems, what are the specific technical contributions of ReTIME? It is hoped for a more precise definition.
  2. If possible, please add simple timing baselines such as immediate recovery and fixed-delay recovery, and try to use the same backbone. If possible, please provide the calibration set and actual calibration parameters of the Timing Gate, and it would be best to conduct tests in at least one non-OSWorld environment.
  3. In the main experiment, how many recovery rollouts were actually used for each checkpoint, and what is the variance of these utility estimates? If a non-zero margin relative to root recovery is required, or if there is a statistically reliable improvement, how many trajectories can be considered as Post-root Better? Please explain it more clearly.
7 comments

Thank you for your detailed feedback. To address your concerns about the reliability of the timing study, we clarify how the recovery rollouts were aggregated and report how often the observed gains exceed positive margins in the available records. To clarify ReTIME’s contribution, we explain how it differs from prior recovery systems and how to interpret the comparison in Table 2 given that our diagnoser was adapted for the task. In response to your requests for simpler timing baselines and calibration details, we added immediate and fixed-delay recovery to Table 4 and report the Timing Gate’s operating parameters in Appendix B.5. We also outline how we plan to evaluate transfer beyond OSWorld. We respond to each point below.

1. “How many recovery rollouts were actually used for each checkpoint, and what is the variance of these utility estimates? If a non-zero margin relative to root recovery is required, or if there is a statistically reliable improvement, how many trajectories can be considered as Post-root Better?”

We ran three recovery rollouts per checkpoint in the main study and compared their mean utilities. We randomly sampled 120 eligible failed OSWorld trajectories, with 20 from each of six source agents, aiming for balanced coverage across task categories. Within each trajectory, we held the recovery policy, prompt, and total interaction budget fixed. In 94 of 120 trajectories (78.3%), at least one later checkpoint had higher observed mean utility than the earliest critical error. The mean best-over-root gain was 0.17, with source-agent group means ranging from 0.13 to 0.26.

To assess the magnitude of these gains, we applied positive margins to the available archival subset. Among the 94 records with a positive observed gain, 78.7% exceeded 0.05, 69.1% exceeded 0.10, and 31.9% exceeded 0.20.

The approximate pooled 95% Wilson interval for the main study’s observed proportion is 70.1%–84.8%. It describes uncertainty in the trajectory-level proportion under the reported comparison rule. Selecting the largest of several checkpoint means can still inflate the observed gains. The positive-margin counts describe the magnitude of the observed gains, while checkpoint-level variances and the number of statistically reliable improvements remain unquantified.

2. “Compared to other recent recovery systems, what are the specific technical contributions of ReTIME?”

ReTIME separates failure recognition from the decision to intervene and diagnoses what needs repair at the chosen intervention point. We study the timing decision through controlled comparisons within the same failed GUI trajectory, then implement it online with a stateful Timing Gate and a checkpoint-specific diagnoser.

  • VLAA-GUI. Its Loop Breaker changes strategy in response to repeated failures or recurring screen states. ReTIME’s timing decision also covers failures that produce no loop. The gate uses observed failure evidence, ongoing progress, and the remaining budget to decide whether to intervene now or continue gathering evidence. VLAA-GUI
  • RoTS. GUI-RobustEval measures recovery at controlled error depths, and RoTS synthesizes recovery trajectories to fine-tune the execution policy. ReTIME compares recovery from the earliest critical error with recovery from later checkpoints along the same observed trajectory, using the same recovery policy, prompt, and total budget. For deployment, we keep the execution backbone frozen and add a calibrated gate and a LoRA-adapted diagnoser. Training is confined to the diagnoser, which avoids large-scale trajectory synthesis for retraining the execution policy. RoTS
  • CUADebug. CUADebug localizes the root cause and uses that diagnosis to guide re-execution. ReTIME identifies the first repair needed at the selected checkpoint. The same initial error can require different repairs as intervening actions change the GUI state. ReTIME-Bench measures intervention timing and checkpoint-specific diagnosis separately and jointly. CUADebug

3. “In Table 2, the ReTIME-8B after task adaptation mainly compares with models without the same adaptation.”

ReTIME-Bench uses a question-answering format so models can participate without a dedicated recovery module. Each checkpoint asks for a binary CONTINUE/RECOVER decision and a diagnosis from the same recovery taxonomy. This gives all evaluated systems the same checkpoints and answer space. We did not have access to comparably recovery-adapted versions of the baseline models, so Table 2 measures each system in its evaluated configuration. The effect of task adaptation is therefore outside the scope of that table. The end-to-end timing ablation in Table 4 addresses a separate question by comparing intervention rules with diagnosis-conditioned guidance held fixed.

4. “If possible, please add simple timing baselines such as immediate recovery and fixed-delay recovery, and try to use the same backbone. If possible, please provide the calibration set and actual calibration parameters of the Timing Gate, and it would be best to conduct tests in at least one non-OSWorld environment.”

We added two baselines that use the same Qwen3-VL-8B-Instruct execution backbone and diagnosis-conditioned guidance. One intervenes when the online self-check flags an error. The other waits three decision steps after that signal. Under a 50-step budget, the mean official OSWorld-Verified evaluator scores are 33.81% for immediate recovery, 31.07% for fixed-delay recovery, and 41.28% for ReTIME. ReTIME scores above both simple rules. The timing ablation evaluates the intervention policies as a whole, with both simple rules anchored to the same self-check signal. And the revised ablation table is also attached below:

VariantIntervention ruleRecovery guidanceScore
Qwen3-VL-8B-Instruct——29.59%
Immediate recoverySelf-check, (d=0)Diagnosis-conditioned33.81%
Fixed-delay recoverySelf-check, (d=3)Diagnosis-conditioned31.07%
Generic guidanceReTIME Timing GateGeneric recovery prompt26.44%
ReTIME-8BReTIME Timing GateDiagnosis-conditioned41.28%

Appendix B.5 now reports the deployed gate parameters, including the evidence weights, decay, thresholds, persistence requirements, grace period, and recovery reserve. For example, the evidence decay is 0.7088, the base accumulated-evidence threshold is 1.7103, and the recovery reserve is 8 steps. We calibrated these parameters on a separate set of OSWorld recovery trajectories disjoint from ReTIME-Bench and fixed them before evaluation.

Our current evidence for end-to-end gains comes from OSWorld-Verified. We agree that evaluation beyond OSWorld would help establish how broadly these gains transfer. In future work, we plan to evaluate ReTIME on benchmarks such as Windows Agent Arena to test whether its timing and recovery guidance remain effective across operating systems and application workflows.

Reviewer_Tgu6OpenReview ↗

Response to Authors

Thank you for the detailed response and additional experiments. The newly added timing baselines strengthen the paper to some extent. However, I still believe that the current experiments do not sufficiently isolate the contribution of adaptive recovery timing itself.

I would appreciate further clarification on whether temporal evidence accumulation provides independent performance gains when keeping the failure detection signals and recovery guidance unchanged. Additionally, I would like to better understand the advantages of taxonomy-based recovery guidance over directly generating state-specific recovery guidance, and whether ReTIME introduces unnecessary interventions in tasks that the original backbone would otherwise complete successfully.

Therefore, while I appreciate the improvements made by the authors, I remain inclined to maintain my original rating at this stage. Given the limited time remaining before the discussion deadline, I would be willing to raise my rating if the authors could satisfactorily address any of the concerns raised above.

Reviewer_WTTeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ReTIME, a framework for deciding on how to recover from failures in GUI agents. They examine unsuccessful trajectories from OSWorld and quantify how intervention timing affects the agent’s recovery. Based on this, they propose their framework ReTIME as well as a benchmark ReTIME-Bench. RETime basically has two modules, one which decides to continue or recover, and a second module which decides how to recover and provides guidance to the executing agent.

Critical strengths

The problem that this paper studies is practical and relevant. There is also not an abundance of prior work on this subject so the analysis and benchmark contributions could also be useful to the community for others building on the work.

The proposed intervention generally makes sense and is clearly motivated by the analysis. I also appreciated the ablations of the two components.

Critical weaknesses

I think that the approach is sensible, but it would be good to see comparisons with simpler baselines. For instance, the timing gate seems mostly heuristic-driven, it would be good to compare it with simpler triggers such as a spaced trigger, or trigger on repeated actions or failures. Additionally for the HOW module, I wonder how much the categories are really necessary, and if similar gains can be realized by just giving the base GUI agent a recovery prompt at all (e.g. about how to recover from errors if it encounters them).

I think that some related work may be missing from the discussion section, in particular [1] which also studies online failure forecasting, while [2] studies the risk of disrupting runs throughout intervention.

[1] https://openreview.net/pdf?id=d3Qq4weuwz

[2] https://arxiv.org/abs/2602.03338

Questions for authors

N/A

2 comments

Thank you for highlighting the practical relevance of recovery timing. To address your request for simpler timing baselines, we added immediate and fixed-delay recovery with the same diagnosis-conditioned guidance. To test whether a generic recovery prompt can yield similar gains, we retained the ReTIME Timing Gate and replaced diagnosis-conditioned guidance with a generic prompt. Table 4 reports both comparisons. We also added the two papers you suggested to Related Work and explain how our study of recovery timing complements their work on failure forecasting and the effects of intervention. We respond to each point below.

1. “It would be good to compare it with simpler triggers such as a spaced trigger, or trigger on repeated actions or failures.”

We added two self-check baselines. When the online self-check flags an error, Immediate recovery intervenes at that decision step and Fixed-delay recovery waits three additional decision steps. Both use the same diagnosis-conditioned guidance as ReTIME. The delay is measured from the same error signal in both baselines, so the comparison tests what happens when the agent intervenes immediately or waits after recognizing a problem. The self-check can flag errors across failure types, including errors without repeated actions.

The updated Table 4 reports mean official OSWorld-Verified evaluator scores, including partial credit, under a 50-step budget.

VariantIntervention ruleRecovery guidanceScore
Qwen3-VL-8B-Instruct——29.59%
Immediate recoverySelf-check, (d=0)Diagnosis-conditioned33.81%
Fixed-delay recoverySelf-check, (d=3)Diagnosis-conditioned31.07%
Generic guidanceReTIME Timing GateGeneric recovery prompt26.44%
ReTIME-8BReTIME Timing GateDiagnosis-conditioned41.28%

ReTIME scores above both simple timing baselines. This supports using a timing policy that responds to the execution history and remaining budget. A fixed delay of three steps did not improve the aggregate score over immediate recovery.

2. “For the HOW module, I wonder how much the categories are really necessary, and if similar gains can be realized by just giving the base GUI agent a recovery prompt at all.”

The categories help the agent choose what to do next at the intervention checkpoint. We drew on GUI failure taxonomies and analyses in RoTS and CUADebug and organized our categories around the recovery needs of the current state. For example, an agent that has already achieved the requested state needs to verify and preserve its work, while an agent stuck on an ineffective action needs to change its approach. The HOW module uses the predicted category to select guidance for that objective.

A generic recovery prompt did not yield similar gains. With the ReTIME Timing Gate and a generic task-resumption prompt, the agent scored 26.44% versus 29.59% for the backbone on the same 361 OSWorld-Verified tasks under a 50-step budget.

Here are two specific examples:

  1. In a GIMP task, the agent correctly flipped the image before intervention. After receiving the generic recovery prompt, it flipped the image back and continued alternating between orientations until the budget ran out.
  2. In a terminal task, the agent kept clicking the same location after intervention, while the backbone completed the task by opening the terminal with a keyboard shortcut.

With the same Timing Gate, generic task-resumption guidance yielded a lower task success rate than diagnosis-conditioned guidance. The two traces illustrate what the categories add in practice. They help the agent choose between recovery objectives such as verifying and preserving a correct edit or switching away from an ineffective action.

3. “I think that some related work may be missing from the discussion section.”

We thank the reviewer for pointing us to these two papers and have added both to Related Work. Sengar et al. predict from a partial execution trace whether continued execution under the current policy will end in failure. Vasudev et al. show that interventions triggered by an accurate failure predictor can disrupt runs that would otherwise succeed. Sengar et al. · Vasudev et al.

ReTIME contributes a controlled way to measure how the choice of intervention point affects GUI recovery. Within the same failed trajectory, we initiate recovery at different checkpoints while holding the recovery policy, prompt, and interaction budget fixed. This lets us examine when further interaction reveals evidence that helps recovery and when it allows errors to propagate. At runtime, ReTIME turns this trade-off into a sequential stopping decision based on failure evidence, ongoing progress, and the remaining budget. Once the gate triggers, a separate diagnoser identifies the first recovery objective from the GUI state at that checkpoint. The same initial error can therefore lead to different recovery guidance as the state evolves.

Reviewer_JkhJOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies when and how a GUI agent should recover from execution failures. The authors argue that detecting an error does not determine the optimal intervention point: continued execution may reveal clarifying evidence, but may also propagate errors and consume the recovery budget. In a counterfactual study of 120 failed OSWorld trajectories, they replay the source agent's actions to reconstruct candidate GUI states, launch the same recovery policy from each checkpoint under a fixed budget, and report that 78.3% of trajectories contain a post-root checkpoint with strictly higher recovery utility than recovery from the earliest critical error (mean best-over-root gain 0.17). This motivates ReTIME, which separates the WHEN and HOW of recovery.

Critical strengths

The paper identifies a genuine, under-studied problem: intervention timing as a sequential stopping decision, decoupled from failure detection. The counterfactual protocol is carefully controlled (same recovery policy, prompt, budget, and rollout protocol across checkpoints, prefix-only information), and the timing observation is tested across six source agents, two recovery policies, and several utility parameterizations.

Critical weaknesses

The central empirical claims rest on small-scale, author-defined measurement. The 78.3% finding uses only 120 trajectories and a hand-crafted structured recovery utility (component weights, failure factor q=0.70, and a post-hoc prompt-assistance correction PAF are all set by the authors); the sensitivity analysis perturbs narrow variants on the same frozen rollouts, and ReTIME-Bench (672 checkpoints, 133 trajectories) inherits this machinery. No inter-annotator agreement is reported for diagnosis labels. The learned components are near-chance: 56.31% timing balanced accuracy is modestly above random, leaf accuracy 31.55%, macro-F1 23.95%, joint accuracy 17.86%, and the WHEN module is not learned at all but a fixed priority rule with calibrated weights. The "learning when" claim is therefore overstated.

End-to-end evidence is weak and confounded. The +8.4-point gain is measured against the authors' own reproduction of the backbone (26.8%), and no ablation isolates diagnosis from the mere presence of a structured recovery prompt (e.g., a trivial or fixed diagnosis in the same template).

Questions for authors

see the weakness

4 comments

Thank you for your detailed review. Regarding your concerns about the scale and scoring of our counterfactual study, we would like to clarify its central contribution. We show that recovery outcomes can vary across intervention points along the same failed GUI trajectory, even when the recovery policy, prompt, and total interaction budget are held fixed. Below, we explain how we sampled the trajectories and tested the sensitivity of this finding to the scoring choices. We also clarify how ReTIME-Bench labels are constructed and how the Timing Gate is calibrated. To address your concerns about the end-to-end evidence, we report the completed backbone evaluation and new ablations for intervention timing and recovery guidance.

1 — Study scale and reliability

“The 78.3% finding uses only 120 trajectories.”

We agree that 120 trajectories constitute a modest sample. The study supports a specific finding within the evaluated settings. Later intervention points often yielded higher observed recovery utility, and this pattern appeared across source agents and recovery policies.

  • Sampling and controls. We randomly sampled eligible failures within OSWorld task categories, aiming for balanced coverage, with 20 trajectories from each of six source agents. We averaged three recovery rollouts per checkpoint and held the recovery policy, prompt, and total interaction budget fixed within each trajectory.
  • Magnitude and consistency. A later checkpoint scored higher than the earliest critical error in 94 of 120 trajectories (78.3%). The observed proportions ranged from 65% to 90% across source agents, with a pooled mean gain of 0.17 on the 0–1 utility scale. On the same 50 trajectories, the two recovery policies yielded proportions of 78% and 76%.
  • Uncertainty. The approximate pooled 95% Wilson interval is 70.1%–84.8%. It describes uncertainty in the observed proportion under our comparison rule. Selecting the largest of several checkpoint means can favor later checkpoints when rollout scores vary.

We therefore interpret 78.3% as the frequency observed under this protocol. The results across source agents and recovery policies support studying when later intervention can help.

2 — Recovery utility and sensitivity analysis

“The sensitivity analysis perturbs narrow variants on the same frozen rollouts.”

The main timing result is stable under substantial changes to the component weights and failure factor. We designed the rubric to capture recovery outcomes that final task completion alone can miss.

  • Why use a recovery rubric? We are unaware of a standardized partial-credit evaluator that can be applied directly to these recovery rollouts. OSWorld 2.0 scores task-specific criteria, WindowsWorld measures intermediate subgoals, and CUADebug tracks whether re-execution fixes the original error or introduces new ones. OSWorld 2.0 · WindowsWorld · CUADebug
  • What does the rubric measure? It credits task progress, error correction, strategy redirection, verification, and useful efficiency, and penalizes damage to valid work and stagnation. The failure factor gives greater weight to full completion. The prompt-assistance factor discounts utility by up to 15% to account for the recovery prompt already telling the agent that execution failed.
  • How much do the results change? Equal weighting reduces the combined weight of task progress and recovery correctness from 65% to 40%, and the count changes from 94 to 93 out of 120. Lowering the failure factor from 0.70 to 0.50 or 0.25 leaves the pooled count at 94. Appendix A.8 holds the rollouts and prompt-assistance factor fixed to measure sensitivity to the tested scoring changes.

The end-to-end evaluation uses the official OSWorld-Verified evaluator and provides a separate measure of system performance.

3 — What ReTIME-Bench inherits from the recovery utility

“ReTIME-Bench (672 checkpoints, 133 trajectories) inherits this machinery.”

The recovery utility determines the WHEN labels. The HOW labels come from a separate annotation process.

  • WHEN. Appendix F.1 derives CONTINUE and RECOVER from counterfactual recovery outcomes.
  • HOW. Appendix F.2 assigns a diagnosis from the task instruction, GUI state, and execution history visible at the checkpoint. The label identifies the first recovery objective supported by that evidence.

The dependence on the utility therefore applies to timing supervision. Diagnosis supervision comes from the recovery taxonomy and the observable state.

4 — Agreement on diagnosis labels

“No inter-annotator agreement is reported for diagnosis labels.”

Two human annotators reviewed the diagnosis checkpoints separately, and an AI system provided an additional set of labels. In our human–AI comparison, the exact diagnosis leaf matched on approximately 95% of the checkpoints compared. The human annotators used the taxonomy guidelines to resolve disagreements and determine the final labels.

5 — Interpretation of the checkpoint scores

“The learned components are near-chance.”

The absolute scores leave substantial room for improvement. ReTIME-8B nevertheless improves over the evaluated models on balanced timing accuracy and fine-grained diagnosis under the same benchmark protocol.

  • WHEN. ReTIME-8B achieves 56.31% balanced accuracy on the binary timing decision, compared with 53.28% for GPT-5.4 and 52.87% for Claude Sonnet 4.5.
  • HOW and joint accuracy. ReTIME-8B achieves 31.55% leaf accuracy and 23.95% leaf macro-F1, compared with 23.96% and 18.17% for GPT-5.4. Its 17.86% joint accuracy requires both the timing decision and the exact diagnosis leaf to be correct at the same checkpoint.
  • Context for diagnosis difficulty. On the Claude-agent split of CUAErrorBench, CUADebug’s tool-augmented Gemini 2.5 Pro improves exact subtype accuracy from 29.9% to 36.8% and reaches 19.4% when both the subtype and exact root-cause step must match. CUADebug, Table 2

The end-to-end evaluation complements these results. With a 50-step budget on OSWorld-Verified, full ReTIME scores 41.28% and the reproduced backbone scores 29.59%. ReTIME-Bench measures where further improvements in timing and diagnosis are possible.

6 — What is learned in ReTIME

“The ‘learning when’ claim is therefore overstated.”

We agree that “learning when” can give the impression that the timing policy itself is learned. As clarified in Section 4.2, WHEN is a stateful stopping policy with a fixed decision structure. Its weights and thresholds are calibrated on recovery trajectories separate from ReTIME-Bench and held fixed during evaluation. The policy uses accumulated failure evidence, ongoing progress, and the remaining interaction budget to decide when to intervene. HOW is the learned component. We adapt the diagnoser with LoRA to identify the recovery need at the intervention checkpoint and select category-specific guidance.

In future work, we plan to explore training a stopping policy on counterfactual recovery outcomes while limiting its runtime inputs to information available at the current step. Collecting this supervision is costly because it requires reconstructing the GUI state at multiple checkpoints and running repeated recovery attempts under a common protocol.

7 — Reproduced backbone performance

“The +8.4-point gain is measured against the authors’ own reproduction of the backbone (26.8%).”

We agree that the reproduced backbone score matters for interpreting the end-to-end gain. The original submission reported 26.8% for Qwen3-VL-8B-Instruct. After re-running all 361 OSWorld validation tasks, the score of the replicated backbone model was 29.59%. In the same set of tasks, the score of ReTIME-8B reached 41.28%. Under the same 50-step budget and official scoring protocol, the difference between the two was 11.69 percentage points. For the updated comparison, both the reproduced backbone and ReTIME-8B used the same execution model and the decoding settings reported for 8B Instruct in the Qwen3-VL Technical Report. The updated comparison shows that ReTIME retains an advantage over the stronger reproduced backbone on the full task set.

8 — Diagnosis-conditioned guidance versus a generic recovery prompt

“No ablation isolates diagnosis from the mere presence of a structured recovery prompt.”

We added a generic-guidance ablation in Table 4, whose result we also show in the table below.

VariantIntervention ruleRecovery guidanceScore
Qwen3-VL-8B-Instruct——29.59%
Immediate recoverySelf-check, (d=0)Diagnosis-conditioned33.81%
Fixed-delay recoverySelf-check, (d=3)Diagnosis-conditioned31.07%
Generic guidanceReTIME Timing GateGeneric recovery prompt26.44%
ReTIME-8BReTIME Timing GateDiagnosis-conditioned41.28%

This variant uses the ReTIME Timing Gate and gives the agent a task-resumption prompt at intervention. The prompt asks the agent to inspect the current GUI state, preserve valid progress where possible, and verify the result. On OSWorld-Verified under a 50-step budget, generic guidance scored 26.44%, compared with 29.59% for the reproduced backbone and 41.28% for full ReTIME. The generic prompt did not reproduce the gains of diagnosis-conditioned guidance.

Here are two specific examples:

  1. In a GIMP task, the agent correctly flipped the image before intervention. After receiving the generic recovery prompt, it flipped the image back and continued alternating between orientations until the budget ran out.
  2. In a terminal task, the agent kept clicking the same location after intervention, while the backbone completed the task by opening the terminal with a keyboard shortcut.

These traces illustrate two different recovery objectives that a generic prompt leaves the agent to infer: verify and preserve a completed edit, or switch away from an ineffective action.

The reviewer’s proposed control would keep the structured template identical and substitute a fixed diagnosis. Our ablation compares generic task-resumption guidance with the full diagnosis-conditioned guidance. The proposed control would further isolate the contribution of the predicted diagnosis within that template.

Reviewer_JkhJOpenReview ↗

response

Thanks for your thorough replies. The additional experimental evidence and concrete trajectory analyses have addressed some of my concerns.

Comments on the paper

2 comments
  1. Added taxonomy and intervention-risk analyses (WTTe, Tgu6). We explained how each recovery category specifies a repair procedure, verification criterion, resumption condition, and safeguards, with definitions and templates in Appendices C–D. To examine disruption of tasks the backbone can solve, we provided paired outcomes across all 361 tasks, an application-level breakdown, and inspected trajectories. ReTIME retains full credit on 96/101 backbone successes and earns full credit on 46 additional tasks. The examples cover a potentially redundant intervention, an execution divergence that preceded intervention, and successful completion after guidance prompted a change of route. We also clarified how the NORMAL category provides a further check after the gate flags a possible problem, allowing execution to continue without corrective guidance.
  2. Clarified benchmark annotation and task adaptation (JkhJ, Tgu6). WHEN labels derive from counterfactual recovery outcomes, while HOW labels independently describe the recovery need supported by checkpoint-visible evidence. We described the two-human review process, additional AI labeling, and human resolution of disagreements. For Table 2, we clarified that it compares systems in their evaluated configurations, including ReTIME’s task-adapted diagnoser. The shared question-answering format supplies the same checkpoints and answer space; isolating the effect of task adaptation requires further controls.
  3. Clarified the learned components, operating parameters, and evaluation scope (JkhJ, Tgu6). Section 4.2 explicitly states that WHEN uses a fixed decision structure with calibrated parameters, HOW uses a LoRA-adapted diagnoser, and the execution backbone remains frozen. Appendix B.5 now reports the deployed evidence weights, decay, thresholds, persistence requirements, grace period, and recovery reserve. We also clarified that current end-to-end evidence comes from OSWorld-Verified, with cross-environment evaluation remaining future work.

We thank the Area Chair and reviewers for their careful reading and constructive feedback. The reviews recognized the value of studying recovery timing, the controls in our counterfactual study, and the potential usefulness of the analysis and benchmark. We would like to further clarify our central contribution: improving task execution with a frozen GUI backbone by choosing when to intervene and diagnosing the recovery need at that point. We compare intervention checkpoints within the same failed trajectory while holding the recovery policy, prompt, and total interaction budget fixed, and use these findings to develop ReTIME and ReTIME-Bench. In response to the reviewers’ concerns, we added experiments and clarified the study, method, and evaluation, as summarized below.

  1. Added timing and guidance controls (JkhJ, WTTe, Tgu6). We added immediate recovery, fixed three-step delay, and generic-guidance ablations. Under a 50-step budget, their mean official evaluator scores are 33.81%, 31.07%, and 26.44%, respectively, compared with 41.28% for full ReTIME and 29.59% for the reproduced backbone. The timing controls retain the same backbone and diagnosis-conditioned guidance; the generic-guidance control retains the ReTIME Timing Gate. These results appear in revised Table 4 and support the practical value of the complete intervention policy and diagnosis-conditioned guidance. We also added RoTS-7B’s published 36.3% result, explicitly identifying its source.

  2. Expanded the statistical interpretation and sensitivity analysis (JkhJ, Tgu6). We clarified sampling across six source agents, averaging three recovery rollouts per checkpoint, and the fixed recovery policy and total budget. For the observed 94/120 (78.3%) result, we reported a trajectory-level confidence interval and explained the uncertainty associated with comparing multiple checkpoint estimates. To examine gains beyond small positive differences, we added positive-margin analysis. Among the positive-gain records analyzed, 78.7% had ΔU > 0.05, 69.1% had ΔU > 0.10, and 31.9% had ΔU > 0.20. These proportions describe gain magnitudes within that supplementary sample. In the utility sensitivity analysis, equal component weighting changes the main-study count from 94/120 to 93/120, while reducing the failure factor to 0.50 or 0.25 retains 94/120.

  3. Strengthened the mechanism analysis and research positioning (JkhJ, Tgu6). Revised Section 3.3 and the new Figure 2 explain how subsequent interaction can make a failure explicit, reject an incorrect hypothesis, or expose an accessible repair target. In the VS Code example, the same recovery policy repeatedly fails to edit a settings field at the earlier checkpoint, while the malformed configuration exposed later provides a direct repair opportunity. We clarified how ReTIME uses evolving evidence and recovery opportunities to improve the practical performance of a frozen execution model. Related Work now explains its relationship to VLAA-GUI, RoTS, CUADebug, and the forecasting and intervention papers suggested by WTTe.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.