- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes Speculative Rollback Correction (SRC) for training GUI agents. Rather than treating every deviation from an expert demonstration as an error, it defines harmful actions as those that reduce the task’s recoverability. During data collection, a student executes a short segment, a teacher identifies the earliest harmful step, and the environment is reset and replayed so the teacher can provide a correction. Verified trajectories and correction actions are then used to train a student that operates without the teacher at deployment. The authors report improved success rates on three GUI benchmarks, although the collection process depends on reliable replay and teacher judgments.
Critical strengths
- Well-motivated problem. Identifying and correcting consequential errors during GUI agent execution is a worthwhile research problem.
- Strong empirical results. SRC improves teacher-free success rates over Expert SFT on all three evaluated benchmarks, and the paper includes controlled comparisons of corrective labels, rollback localization, and replay reliability.
Critical weaknesses
- Unclear introduction. The introduction moves into methodological details and formulas too early, making the core problem difficult to grasp.
- Unclear role of the theory. The extensive formulas and proofs make the method harder to follow, while their necessity for designing or implementing SRC is not clearly established. If the theoretical results are the main contribution, the paper may be better positioned as theory-oriented work.
Questions for authors
Related work and motivation: Could the authors discuss more closely related GUI agent studies, particularly those using reflection or corrective data? Multiple valid paths to a task goal also occur in general agents. Why is this problem framed primarily as GUI-specific?
Replay reliability: Replaying GUI actions may fail to restore the intended state. How are replay mismatches detected and handled? Have the authors considered state snapshots, such as Docker snapshots? How do replay failures affect corrective-data quality and final success rates?
Teacher judgment bias: The teacher reviews short segments of student behavior and decides whether to accept or roll back, rather than performing a full rollout. What biases might this introduce, and how are they controlled?
Localization accuracy: Is there independent annotation or another means of measuring whether the teacher identifies the earliest harmful step correctly? Outperforming random rollback shows that localization is useful, but does not directly establish its accuracy.
Initial data source: Where do the initial Expert SFT trajectories come from?
Teacher-model ablation: How sensitive is SRC to the teacher VLM’s GUI capabilities? How do localization quality and final student performance change when different teacher models are used?
Training tasks and generalization: Where do the tasks used for online collection come from, and how are they separated from evaluation tasks? Does the method generalize to unseen applications or task distributions?
Trajectory length: Could the authors report the average number of steps and the length distribution of SRC-collected trajectories and deployed-student trajectories?
4 comments
Response to Reviewer ST2V (2/4): Replay Handling, Teacher Bias, and Localization Utility
Prefix-length breakdown (= Table 4 in the revision).
| Replayed prefix length on WebArena-Infinity | Agreement |
|---|---|
| 0–2 actions | 99.0% |
| 3–7 actions | 95.5% |
| 8 or more actions | 92.4% |
The WebArena-Infinity agreement denominator excludes 85 rollbacks with missing snapshots. Both missing observations and declining fidelity for longer prefixes limit the interpretation of these rates.
Two safeguards reduce, but do not eliminate, the consequences of replay drift:
- The corrective action is generated from a fresh screenshot of the actually restored state, rather than solely from the pre-rollback screenshot.
- A retained completed trajectory must pass the hard task verifier.
These safeguards do not establish that the original state was recovered exactly, that historical observations remain aligned, or that every local label is correct. We do not have an isolated ablation quantifying the effect of replay mismatch on final student SR.
OSWorld restores the initial environment from a VM snapshot; SRC then replays the accepted prefix rather than storing a snapshot after every action. Intermediate VM or container snapshots could reduce replay drift, but their benefit depends on whether they capture all relevant application and external-service state. We will discuss this option without claiming that container snapshots necessarily eliminate all mismatches.
Q3. What biases arise from reviewing only short segments?
The main risks and current mitigations are:
| Risk | Current mitigation | Remaining limitation |
|---|---|---|
| Delayed consequences are missed | Include preceding context; verify completed trajectories | A short window cannot reveal every downstream failure |
| Valid exploration is rejected | Prompt explicitly allows reasonable alternative routes and uncertain cases | Efficiency preferences can still remove useful exploration |
| Teacher misunderstands the GUI state | Review screenshots and use a fresh observation for correction | Errors remain dependent on teacher capability |
| Successful trajectories are overrepresented | Preserve multiple verifier-passing branches | Success filtering does not recover rejected alternatives |
The horizon ablation illustrates the trade-off: yields 45.6% collector SR with 3,324 teacher queries, whereas yields 51.9% with 2,529 queries (Appendix C.7). This supports the chosen horizon empirically; it does not prove that teacher myopia has been eliminated or that is universally optimal.
Q4. Is the earliest harmful step independently validated?
Not directly. We agree that better performance than random rollback establishes utility, not localization accuracy.
The current evidence comprises the controlled random-rollback comparison, distributions of teacher-selected positions, and continuation diagnostics. None provides an independently annotated ground-truth harmful position. In particular, the 5.4% success rate when retaining an interior-flagged segment measures success under a particular student continuation, not optimal recoverability.
The closest available diagnostic is the 167-branch replay study in our response to MV6h (W2/Q5–Q6, part b): on branches where the teacher selected an interior position, teacher-selected rollback succeeded on 27/167 branches versus 19/167 for forced segment-start rollback () and 9/167 for keeping the flagged actions (). This indicates that the selected position is useful, but it is still not ground-truth accuracy: comparing rollback positions is also affected by correction quality and continuation stochasticity. Such a comparison alone would not prove that the selected position is the earliest harmful action. We will describe the result as teacher-guided localization utility, and remove claims of measured earliest-error accuracy.
Q5. Where do the initial Expert SFT trajectories come from?
For WebArena-Infinity, the expert source combines successful Kimi-K2.5 and Qwen3.5 Plus trajectories from the public WebArena-Infinity trajectory dataset. Task coverage is merged using the (environment, task_id) key, rather than adding the two models' trajectory counts. The current public manifest covers 843 distinct task IDs across these two models; after screening for usable action-space outputs, required screenshots, and trajectory completeness, our eligible source coverage is 813 distinct tasks. SRC is collected on the remaining 1,620 − 813 = 807 tasks. This is a valid-source-coverage criterion, not a claim that a teacher failed on every one of those tasks.
Response to Reviewer ST2V (3/4): Data Preparation and Teacher Sensitivity
Data preparation also filters unsupported or malformed action outputs, duplicate action emissions, excessive repetitions of the same action, and loop-like trajectories, followed by action-format alignment. The final expert training export contains 8,025 next-action examples from 750 expert trajectory records. The 813 covered source task IDs and the 750 training trajectory records are different reporting units and stages; they must not be substituted for one another. Every augmented policy combines the common expert demonstrations with its own collector's increment and is trained from base weights. The Expert SFT checkpoint serves as the collection student.
For the Kimi-K2.5 SRC collection, the trajectory-length statistics in Table R9 describe 108 newly collected trajectories with 1,382 actions, giving 1,382 / 108 = 12.8 actions per trajectory after rounding. The final SRC training increment contains 1,158 examples, so the combined training set still has 8,025 + 1,158 = 9,183 examples. Collected actions and final training examples are different accounting stages. Action-space validation, screenshot/completeness checks, repetition and loop filtering, and training-format preparation determine which supervision records are retained; the action count of 1,382 must not replace 1,158 in the training-size comparisons. We do not infer the number removed by each individual filter from these aggregate totals. The earlier description of “227 newly collected trajectories” conflated these reporting units and will be corrected.
Within the final 1,158-example SRC increment, 959 examples are accepted student actions and 199 are teacher corrections. The earlier 7,882 / 1,301 figures summed to all 9,183 expert-plus-SRC training examples and were incorrectly presented as SRC's student-action / correction composition. This was a reporting-scope error: that composition should be counted within the SRC increment, while the 8,025 expert examples are reported separately. We withdraw the earlier combined-set attribution and correct Appendix C.1 to distinguish collection trajectories, collected actions, and final training examples. The training mixture and its 9,183 examples are unchanged.
Q6. How sensitive is SRC to the teacher model?
The teacher comparison uses the same 807 WebArena-Infinity tasks without validated teacher-data coverage for all three teachers: 1,620 total tasks minus the 813 distinct task IDs covered by the quality-screened Kimi/Qwen source union. The Kimi-K2.5 run supplies the SRC training increment used by the main Table 1 policy; its query total is 11,475, as reported in Table R10. Its collector SR is conditioned on this selected pool without validated teacher-data coverage and is not directly comparable to the earlier 42.5% full-pool collector diagnostic in Table 5.
Table R8 (= Table 20 in the revision). Teacher-specific collection and downstream deployment results.
| Teacher | Successful collection tasks / 807 | Collector SR | Review calls | Correction calls | Training examples (expert + SRC) | Deployed student SR / 1,620 |
|---|---|---|---|---|---|---|
| Kimi-K2.5 | 136 | 16.9 | 7,250 | 4,225 | 8,025 + 1,158 | 35.0 |
| Qwen3.8-max | 151 | 18.7 | 9,653 | 2,374 | 8,025 + 1,663† | 34.8 |
| GPT-6.1-sol | 178 | 22.1 | 8,036 | 3,779 | 8,025 + 995 | 32.3 |
These results support two observations:
- All three teacher-specific datasets improve the deployed student over the 25.3% Expert SFT baseline.
- Higher collection SR does not necessarily yield a better trained student. The teacher with the highest collection SR is not the one with the highest downstream SR.
Each student is trained from base weights on the expert demonstrations plus that teacher's SRC data. †The SRC addition is subsampled to match the student-success addition in Table R3; this is the same Qwen3.8-max model as in Table R3. The comparison does not isolate teacher capability: usable dataset sizes, query totals, and output-format compatibility differ. Consequently, it does not establish equal-budget efficiency, localization accuracy, or teacher insensitivity. The lower GPT-trained student result may reflect these data-processing differences, but no format-normalized ablation currently establishes the cause. We will retain that explanation as a hypothesis rather than a demonstrated finding.
Response to Reviewer ST2V (4/4): Generalization and Trajectory Lengths
Q7. Do the results demonstrate generalization to unseen tasks or applications?
The controlled experiments support task/template generalization within the evaluated setting; cross-application generalization has not been demonstrated.
- WebArena-Infinity: the controlled experiment uses 334 collection tasks and 346 held-out tasks across six applications. Expert demonstrations plus SRC data achieve 13.29%, compared with 8.38% for Expert SFT.
- OSWorld: the controlled supervision study separates 200 training templates from 125 held-out templates.
- Main Table 1: collection and evaluation share task pools, so those gains measure adaptation and must not be presented as clean held-out generalization.
The WebArena-Infinity holdout is also harder than its collection set: approximately 83% versus 46% hard tasks. This helps contextualize the low absolute rates but does not justify comparing them directly with Table 1 or claiming unbiased out-of-distribution performance.
Q8. What are the trajectory lengths and their distributions?
Table R9 (= Table 21 in the revision). Actions per trajectory on WebArena-Infinity. Collection uses a 50-action cap; deployed rows use the 30-action evaluation cap (Eval. 1 in Table R6).
| Population | n | Mean | SD | Median | P25 | P75 | P90 | Max | At cap |
|---|---|---|---|---|---|---|---|---|---|
| Expert demonstrations (Expert SFT data) | 750 | 10.7 | 8.1 | 8 | 5 | 13 | 21 | 49 | 0 |
| SRC-collected, Kimi-K2.5 teacher | 108 | 12.8 | 6.6 | 12 | 8 | 16 | 21 | 39 | 0 |
| SRC-collected, Qwen3.8-max teacher | 146 | 18.2 | 12.1 | 14 | 9 | 24 | 37 | 50 | 3 |
| SRC-collected, GPT-6.1-sol teacher | 153 | 18.1 | 11.4 | 15 | 10 | 24 | 36 | 50 | 7 |
| Deployed Expert SFT, solved | 410 | 11.2 | 6.4 | 8 | 5 | 13 | 21 | 30 | 23 |
| Deployed Expert SFT, failed | 1,210 | 24.9 | 6.8 | 30 | 16 | 30 | 30 | 30 | 665 |
| Deployed Expert SFT + SRC (Kimi-K2.5), solved | 567 | 12.0 | 6.2 | 9 | 6 | 14 | 23 | 30 | 32 |
| Deployed Expert SFT + SRC (Kimi-K2.5), failed | 1,053 | 25.3 | 6.8 | 30 | 17 | 30 | 30 | 30 | 683 |
SRC-collected rows count trajectories admitted without errors.
SRC-collected trajectories are longer than expert demonstrations (median 12–15 versus 8 actions). They contain detours followed by corrections and include harder tasks that the student rarely completes on its own, consistent with the SRC gains in Table R3 being largest on medium tasks. Deployed trajectories are bimodal: solved episodes are short (median 8–9 actions), whereas most failed episodes run until the 30-action cap (665 of 1,210 and 683 of 1,053 failures), so the overall mean mainly reflects the failure rate. Solved episodes of the Expert SFT + SRC policy are slightly longer (mean 12.0 versus 11.2 actions), consistent with its additional successes on medium tasks.
The strongest supported result is that teacher-guided rollback collection produces useful training data for an autonomous GUI policy, with controlled evidence for corrective-label value and within-application held-out performance. We will align the abstract, introduction, experimental claims, and limitations with that evidence, while clearly identifying the controls and measurements that remain incomplete.
We sincerely thank the reviewers again for their thoughtful feedback. We hope the clarifications and additional evidence above address your concerns, and we would appreciate your reconsideration of your assessment in light of these responses. Thank you for your time and consideration.
Response to Reviewer ST2V (1/4): Presentation, Related Work, and Replay Fidelity
Thank you for recognizing the motivation and empirical results, and for identifying how the presentation can better connect the problem, method, and evidence.
W1–W2. The introduction and the role of theory are unclear.
We will lead with the practical problem and position the theory as a design rationale. The introduction will begin with the example in Figure 1: a user request can be completed through either search or menu navigation. Different actions can both be valid, while an apparently reasonable action can create a consequential error that becomes visible only later.
The revised presentation will make the contribution clear in this order:
The implemented procedure can be summarized without mathematical prerequisites:
We will remove wording implying that the theory proves the reliability of the teacher or that the collector explicitly estimates V∗.
Q1. How does SRC relate to reflection and corrective-data methods, and why GUI agents?
We agree that multiple valid solutions are not specific to GUI tasks. The contribution should be situated within interactive imitation and corrective-data collection more broadly.
The UI-TARS comparison refers specifically to its reflection-tuning procedure in Section 4.5. We will not characterize all prior methods as rejecting every deviation from a single reference action.
GUI environments are a useful testbed because they combine multiple valid interaction routes, stateful consequences, and sandbox-based restoration. Extending the procedure to other resettable tool-use environments is plausible, but has not been evaluated here.
Q2. How are replay mismatches detected and handled?
Replay fidelity is measured by comparing recorded application-state snapshots before rollback with the states regenerated by reset-and-replay. Mismatches are logged, but the current collector does not reject every mismatch online.
Table R7 (overall agreement: Section 5.4; prefix-length breakdown: Table 4 in the revision). Replay agreement reported in Section 5.4 and Appendix C.4.