- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Rewind-OPD, which adds trajectory rollback to teacher–student switching during distillation. When teacher–student disagreement triggers intervention, the method selects an earlier turn and lets the teacher repair the continuation. Experiments on three agent benchmarks show improved student performance and lower training time than vanilla OPD.
Critical strengths
-
The approach is reasonable and easy to follow. It addresses a useful problem: intervention at the current state may be too late to correct an earlier student action.
-
The paper includes several relevant OPD baselines. DASH-OPD is a particularly useful comparison because it shares the main switching mechanism but does not rewind. The earlier-intervention and fixed-depth ablations also test simpler alternatives.
-
The experiments cover three environments, two student sizes, and both general and specialized teachers. The student acts alone during evaluation, so the improvements persist without teacher assistance. The reported training time and inference-token reductions also support the method’s practical value.
Critical weaknesses
-
The comparison with FutureBridge-OPD is missing, although the paper cites it. FutureBridge-OPD already restores an earlier environment state and replaces a student action with a teacher action. A direct comparison would help establish the value of Rewind-OPD’s intervention timing and rewind-point selection.
-
Figure 4 tests a different MCOL rule from the one used in training. Training uses the current student segment, including the triggering turn; the diagnostic uses all pre-trigger history and excludes that turn. Thus, the diagnostic does not directly validate the rule used for the main results.
-
Rollback also changes the training data and interaction budget. Replaced branches remain training data on ALFWorld and WebShop, and removed actions do not count toward the episode horizon. The experiments do not separate these effects from the benefit of selecting better rewind points (Appendices A and C.3).
Questions for authors
-
How does Rewind-OPD compare with FutureBridge-OPD under the same training and evaluation settings?
-
The main 4B configuration can rewind three turns, but Table 3 tests fixed depths of only one and two turns. Can you compare the actual training-time MCOL rule with fixed depth three and a random rule with the same depth distribution? This would help isolate the value of discrepancy-based selection.
-
How much do the results change when replaced branches are discarded? Comparisons at equal interaction or teacher-token budgets would also help explain the source of the gains.
-
Appendix B states that ALFWorld Hard comes from training-split tasks. Were these tasks excluded from student training and specialized-teacher training?
4 comments
Response to Reviewer uan5 (2/3)
Q3: Connection between diagnostics (Figure 4, Section 5.6) and end-to-end distillation.
To further clarify the benefits of MCOL and the connection between diagnostics and end-to-end distillation, we compare adaptive rewind methods for distillation from Qwen3-30B-A3B to Qwen3-4B on WebShop, while keeping all other parameters fixed.
| Method | Score↑ | SR↑ | Round↓ |
|---|---|---|---|
| Causal-EMA | 40.5 | 13.7 | 7.3 |
| Positive-run | 41.0 | 14.3 | 7.0 |
| Evidence-onset | 38.8 | 13.0 | 7.3 |
| MCOL (Rewind-OPD) | 42.1 | 16.7 | 7.1 |
As shown in the table above, MCOL achieves the highest Score and SR among the four adaptive methods. These results are consistent with the diagnostic experiments in Section 5.6, further supporting the connection between the rollback position and final performance. Detailed result is shown in Appendix E.2.
Q4: How much do the results change when replaced branches are discarded?
We conduct ablation experiments to compare training with and without replaced branches.
| Method | ALFWorld Seen | ALFWorld Unseen | ALFWorld Hard | WebShop | |||||
|---|---|---|---|---|---|---|---|---|---|
| SR↑ | Round↓ | SR↑ | Round↓ | SR↑ | Round↓ | Score↑ | SR↑ | Round↓ | |
| Without replaced branches | 36.0 | 24.3 | 40.3 | 23.6 | 16.8 | 28.2 | 42.1 | 14.0 | 7.1 |
| With replaced branches | 39.8 | 23.3 | 42.8 | 23.1 | 22.3 | 27.8 | 42.1 | 16.7 | 7.1 |
On ALFWorld and WebShop, retaining replaced branches for training yields better overall performance than excluding them. This suggests that rejected steps can provide useful teacher supervision for updating the student, even when the generated steps exhibit teacher–student distribution gaps.
| Method | Score | SR | Round | |||
|---|---|---|---|---|---|---|
| Student | 1.7B ↑ | 4B ↑ | 1.7B ↑ | 4B ↑ | 1.7B ↓ | 4B ↓ |
| Without replaced branches | 40.8 | 52.0 | 12.2 | 22.6 | 23.7 | 20.5 |
| With replaced branches | 41.5 | 48.2 | 12.7 | 23.0 | 23.7 | 21.1 |
For ScienceWorld, each setting showing advantages on different metrics. We observe that, in this environment, less capable models are more prone to generating invalid actions and becoming trapped in repetitive states, which may lead to erroneous reward signals from the teacher model [2]. We report the best results in Table 1 and add further clarification and discussion in Appendix E.3.
[2] Yu Z, Yu C, Xu S, et al. Mismatch matters: On-policy distillation beyond token agreement [J]. arXiv preprint arXiv:2608.09836, 2026.
Response to Reviewer uan5 (3/3)
Q5: Comparisons at equal interaction or teacher-token budgets would also help explain the source of the gains.
To examine whether the improvements of Rewind-OPD arise from increased training or supervision budgets, we conduct additional experiments on WebShop using the Qwen3-30B-A3B teacher and Qwen3-1.7B/4B students. We compare baseline endpoints with selected Rewind-OPD checkpoints and report cumulative training tokens, teacher training tokens, and relative training time (normalized to Vanilla-OPD).
| Student | Method | Step | Total tokens | Teacher tokens | Score↑ | SR (%)↑ | Round↓ | Training time↓ |
|---|---|---|---|---|---|---|---|---|
| 1.7B | Vanilla-OPD | 150 | 728,903 | 0 | 37.6 | 12.0 | 7.3 | 1.00× |
| 1.7B | DASH-OPD | 150 | 849,187 | 307,458 | 38.9 | 12.0 | 6.9 | 0.89× |
| 1.7B | Rewind-OPD | 100 | 596,239 | 322,179 | 38.3 | 11.7 | 6.9 | 0.65× |
| 1.7B | Rewind-OPD | 130 | 791,340 | 389,477 | 39.0 | 12.7 | 6.9 | 0.83× |
| 1.7B | Rewind-OPD | 150 | 896,596 | 414,569 | 42.9 | 14.0 | 6.5 | 0.94× |
| 4B | Vanilla-OPD | 150 | 165,927* | 0 | 37.4 | 12.7 | 7.7 | 1.00× |
| 4B | DASH-OPD | 150 | 838,316 | 287,311 | 40.9 | 13.7 | 7.3 | 0.83× |
| 4B | Rewind-OPD | 80 | 480,861 | 272,778 | 38.3 | 14.7 | 7.7 | 0.50× |
| 4B | Rewind-OPD | 140 | 832,012 | 393,343 | 41.4 | 15.3 | 7.2 | 0.85× |
| 4B | Rewind-OPD | 150 | 900,468 | 397,527 | 42.1 | 16.7 | 7.1 | 0.91× |
* Note: The 4B Vanilla-OPD run stopped generating explicit reasoning and produced unusually short action-only responses; its low token count should not be interpreted as greater token efficiency.
(1) Total training-token budgets. For the 1.7B student, Rewind-OPD at step 130 achieves higher Score (39.0 vs. 38.9) and SR (12.7% vs. 12.0%) than DASH-OPD with fewer training tokens and lower training time (0.83× vs. 0.89×). Similarly, for the 4B student, Rewind-OPD at step 140 achieves higher Score (41.4 vs. 40.9) and SR (15.3% vs. 13.7%) with comparable training-token budgets and training time.
(2) Teacher supervision budgets. Under comparable or lower teacher training-token budgets, Rewind-OPD maintains competitive performance with reduced training time. For the 1.7B student at step 100, training time decreases from 0.89× to 0.65× with comparable performance. For the 4B student at step 80, Rewind-OPD achieves higher SR (14.7% vs. 13.7%) with fewer teacher tokens and lower training time (0.50× vs. 0.83×).
(3) End-to-end training efficiency. At the full 150-step training budget, Rewind-OPD achieves a higher Score and SR than Vanilla-OPD for both student sizes, while reducing actual training time by 6% for 1.7B and 9% for 4B. Rewind-OPD also achieves higher Score and SR than DASH-OPD at these endpoints.
Overall, these results demonstrate that Rewind-OPD's performance gains are not solely attributable to increased training-token budgets and can be achieved with comparable or lower training time. The findings further support the effectiveness of discrepancy-guided rollback and adaptive teacher intervention. We have added these results and the accompanying discussion to Appendix E.4.
Q6: The source and purpose of the Hard split in ALFWorld.
As discussed in Appendix B, the Hard split is used to evaluate the models’ ability to solve challenging tasks.. We follow the data split provided by the TCOD project [3] and excluded the hard set from training.
[3] Wang J, Zhang W, Shi W, et al. TCOD: Exploring temporal curriculum in on-policy distillation for multi-turn autonomous agents [J]. arXiv preprint arXiv:2604.24005, 2026.
To Reviewer uan5
Dear Reviewer uan5,
We hope this message finds you well. As the discussion period draws to a close, we would like to kindly follow up to ensure that we have fully addressed your concerns. If you have any further questions or feedback, we would be grateful if you could let us know. We particularly appreciate your thoughtful suggestions on comparing with FutureBridge-OPD and further validating the effectiveness of adaptive rollback localization, which motivated our additional baseline comparisons and ablation experiments. We sincerely welcome any further suggestions to help us improve the paper. Thank you so much for the time and effort you have dedicated to reviewing our work.
Best regards,
The Authors
Response to Reviewer uan5 (1/3)
Thank you for your constructive suggestions. We offer further clarifications and additional experimental evidence below.
Q1: How does Rewind-OPD compare with FutureBridge-OPD under the same training and evaluation settings?
We reproduce FutureBridge-OPD [1] following its official training and evaluation protocol. For a consistent comparison, we use the same number of training steps as in our paper: 250 for ALFWorld and ScienceWorld, and 150 for WebShop. The results are shown below.
The table demonstrates that Rewind-OPD outperforms FutureBridge-OPD uniformly. Although both methods involve teacher intervention followed by student regeneration, the advantages of Rewind-OPD mainly stem from two aspects:
More effective intervention localization. FutureBridge-OPD selects the turn with the largest teacher–student discrepancy for teacher intervention. However, as discussed in our motivation, the point of maximum discrepancy is not necessarily the most effective point for the teacher to correct the trajectory.
Adaptive intervention control. FutureBridge-OPD employs a fixed intervention scheme in which the teacher generates a single action, followed by three steps of student continuation. In contrast, Rewind-OPD leverages MCOL to adaptively determine the rollback point and Welford-based discrepancy normalization to guide switching between the teacher and student, thereby constructing higher-quality trajectories.
Q2: How does MCOL compare with fixed-depth rollback with k=3 and a random selector matched to its empirical depth distribution?
We conduct additional ablation experiments on our adaptive rewind method, MCOL. The table below compares fixed rollback depths of one, two, and three steps at each intervention trigger, along with random rollback under the same rewind budget as MCOL.
The results show that deeper rollback does not always lead to better performance. For example, on WebShop, fixed-depth rollback with k=3 underperforms k=2 in both score (39.2 vs. 40.9) and SR (14.0% vs. 15.0%). This suggests that additional teacher intervention may be unnecessary at some steps that the student can handle independently. Moreover, random rollback with the same rewind budget consistently underperforms MCOL across all reported metrics, supporting the value of adaptively selecting rewind depths based on discrepancy history.
[1] Chen C, Fan Y, Sun T, et al. Look ahead before you distill: Future trajectory validation of teacher guidance for agentic on-policy distillation [J]. arXiv preprint arXiv:2608.01953, 2026.