acceptodds
Under review as a conference paper at ICLR 2027

Rewind-OPD: Discrepancy-Guided Trajectory Rollback for On-Policy Distillation in Agentic Tasks

Abstract

On-policy distillation (OPD) provides dense teacher supervision on student-generated trajectories. In agentic tasks, an early divergent action can change subsequent observations, allowing teacher–student discrepancies to accumulate across turns. Existing intervention methods typically act after this drift has become apparent, when the current state may already reflect an earlier mistake. We propose Rewind-OPD, which uses the history of teacher–student discrepancies to select an earlier intervention point, then rolls back to that state and reconstructs the trajectory under teacher guidance before returning control to the student. Across three teacher–student pairs and three agentic benchmarks, Rewind-OPD achieves the best performance among six baselines. With a general 30B teacher, it improves success rate over standard OPD by 4.14 and 4.66 percentage points for 1.7B and 4B students, while reducing average output tokens by 16.0% and 8.0%, respectively. With specialized 8B teachers, the 4B student exceeds the strongest baseline and teachers in average success rate by 1.72 and 1.49 percentage points, respectively, supporting its effectiveness and generalizability.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-policy distillation, LLM agents, Teacher intervention, Trajectory rollback

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_uan5OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Rewind-OPD, which adds trajectory rollback to teacher–student switching during distillation. When teacher–student disagreement triggers intervention, the method selects an earlier turn and lets the teacher repair the continuation. Experiments on three agent benchmarks show improved student performance and lower training time than vanilla OPD.

Critical strengths
  • The approach is reasonable and easy to follow. It addresses a useful problem: intervention at the current state may be too late to correct an earlier student action.

  • The paper includes several relevant OPD baselines. DASH-OPD is a particularly useful comparison because it shares the main switching mechanism but does not rewind. The earlier-intervention and fixed-depth ablations also test simpler alternatives.

  • The experiments cover three environments, two student sizes, and both general and specialized teachers. The student acts alone during evaluation, so the improvements persist without teacher assistance. The reported training time and inference-token reductions also support the method’s practical value.

Critical weaknesses
  • The comparison with FutureBridge-OPD is missing, although the paper cites it. FutureBridge-OPD already restores an earlier environment state and replaces a student action with a teacher action. A direct comparison would help establish the value of Rewind-OPD’s intervention timing and rewind-point selection.

  • Figure 4 tests a different MCOL rule from the one used in training. Training uses the current student segment, including the triggering turn; the diagnostic uses all pre-trigger history and excludes that turn. Thus, the diagnostic does not directly validate the rule used for the main results.

  • Rollback also changes the training data and interaction budget. Replaced branches remain training data on ALFWorld and WebShop, and removed actions do not count toward the episode horizon. The experiments do not separate these effects from the benefit of selecting better rewind points (Appendices A and C.3).

Questions for authors
  • How does Rewind-OPD compare with FutureBridge-OPD under the same training and evaluation settings?

  • The main 4B configuration can rewind three turns, but Table 3 tests fixed depths of only one and two turns. Can you compare the actual training-time MCOL rule with fixed depth three and a random rule with the same depth distribution? This would help isolate the value of discrepancy-based selection.

  • How much do the results change when replaced branches are discarded? Comparisons at equal interaction or teacher-token budgets would also help explain the source of the gains.

  • Appendix B states that ALFWorld Hard comes from training-split tasks. Were these tasks excluded from student training and specialized-teacher training?

4 comments
AuthorsOpenReview ↗

Response to Reviewer uan5 (1/3)

Thank you for your constructive suggestions. We offer further clarifications and additional experimental evidence below.

Q1: How does Rewind-OPD compare with FutureBridge-OPD under the same training and evaluation settings?

We reproduce FutureBridge-OPD [1] following its official training and evaluation protocol. For a consistent comparison, we use the same number of training steps as in our paper: 250 for ALFWorld and ScienceWorld, and 150 for WebShop. The results are shown below.

StudentMethodALFWorld SeenALFWorld UnseenALFWorld HardWebShopScienceWorld
SR↑Round↓SR↑Round↓SR↑Round↓Score↑SR↑Round↓Score↑SR↑Round↓
1.7BFutureBridge-OPD31.424.934.124.411.828.838.911.07.127.88.623.9
1.7BRewind-OPD31.924.336.624.214.328.542.914.06.541.512.723.7
4BFutureBridge-OPD32.125.038.824.112.428.837.413.77.643.019.222.3
4BRewind-OPD39.823.342.823.122.327.842.116.77.152.022.620.5

The table demonstrates that Rewind-OPD outperforms FutureBridge-OPD uniformly. Although both methods involve teacher intervention followed by student regeneration, the advantages of Rewind-OPD mainly stem from two aspects:

  1. More effective intervention localization. FutureBridge-OPD selects the turn with the largest teacher–student discrepancy for teacher intervention. However, as discussed in our motivation, the point of maximum discrepancy is not necessarily the most effective point for the teacher to correct the trajectory.

  2. Adaptive intervention control. FutureBridge-OPD employs a fixed intervention scheme in which the teacher generates a single action, followed by three steps of student continuation. In contrast, Rewind-OPD leverages MCOL to adaptively determine the rollback point and Welford-based discrepancy normalization to guide switching between the teacher and student, thereby constructing higher-quality trajectories.


Q2: How does MCOL compare with fixed-depth rollback with k=3k=3 and a random selector matched to its empirical depth distribution?

We conduct additional ablation experiments on our adaptive rewind method, MCOL. The table below compares fixed rollback depths of one, two, and three steps at each intervention trigger, along with random rollback under the same rewind budget as MCOL.

MethodALFWorld IIDALFWorld OODALFWorld HardWebShop
SR↑Round↓SR↑Round↓SR↑Round↓Score↑SR↑Round↓
Rewind k=1k=132.424.641.023.217.128.240.013.37.2
Rewind k=2k=235.523.837.823.714.928.540.915.07.1
Rewind k=3k=337.623.840.923.321.527.739.214.07.3
Same budget (random)34.824.135.623.714.128.541.113.77.4
MCOL (Rewind-OPD)39.823.342.823.122.327.842.116.77.1

The results show that deeper rollback does not always lead to better performance. For example, on WebShop, fixed-depth rollback with k=3k=3 underperforms k=2k=2 in both score (39.2 vs. 40.9) and SR (14.0% vs. 15.0%). This suggests that additional teacher intervention may be unnecessary at some steps that the student can handle independently. Moreover, random rollback with the same rewind budget consistently underperforms MCOL across all reported metrics, supporting the value of adaptively selecting rewind depths based on discrepancy history.

[1] Chen C, Fan Y, Sun T, et al. Look ahead before you distill: Future trajectory validation of teacher guidance for agentic on-policy distillation [J]. arXiv preprint arXiv:2608.01953, 2026.

AuthorsOpenReview ↗

Response to Reviewer uan5 (2/3)

Q3: Connection between diagnostics (Figure 4, Section 5.6) and end-to-end distillation.

To further clarify the benefits of MCOL and the connection between diagnostics and end-to-end distillation, we compare adaptive rewind methods for distillation from Qwen3-30B-A3B to Qwen3-4B on WebShop, while keeping all other parameters fixed.

MethodScore↑SR↑Round↓
Causal-EMA40.513.77.3
Positive-run41.014.37.0
Evidence-onset38.813.07.3
MCOL (Rewind-OPD)42.116.77.1

As shown in the table above, MCOL achieves the highest Score and SR among the four adaptive methods. These results are consistent with the diagnostic experiments in Section 5.6, further supporting the connection between the rollback position and final performance. Detailed result is shown in Appendix E.2.


Q4: How much do the results change when replaced branches are discarded?

We conduct ablation experiments to compare training with and without replaced branches.

MethodALFWorld SeenALFWorld UnseenALFWorld HardWebShop
SR↑Round↓SR↑Round↓SR↑Round↓Score↑SR↑Round↓
Without replaced branches36.024.340.323.616.828.242.114.07.1
With replaced branches39.823.342.823.122.327.842.116.77.1

On ALFWorld and WebShop, retaining replaced branches for training yields better overall performance than excluding them. This suggests that rejected steps can provide useful teacher supervision for updating the student, even when the generated steps exhibit teacher–student distribution gaps.

MethodScoreSRRound
Student1.7B ↑4B ↑1.7B ↑4B ↑1.7B ↓4B ↓
Without replaced branches40.852.012.222.623.720.5
With replaced branches41.548.212.723.023.721.1

For ScienceWorld, each setting showing advantages on different metrics. We observe that, in this environment, less capable models are more prone to generating invalid actions and becoming trapped in repetitive states, which may lead to erroneous reward signals from the teacher model [2]. We report the best results in Table 1 and add further clarification and discussion in Appendix E.3.

[2] Yu Z, Yu C, Xu S, et al. Mismatch matters: On-policy distillation beyond token agreement [J]. arXiv preprint arXiv:2608.09836, 2026.

AuthorsOpenReview ↗

Response to Reviewer uan5 (3/3)

Q5: Comparisons at equal interaction or teacher-token budgets would also help explain the source of the gains.

To examine whether the improvements of Rewind-OPD arise from increased training or supervision budgets, we conduct additional experiments on WebShop using the Qwen3-30B-A3B teacher and Qwen3-1.7B/4B students. We compare baseline endpoints with selected Rewind-OPD checkpoints and report cumulative training tokens, teacher training tokens, and relative training time (normalized to Vanilla-OPD).

StudentMethodStepTotal tokensTeacher tokensScore↑SR (%)↑Round↓Training time↓
1.7BVanilla-OPD150728,903037.612.07.31.00×
1.7BDASH-OPD150849,187307,45838.912.06.90.89×
1.7BRewind-OPD100596,239322,17938.311.76.90.65×
1.7BRewind-OPD130791,340389,47739.012.76.90.83×
1.7BRewind-OPD150896,596414,56942.914.06.50.94×
4BVanilla-OPD150165,927*037.412.77.71.00×
4BDASH-OPD150838,316287,31140.913.77.30.83×
4BRewind-OPD80480,861272,77838.314.77.70.50×
4BRewind-OPD140832,012393,34341.415.37.20.85×
4BRewind-OPD150900,468397,52742.116.77.10.91×

* Note: The 4B Vanilla-OPD run stopped generating explicit reasoning and produced unusually short action-only responses; its low token count should not be interpreted as greater token efficiency.

(1) Total training-token budgets. For the 1.7B student, Rewind-OPD at step 130 achieves higher Score (39.0 vs. 38.9) and SR (12.7% vs. 12.0%) than DASH-OPD with fewer training tokens and lower training time (0.83× vs. 0.89×). Similarly, for the 4B student, Rewind-OPD at step 140 achieves higher Score (41.4 vs. 40.9) and SR (15.3% vs. 13.7%) with comparable training-token budgets and training time.

(2) Teacher supervision budgets. Under comparable or lower teacher training-token budgets, Rewind-OPD maintains competitive performance with reduced training time. For the 1.7B student at step 100, training time decreases from 0.89× to 0.65× with comparable performance. For the 4B student at step 80, Rewind-OPD achieves higher SR (14.7% vs. 13.7%) with fewer teacher tokens and lower training time (0.50× vs. 0.83×).

(3) End-to-end training efficiency. At the full 150-step training budget, Rewind-OPD achieves a higher Score and SR than Vanilla-OPD for both student sizes, while reducing actual training time by 6% for 1.7B and 9% for 4B. Rewind-OPD also achieves higher Score and SR than DASH-OPD at these endpoints.

Overall, these results demonstrate that Rewind-OPD's performance gains are not solely attributable to increased training-token budgets and can be achieved with comparable or lower training time. The findings further support the effectiveness of discrepancy-guided rollback and adaptive teacher intervention. We have added these results and the accompanying discussion to Appendix E.4.


Q6: The source and purpose of the Hard split in ALFWorld.

As discussed in Appendix B, the Hard split is used to evaluate the models’ ability to solve challenging tasks.. We follow the data split provided by the TCOD project [3] and excluded the hard set from training.

[3] Wang J, Zhang W, Shi W, et al. TCOD: Exploring temporal curriculum in on-policy distillation for multi-turn autonomous agents [J]. arXiv preprint arXiv:2604.24005, 2026.

AuthorsOpenReview ↗

To Reviewer uan5

Dear Reviewer uan5,

We hope this message finds you well. As the discussion period draws to a close, we would like to kindly follow up to ensure that we have fully addressed your concerns. If you have any further questions or feedback, we would be grateful if you could let us know. We particularly appreciate your thoughtful suggestions on comparing with FutureBridge-OPD and further validating the effectiveness of adaptive rollback localization, which motivated our additional baseline comparisons and ablation experiments. We sincerely welcome any further suggestions to help us improve the paper. Thank you so much for the time and effort you have dedicated to reviewing our work.

Best regards,
The Authors

Reviewer_67iKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an on-policy distillation method (REWIND-OPD) for agentic tasks. When a student’s early action causes teacher-student discrepancies to accumulate, the method uses a discrepancy-based switching controller to detect drift, then applies MCOL to estimate an earlier intervention point, rolls the environment back to it, and lets the teacher regenerate the trajectory until control returns to the student. On ALFWorld, WebShop, and ScienceWorld, it improves average success rate over Vanilla OPD by 4.14 and 4.66 points for 1.7B and 4B students, while reducing output tokens.

Critical strengths

Clear motivation and a well-designed ablation. The authors articulate the limitation of intervening only after drift is detected.

Consistent results across settings, with reproducible efficiency reporting. Across three benchmarks, two student sizes, and two teacher setups, REWIND-OPD performs best on most metrics.

Critical weaknesses
  • Possible confound between rollback and training data volume or teacher supervision. According to Section 4.4 and Appendix A, replaced branches remain in the training buffers and are used for training. Appendix E also shows that REWIND-OPD produces more valid response tokens per step (3,486 vs. 3,259 for DASH-OPD). The teacher-generation budgets (15, 8, and 15 turns) are not clearly matched to the baselines. As a result, some of the gain may come from more or different supervision rather than from the rollback mechanism itself. Without controls matched for training tokens or teacher budget, the central claim is hard to verify.

  • Limited statistical support for several comparisons. The evaluation sets are small (roughly 120 to 140 episodes per ALFWorld split and 100 for WebShop), and there are only three random seeds. Several gaps fall within the reported standard deviations. For example, in Table 2, REWIND-OPD on WebShop scores 34.7 ± 3.1 while TurnOPD scores 33.0 ± 0.8. The claim of exceeding the teacher by 1.49 points also depends heavily on aggregation: on ALFWorld Seen, the student reaches 82.6 versus 86.2 for the teacher, and the weighted ALFWorld SR is 70.6 versus 71.0. The authors acknowledge this, but the abstract and conclusion phrase the result more strongly than the evidence supports. No task-level paired tests or confidence intervals are provided.

Questions for authors
  • Data-matched control: Can you retrain Vanilla OPD and DASH-OPD with the same number of training tokens and teacher-generation turns as REWIND-OPD? Also, how does performance change if replaced branches are excluded from the buffer?
  • MCOL evaluation consistency: Appendix C.4 indicates that the diagnostic MCOL in Figure 4 uses all pre-trigger history as its reference, whereas the main method (Eqs. 8–9) uses only the current student segment. Can you evaluate the main-method version in Figure 4? Additionally, with δ = 0 the penalty term is inactive. Please discuss how this affects depth selection and the sensitivity to K_max.
  • Statistical significance: Can you increase the number of random seeds, or report task-level bootstrap confidence intervals and paired tests, particularly for the comparisons with Guided-OPD and TurnOPD in Table 2?
  • Closely related baselines: Related Work identifies FutureBridge-OPD and Relay-OPD as the closest methods, but neither is included as a baseline. Can you add these comparisons?
  • Environment resettability: The method relies on restoring environment state exactly. How does it apply to non-resettable environments or environments with irreversible side effects, such as actually placing an order?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 67iK (1/3)

Thank you for your careful and thorough review of our work. We offer further clarifications and additional experimental evidence below.

Q1: How does performance change if replaced branches are excluded from the buffer?

We conduct ablation experiments to compare training with and without replaced branches.

MethodALFWorld SeenALFWorld UnseenALFWorld HardWebShop
SR↑Round↓SR↑Round↓SR↑Round↓Score↑SR↑Round↓
Without replaced branches36.024.340.323.616.828.242.114.07.1
With replaced branches39.823.342.823.122.327.842.116.77.1

On ALFWorld and WebShop, retaining replaced branches for training yields better overall performance. This suggests that rejected steps can provide useful teacher supervision for updating the student, even when the generated steps exhibit teacher–student distribution gaps.

MethodScoreSRRound
Student1.7B ↑4B ↑1.7B ↑4B ↑1.7B ↓4B ↓
Without replaced branches40.852.012.222.623.720.5
With replaced branches41.548.212.723.023.721.1

For ScienceWorld, each setting showing advantages on different metrics. We observe that, in this environment, less capable models are more prone to generating invalid actions and becoming trapped in repetitive states, which may lead to erroneous reward signals from the teacher model [3]. We report the best results in Table 1 and add further clarification and discussion in Appendix E.3.


Q2: Data-matched control: Can different methods be compared using the same number of training tokens and teacher-generation turns?

To examine whether the improvements of Rewind-OPD arise from increased training or supervision budgets, we conduct additional experiments on WebShop using the Qwen3-30B-A3B teacher and Qwen3-1.7B/4B students. We compare baseline endpoints with selected Rewind-OPD checkpoints and report cumulative training tokens, teacher training tokens, and relative training time (normalized to Vanilla-OPD).

StudentMethodStepTotal tokensTeacher tokensScore↑SR (%)↑Round↓Training time↓
1.7BVanilla-OPD150728,903037.612.07.31.00×
1.7BDASH-OPD150849,187307,45838.912.06.90.89×
1.7BRewind-OPD100596,239322,17938.311.76.90.65×
1.7BRewind-OPD130791,340389,47739.012.76.90.83×
1.7BRewind-OPD150896,596414,56942.914.06.50.94×
4BVanilla-OPD150165,927*037.412.77.71.00×
4BDASH-OPD150838,316287,31140.913.77.30.83×
4BRewind-OPD80480,861272,77838.314.77.70.50×
4BRewind-OPD140832,012393,34341.415.37.20.85×
4BRewind-OPD150900,468397,52742.116.77.10.91×

* Note: The 4B Vanilla-OPD run stopped generating explicit reasoning and produced unusually short action-only responses; its low token count should not be interpreted as greater token efficiency.

(1) Total training-token budgets. For the 1.7B student, Rewind-OPD at step 130 achieves higher Score (39.0 vs. 38.9) and SR (12.7% vs. 12.0%) than DASH-OPD with fewer training tokens and lower training time (0.83× vs. 0.89×). Similarly, for the 4B student, Rewind-OPD at step 140 achieves higher Score (41.4 vs. 40.9) and SR (15.3% vs. 13.7%) with comparable training-token budgets and training time.

(2) Teacher supervision budgets. Under comparable or lower teacher training-token budgets, Rewind-OPD maintains competitive performance with reduced training time. For the 1.7B student at step 100, training time decreases from 0.89× to 0.65× with comparable performance. For the 4B student at step 80, Rewind-OPD achieves higher SR (14.7% vs. 13.7%) with fewer teacher tokens and lower training time (0.50× vs. 0.83×).

(3) End-to-end training efficiency. At the full 150-step training budget, Rewind-OPD achieves a higher Score and SR than Vanilla-OPD for both student sizes, while reducing actual training time by 6% for 1.7B and 9% for 4B. Rewind-OPD also achieves higher Score and SR than DASH-OPD at these endpoints.

Overall, these results demonstrate that Rewind-OPD's performance gains are not solely attributable to increased training-token budgets and can be achieved with comparable or lower training time. The findings further support the effectiveness of discrepancy-guided rollback and adaptive teacher intervention. We have added these results and the accompanying discussion to Appendix E.4.

[1] Yu Z, Yu C, Xu S, et al. Mismatch matters: On-policy distillation beyond token agreement [J]. arXiv preprint arXiv:2608.09836, 2026.

AuthorsOpenReview ↗

Response to Reviewer 67iK (2/3)

Q3: MCOL design and ablation experiments.

Figure 4 is designed as a controlled diagnostic experiment to isolate the effect of rollback localization on downstream performance. Using the full pre-trigger history provides a common reference for comparing different localization methods. In the main Rewind-OPD implementation, MCOL instead uses the current student segment, as defined in Eqs. 8–9.

To further establish the connection between the diagnostic evaluation and end-to-end distillation, we conduct additional experiments on WebShop with Qwen3-30B-A3B as the teacher and Qwen3-4B as the student, while keeping all other settings fixed. The results are shown below.

MethodScore↑SR↑Round↓
Causal-EMA40.513.77.3
Positive-run41.014.37.0
Evidence-onset38.813.07.3
MCOL (Rewind-OPD)42.116.77.1

MCOL achieves the highest task score and success rate among the four adaptive localization methods, consistent with the findings in Figure 4. These end-to-end results further demonstrate that more effective rollback localization translates into better distillation performance. Detailed results can be found in Appendix E.2.

Regarding the hyper-parameters, δ=0\delta=0 is our default setting, corresponding to no explicit per-step penalty in the cumulative discrepancy score. Under this setting, the rollback depth is determined entirely by the accumulated normalized discrepancies, without an additional preference for shorter rollbacks. For Kmax⁡K_{\max}, we set it to 5 for the 1.7B student, whose limited capabilities may lead to earlier deviations that require deeper rollback for effective teacher intervention. For more capable 4B students, we use Kmax⁡=3K_{\max}=3, allowing to retain more of its own trajectory while limiting unnecessary rollback.


Q4: Statistical significance: Can you increase the number of random seeds, particularly for the comparisons with Guided-OPD and TurnOPD in Table 2?

Evaluating agentic tasks is computationally expensive due to long-horizon reasoning and multi-turn environment interactions. Moreover, the relatively small evaluation sets of ALFWorld and WebShop introduce variability, as even a single different action may alter subsequent trajectories and affect task success. Here we extend the ALFWorld and WebShop evaluations from three to five random seeds. The updated five-seed results (mean ± standard deviation) are shown below.

MethodALFWorldWebShopOverall
SR↑Round↓Score↑SR↑Round↓SR↑Round↓
Teacher72.9±2.415.3±0.358.1±1.131.0±1.15.3±0.151.9±1.110.3±0.2
Vanilla OPD47.8±1.719.3±0.358.9±0.930.0±2.55.0±0.238.9±2.012.1±0.1
TCOD-F2B52.2±1.318.4±0.259.7±1.731.6±2.44.8±0.141.9±1.311.6±0.2
Guided-OPD70.6±1.415.8±0.356.7±1.432.2±1.75.8±0.151.4±1.410.8±0.2
TurnOPD50.6±1.519.0±0.360.9±1.232.4±1.04.6±0.241.5±0.911.8±0.2
DASH-OPD69.1±1.715.8±0.357.6±1.531.2±2.85.0±0.250.2±1.210.4±0.2
Rewind-OPD71.0±1.015.6±0.261.4±1.933.6±2.84.8±0.152.3±1.210.2±0.1

The five-seed results preserve the overall trends reported in Table 2. Rewind-OPD achieves the highest mean overall SR (52.3%) and the fewest overall interaction rounds (10.2), as well as the highest WebShop Score (61.4) and SR (33.6%). Compared with Guided-OPD and TurnOPD, it improves overall SR by 0.9 and 10.8 percentage points, respectively. Notably, Rewind-OPD maintains the highest mean overall SR across the evaluated methods and achieves performance comparable to the teacher (52.3% vs. 51.9%).

the result provides a more reliable assessment of performance variability and confirms the overall performance trends. We have added these results and further discussion to Appendix E.5, and refined the claims in the abstract and conclusion to more accurately reflect the observed improvements.

AuthorsOpenReview ↗

Response to Reviewer 67iK (3/3)

Q5: Add more closely related baselines for comparison.

Although Relay-OPD [2] is discussed in the related work, its setting differs from ours. It is designed for single-turn reasoning tasks and uses discrepancies at thinking tokens and speculative decoding to interleave generation. These mechanisms do not directly apply to our multi-turn agentic setting.

We reproduce FutureBridge-OPD [3] using its official training and evaluation code. For a consistent comparison, we use the same number of training steps as in our paper: 250 for ALFWorld and ScienceWorld, and 150 for WebShop. The results are shown below.

StudentMethodALFWorld SeenALFWorld UnseenALFWorld HardWebShopScienceWorld
SR↑Round↓SR↑Round↓SR↑Round↓Score↑SR↑Round↓Score↑SR↑Round↓
1.7BFutureBridge-OPD31.424.934.124.411.828.838.911.07.127.88.623.9
1.7BRewind-OPD31.924.336.624.214.328.542.914.06.541.512.723.7
4BFutureBridge-OPD32.125.038.824.112.428.837.413.77.643.019.222.3
4BRewind-OPD39.823.342.823.122.327.842.116.77.152.022.620.5

The table demonstrates that Rewind-OPD outperforms FutureBridge-OPD uniformly. Although both methods involve teacher intervention followed by student regeneration, the advantages of Rewind-OPD mainly stem from two aspects:

  1. More effective intervention localization. FutureBridge-OPD selects the turn with the largest teacher–student discrepancy for teacher intervention. However, as discussed in our motivation, the point of maximum discrepancy is not necessarily the most effective point for the teacher to correct the trajectory.

  2. Adaptive intervention control. FutureBridge-OPD employs a fixed intervention scheme in which the teacher generates a single action, followed by three steps of student continuation. In contrast, Rewind-OPD leverages MCOL to adaptively determine the rollback point and Welford-based discrepancy normalization to guide switching between the teacher and student, thereby constructing higher-quality trajectories.


Q6: Environment resettability: How does it apply to non-resettable environments or environments with irreversible side effects?

For the three benchmarks considered in this work (ALFWorld, WebShop, and ScienceWorld), environment rollback is implemented through reset-and-replay: we reset the environment to its initial state and replay the recorded action sequence up to the selected intervention point. This implementation is similar to FutureBridge-OPD's implementation [2].

For environments that are non-resettable, involve costly action execution, or have irreversible side effects, a sliding-window checkpoint strategy can be adopted. Specifically, a backup environment is maintained at turn t−Kmax⁡t-K_{\max}, where Kmax⁡K_{\max} is the maximum rollback depth. When rollback is triggered, the backup can be advanced to the desired intervention state using at most Kmax⁡−k∗K_{\max}-k^* actions, then used to replace the current environment. A new backup is subsequently created and maintained as the window advances. This bounds the replay cost by the maximum rollback depth, independent of the total trajectory length. The rewind mechanism is described in Algorithm 3 of Appendix A in the revised manuscript.

[2] Xu H, Xu X, Hong H, et al. Pass the baton: Trajectory-relayed on-policy distillation [J]. arXiv preprint arXiv:2607.26057, 2026.

[3] Chen C, Fan Y, Sun T, et al. Look ahead before you distill: Future trajectory validation of teacher guidance for agentic on-policy distillation [J]. arXiv preprint arXiv:2608.01953, 2026.

AuthorsOpenReview ↗

To Reviewer 67iK

Dear Reviewer 67iK,

We hope this message finds you well. As the discussion period draws to a close, we would like to kindly follow up to ensure that we have fully addressed your concerns. If you have any further questions or feedback, we would be grateful if you could let us know. We particularly appreciate your thoughtful suggestions on controlling supervision budgets, strengthening statistical reliability, and including closely related baselines, which motivated our additional experiments and more comprehensive comparisons. We sincerely welcome any further suggestions to help us improve the paper. Thank you so much for the time and effort you have dedicated to reviewing our work.

Best regards,
The Authors

Reviewer_epB7OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses compounding teacher–student discrepancies in multi-turn agentic on-policy distillation. It proposes REWIND-OPD, which uses discrepancy history to select an earlier intervention point, restores the corresponding environment state and interaction history, and collects teacher-guided continuations for student training. Experiments across three agentic benchmarks and three teacher–student settings report improved aggregate student-only success rates over the evaluated baselines.

Critical strengths

The paper targets a meaningful failure mode in agent training, where delayed teacher intervention may inherit the consequences of earlier student mistakes. The framework coherently integrates discrepancy-based switching, retrospective rollback, and supervision from both student-generated and teacher-generated responses. Evaluation across multiple environments, student sizes, and teacher types, together with student-only testing and efficiency measurements, provides useful evidence of practical utility.

Critical weaknesses

The current ablations do not fully isolate the contribution of adaptive localization from access to deeper rollback. The diagnostic comparison of localization methods uses a configuration different from the main training setup, limiting its support for the specific MCOL variant used in distillation. Changes in collected supervision and actual interaction budgets introduce an additional attribution uncertainty despite the reported wall-clock improvements.

Questions for authors

1. Does MCOL outperform depth-matched rollback baselines?

Table 3 evaluates fixed depths of k=1k=1 and k=2k=2, whereas the main 4B configuration permits depths up to k=3k=3. Could the authors add a fixed-k=3k=3 baseline and a randomized selector matched to MCOL’s empirical depth distribution, while controlling the intervention trigger and teacher budget? These comparisons would clarify whether conditioning rollback depth on the discrepancy history provides value beyond the choice of depth distribution.

2. Do the localization comparisons transfer to end-to-end distillation?

Section 5.6 evaluates selectors using reference histories and depth limits that differ from the main training configuration. Could the authors compare these alternatives during end-to-end distillation, using the same available history, intervention controller, feasible depth limits, and supervision settings, and report final student-only performance? This would directly test whether the localization advantage observed in the diagnostic setting translates into better trained students.

3. Can the gains be separated from changes in interaction and supervision budgets?

Appendix A states that removed interactions do not count toward the retained-trajectory horizon, while replaced branches remain eligible for training on ALFWorld and WebShop. Could the authors provide separate comparisons matched by total supervised response tokens, teacher-generated tokens, and actual environment interactions, alongside an ablation excluding replaced branches from training? These controls would help distinguish the contribution of discrepancy-guided rollback from changes in the amount of collected supervision.

5 comments
AuthorsOpenReview ↗

Response to Reviewer epB7 (1/2)

We appreciate your careful review and your positive assessment of our work. We provide detailed responses to your comments below.

Q1: How does MCOL compare with fixed-depth rollback with k=3k=3 and a random selector matched to its empirical depth distribution?

We conduct additional ablation experiments on our adaptive rewind method, MCOL. The table below compares fixed rollback depths of one, two, and three steps at each intervention trigger, along with random rollback under the same rewind budget as MCOL.

MethodALFWorld IIDALFWorld OODALFWorld HardWebShop
SR↑Round↓SR↑Round↓SR↑Round↓Score↑SR↑Round↓
Rewind k=1k=132.424.641.023.217.128.240.013.37.2
Rewind k=2k=235.523.837.823.714.928.540.915.07.1
Rewind k=3k=337.623.840.923.321.527.739.214.07.3
Same budget (random)34.824.135.623.714.128.541.113.77.4
MCOL (Rewind-OPD)39.823.342.823.122.327.842.116.77.1

The results show that deeper rollback does not always lead to better performance. For example, on WebShop, fixed-depth rollback with k=3k=3 underperforms k=2k=2 in both score (39.2 vs. 40.9) and SR (14.0% vs. 15.0%). This suggests that additional teacher intervention may be unnecessary at some steps that the student can handle independently. Moreover, random rollback with the same rewind budget consistently underperforms MCOL across all reported metrics, supporting the value of adaptively selecting rewind depths based on discrepancy history. Please find detailed results in Appendix E.1.


Q2: Do the localization comparisons transfer to end-to-end distillation?

To further clarify the benefits of MCOL and the connection between diagnostics and end-to-end distillation, we compare adaptive rewind methods for distillation from Qwen3-30B-A3B to Qwen3-4B on WebShop, while keeping all other parameters fixed.

MethodScore↑SR↑Round↓
Causal-EMA40.513.77.3
Positive-run41.014.37.0
Evidence-onset38.813.07.3
MCOL (Rewind-OPD)42.116.77.1

As shown in the table above, MCOL achieves the highest Score and SR among the four adaptive methods. These results are consistent with the diagnostic experiments in Section 5.6, further supporting the connection between the rollback position and final performance. A detailed discussion is provided in Appendix E.2.


Q3: How does performance change if replaced branches are excluded from the buffer?

We conduct ablation experiments to compare training with and without replaced branches.

MethodALFWorld SeenALFWorld UnseenALFWorld HardWebShop
SR↑Round↓SR↑Round↓SR↑Round↓Score↑SR↑Round↓
Without replaced branches36.024.340.323.616.828.242.114.07.1
With replaced branches39.823.342.823.122.327.842.116.77.1

On ALFWorld and WebShop, retaining replaced branches for training yields better overall performance than excluding them. This suggests that rejected steps can provide useful teacher supervision for updating the student, even when the generated steps exhibit teacher–student distribution gaps.

MethodScoreSRRound
Student1.7B ↑4B ↑1.7B ↑4B ↑1.7B ↓4B ↓
Without replaced branches40.852.012.222.623.720.5
With replaced branches41.548.212.723.023.721.1

For ScienceWorld, the results are mixed, with each setting showing advantages on different metrics. We observe that, in this environment, less capable models are more prone to generating invalid actions and becoming trapped in repetitive states, which may lead to erroneous reward signals from the teacher model [1]. Therefore, we report the best-performing results in Table 1 of Section 5 and have added further clarification and discussion in Appendix E.3.

[1] Yu Z, Yu C, Xu S, et al. Mismatch matters: On-policy distillation beyond token agreement[J]. arXiv preprint arXiv:2608.09836, 2026.

AuthorsOpenReview ↗

Response to Reviewer epB7 (2/2)

Q4: Can the gains be separated from changes in interaction and supervision budgets?

To examine whether the improvements of Rewind-OPD arise from increased training or supervision budgets, we conduct additional experiments on WebShop using the Qwen3-30B-A3B teacher and Qwen3-1.7B/4B students. We compare baseline endpoints with selected Rewind-OPD checkpoints and report cumulative training tokens, teacher training tokens, and relative training time (normalized to Vanilla-OPD).

StudentMethodStepTotal tokensTeacher tokensScore↑SR (%)↑Round↓Training time↓
1.7BVanilla-OPD150728,903037.612.07.31.00×
1.7BDASH-OPD150849,187307,45838.912.06.90.89×
1.7BRewind-OPD100596,239322,17938.311.76.90.65×
1.7BRewind-OPD130791,340389,47739.012.76.90.83×
1.7BRewind-OPD150896,596414,56942.914.06.50.94×
4BVanilla-OPD150165,927*037.412.77.71.00×
4BDASH-OPD150838,316287,31140.913.77.30.83×
4BRewind-OPD80480,861272,77838.314.77.70.50×
4BRewind-OPD140832,012393,34341.415.37.20.85×
4BRewind-OPD150900,468397,52742.116.77.10.91×

* Note: The 4B Vanilla-OPD run stopped generating explicit reasoning and produced unusually short action-only responses; its low token count should not be interpreted as greater token efficiency.

(1) Total training-token budgets. For the 1.7B student, Rewind-OPD at step 130 achieves higher Score (39.0 vs. 38.9) and SR (12.7% vs. 12.0%) than DASH-OPD with fewer training tokens and lower training time (0.83× vs. 0.89×). Similarly, for the 4B student, Rewind-OPD at step 140 achieves higher Score (41.4 vs. 40.9) and SR (15.3% vs. 13.7%) with comparable training-token budgets and training time.

(2) Teacher supervision budgets. Under comparable or lower teacher training-token budgets, Rewind-OPD maintains competitive performance with reduced training time. For the 1.7B student at step 100, training time decreases from 0.89× to 0.65× with comparable performance. For the 4B student at step 80, Rewind-OPD achieves higher SR (14.7% vs. 13.7%) with fewer teacher tokens and lower training time (0.50× vs. 0.83×).

(3) End-to-end training efficiency. At the full 150-step training budget, Rewind-OPD achieves a higher Score and SR than Vanilla-OPD for both student sizes, while reducing actual training time by 6% for 1.7B and 9% for 4B. Rewind-OPD also achieves higher Score and SR than DASH-OPD at these endpoints.

Overall, these results demonstrate that Rewind-OPD's performance gains are not solely attributable to increased training-token budgets and can be achieved with comparable or lower training time. The findings further support the effectiveness of discrepancy-guided rollback and adaptive teacher intervention. We have added these results and the accompanying discussion to Appendix E.4.

AuthorsOpenReview ↗

To Reviewer epB7

Dear Reviewer epB7,

We hope this message finds you well. As the discussion period draws to a close, we would like to kindly follow up to ensure that we have fully addressed your concerns. If you have any further questions or feedback, we would be grateful if you could let us know. We particularly appreciate your thoughtful suggestions on isolating the contribution of adaptive rollback localization and controlling training and supervision budgets, which motivated our additional ablation experiments and budget-matched comparisons. We sincerely welcome any further suggestions to help us improve the paper. Thank you so much for the time and effort you have dedicated to reviewing our work.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.