- Rating
- 4
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The look at on-policy distillation and carefully determine what goes into the training batch so that it is not dominated by a few long rollouts, and they prioritize samples with high student-teacher disagreement. They also experiment with allowing the teacher to take over for several turns to avoid stagnation on hard tasks - As I understand it this is a neat way to mix in SFT on teacher trajectories.
Critical strengths
- Very relevant work - the detail of setting up batch composition well for these methods.
- Very sensible method with many aspects that combine together to make it work well - illustrated clearly with several ablations.
- Presented and explained clearly.
Critical weaknesses
- I will read the reviews from the other reviewers but unusually I can't see any issues other than the common ask of more model families etc.
Questions for authors
All my questions were answered in the appendix. I will ask during the discussion period if I have any more.
2 comments
Thanks again for Your Time and Thoughtful Review in ICLR 2027
Thank you for your thoughtful reading of the paper and appendix, and for your encouraging assessment of the batch-composition approach. We clarified the role of teacher recovery and added independent runs on ALFWorld and WebShop. We hope the revision addresses your concern by making the scope of the current work clearer!
Response: Additional Results and Clarification of Teacher Recovery
Thank you for your positive assessment and careful reading of the appendix. We appreciate your recognition of our batch-composition approach and ablations.
During rebuttal, we added three-run comparisons of DivOPD and vanilla OPD on ALFWorld 1.7B and WebShop 3B, along with three-seed cap-sensitivity experiments across all six settings. We also added length-based selection controls and system-level cost measurements to examine how the choice of turns affects performance and when learner-time savings translate into lower overall cost.
Table Gbzt-1. Full-method Final-5 success rate (%), three independent runs per method. For each run, Final-5 is the mean success rate over the last five evaluations. We report the mean ± sample SD across runs. The learner-update budgets are 250 for ALFWorld and 150 for WebShop, with evaluation every five updates.
DivOPD has higher mean Final-5 in both settings. The revised PDF reports these independent runs in Section 4.6, Table 3 (p. 9), the cap sweep in Appendix G, Table 20 (p. 24), and the additional controls and cost measurements in Appendix H, Tables 21–26 (pp. 26–29).
One detail about the optional recovery mechanism may help clarify how the teacher contributes. When an interaction stalls, the teacher briefly takes control to help reach a state from which the student can continue. Teacher-generated actions are excluded from the distillation loss. Once the student resumes control, its subsequent turns are eligible for training. Recovery is disabled during evaluation, so the student acts independently. Section 3.4 of the revised PDF (p. 5) makes this explicit.
We appreciate your suggestion to broaden model-family coverage and see this as an important direction for further evaluation. Thank you again for your thoughtful review. We welcome any further questions during the discussion.