- Rating
- 3
- Confidence
- 3(+1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
The authors study on-policy distillation (OPD) setting. They argue that always following the teacher might not always be helpful for student performance. They therefore propose a strategy to essentially scale the student logits on teacher trajectories by the student advantage, so the responses which are helpful for the student should receive more learning signal than others. The method is then Advantage-based on-policy distillation with an additional importance-sampling correction (since the trajectories are coming from the student).
The empirical results suggest that overall their method seesm to be better than OPD and GRPO in most of the scenarios. (while sometimes it underperforms)
Critical strengths
Simple and well motivated method -- rescale the logits based on student advantages, therefore emphasizing the responses which are more helpful for the student.
Overall, empirical performance suggests that the method is better than baselines.
Critical weaknesses
I do not fully understand why the method sometimes is worsse than OPD or than GRPO. Could the authors explain why it happens?
When I look at figure 2, the X-axis is not the same for all the methods. Why is it the case? What happens to the method if we run it for longer? Will it collapse? Figure 2 middle suggests that the performance of blue method starts to go down. Can you show the same figure 2 but for 1000 iterations?
Why is it the case that GRPO doesnt learn much on AIME and OPD doesnt learn much on TACO?
The authors use different sampling settings for their method and for and for OPD / OPSD. Are these parameters only for training-time trajectories or is it the same for evaluation? Would it be possible to report performance under matched hyperparameters settings for both, the method and the baselines? The fact that the parameters are different makes it a bit hard to judge where the advantage comes from.
Also, you use different G for GRPO and OPD/AOPD. Would it be possible to report results under matched settings? Does your method improves / worsens with increased G?
Questions for authors
See above.
3 comments
Thank you for the rebuttal
Thank you for the rebuttal.
You have addressed my concerns. I'm maintaining my score, but increasing confidence.
We thank the reviewer for their positive assessment of our paper. We are glad that our additional experiments and clarifications addressed their concerns. We will incorporate these points in the revised version.
Rebuttal
We thank the reviewer for the positive and constructive feedback to our paper. Below we address their questions:
As figure 2 (left/right) and figure 3 show, AOPD achieves higher evaluation rewards than OPD and GRPO in most of our experiments. However, maximizing the reward does not always transfer to the best benchmark results since benchmarks evaluate different capabilities. For instance, the TACO dataset consists of python codes only, but Multipl-E evaluates the models on other languages with.
Nevertheless, the few exceptions are relatively small. For example, OPD achieves 33.3 versus 32.5 on MultiPL-E and 16.0 versus 15.8 on HMMT, while AOPD achieves better results on AMC and HumanEval.
The difference in the x-axis is between experimental settings (coding and math), not between methods within the same comparison. For external-teacher distillation, we train all methods for 450 iterations on math and 350 iterations on coding. For self-distillation, we use 400 iterations for all methods. These are arbitrary fixed training budgets chosen for each setting, and all methods within each comparison use the same number of iterations.
In order to assure the reviewer that our algorithm is stable for longer trainings, we ran our algorithm for 750 steps. This is shown in this figure: https://ibb.co/5WWyjwCD. The evaluation reward remains stable and it does not collapse.
Regarding the drop in the last iteration on AIME, this is because of oscillations during training rather than a sustained decline. The performance on AIME remains high over the iterations as the attached figure above shows. Note that other methods such as OPD oscillate as well.
GRPO optimizes the training reward using only student rollouts and without external teacher signals. On harder problems and harder benchmarks such as AIME, this limits the benefit of GRPO since the student cannot discover the solutions based on its own exploration. In contrast, OPD provides dense supervision from a stronger teacher regardless of the reward. This can help transfer capabilities beyond what the student can discover.
OPD imitates the teacher regardless of whether its actions improve the student's reward. This can explain why OPD makes little progress on TACO, while it still improves on other benchmarks. For example, OPD outperforms GRPO on HumanEval+ despite achieving lower TACO evaluation reward. AOPD combines these benefits by using teacher supervision while weighting it according to its usefulness to the student.
We thank the reviewer for pointing this out. These parameters are only at training time. We evaluate all the models at temp = 0.6, top_p = 0.95 and top_k = 20, similar to [1, 3]. We added this to our explanation at the end of the hyperparameters paragraph in the updated file.
We also ran OPD under the same training setting as ours ( temp = 0.8, top_p = 1). As the following figure shows they achieve similar performance: https://ibb.co/TDjNzgKS.
We report the effect of increasing group size in Appendix A, where increasing G from 4 to 8 provides only negligible improvements and does not consistently improve benchmark performance. Similar observations for OPD have been reported in previous work [4]. Therefore, we use G=4 for AOPD and OPD in our main experiments to reduce computational cost, while GRPO uses G=8 following its standard setting. All other training settings are kept consistent within each comparison.
[1] Zhao et al, “Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models”.
[2] Hubotter et al, “Reinforcement Learning via Self-Distillation”.
[3] Jin et al, “Entropy-Aware On-Policy Distillation of Language Models”.
[4] Hou et al, “Uni-opd: Unifying on-policy distillation with a dual-perspective recipe.”