- Rating
- 4
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
On-policy distillation (OPD) is an increasingly used algorithm for post-training large language models. It relies on using reverse KL divergence as a way to measure the discrepancy between the student's next token distribution and that of the teacher. This is akin to doing RL on a reward function that is the difference between the log probabilities of the student and the teacher on the sampled text. The authors show that replacing this difference with just its sign is enough to reach the same performance as full OPD, giving us insights on the geometry of the OPD optimization objective. Further analysis reveals that only the tokens with the largest disagreement between the student and the teacher matter. The authors also show that the binary reward idea can successfully be extended to multi teacher OPD. Overall, it provides insights into the minimal components needed to post train LLMs to be good reasoning models.
Critical strengths
The main finding is a cute result and is well demonstrated through extensive experiments. The refinements--only certain tokens matter, extension to the multi-teacher setting--make sense and enrich the results of the paper.
Critical weaknesses
While the findings are interesting, the scope of the results remains relatively limited. I expect the paper to be a useful data point, helping shape how the field understands on policy distillation, but not more. That said, I believe the paper should be accepted, hence my score, but I don't think it should be highlighted as a spotlight or an oral.
Minor: on the writing side, the intro reads as a repetition of the abstract; it could be cut down.
Questions for authors
Would there be some practical interest when running this algorithm, for example reduced memory (only need to store 1 bit, instead of 8 bit floating point probabilities)?
OPD is usually run for a limited number of steps; what would happen if we were to run it for longer? Would we expect differences between OPD and BinaryOPD to emerge when training for much longer, as the more fine-grained information in this signal would start to matter more?
3 comments
I have read the authors' rebuttal and the other reviewers' reviews and keep my (positive) score as it is.
Dear Reviewer,
Thank you for taking the time to review our rebuttal and for your continued support. We sincerely appreciate your time and effort throughout the review process.
Best regards, The Authors
We sincerely thank the reviewer for the positive assessment. It encourages us a lot. We hope our responses below address your concerns and questions.
All revisions made during the rebuttal are highlighted in blue in the revised PDF.
Q1: Potential memory savings.
Yes, binary rewards could in principle reduce storage through bit-packing, but the absolute saving would be very small. The compressible quantity is only one scalar reward per sampled token, rather than a full-vocabulary distribution. With our batch size of 256 and maximum response length of 8,192 (Appendix E, Table 10), an ideal 1-bit encoding would reduce the reward buffer from 2 MiB to 0.25 MiB compared with 16-bit storage, saving just 3.75 MiB across the entire batch. Even compared with FP32, the saving is only 7.75 MiB. This does not reduce model parameters, optimizer states, or activations, and teacher forward passes are still required. However, regarding practical applications, our finding that only a small fraction of high-disagreement tokens is sufficient for effective OPD suggests potential memory and computational savings during student updates. Specifically, we could compute the output logits and loss only for selected tokens, and potentially further reduce backward computation by avoiding unnecessary gradient calculations for unselected tokens where possible, while retaining the computations needed for attention dependencies. However, realizing these optimizations would require substantial modifications to training infrastructure, which were not feasible for us within the limited rebuttal period. We leave this direction for future work. Thus, our contribution is primarily understanding what really OPD needs.
Q2: Long-term training behavior of BinaryOPD.
To our knowledge, the reason OPD is typically trained for a relatively small number of steps is that it converges rapidly, making substantially longer training unnecessary. Nevertheless, to address the concern that reward magnitude may become important over longer training, we conducted 1,000-step experiments on the Qwen3-4B-Non-Thinking student and Qwen3-4B-Non-Thinking-RL-Math teacher pair, corresponding to 256,000 training samples.
In addition to OPD and BinaryOPD, we further evaluated two variants over the same 1,000 steps to investigate whether only the reward direction matters, or whether the fine-grained magnitude information also becomes important during longer training. Let xt=log(pT/pθ) denote the teacher–student log-ratio for the sampled token:
The results after 1,000 training steps are:
All four methods remain comparable after 1,000 steps, with BinaryOPD showing no degradation relative to OPD. Moreover, RandomOPD and SinusoidalOPD also maintain competitive performance despite their arbitrary, non-monotonic reward magnitudes. Furthermore, as shown in Appendix E, Figure 9, all four methods exhibit nearly identical training dynamics in terms of training accuracy, entropy, and response length, despite using different reward magnitudes. These results further support our central finding that the direction of teacher–student updates, rather than the precise reward magnitude, is the critical ingredient for effective OPD, even under substantially extended training.
We have cut down the repetition in introduction.
We thank you again for your positive feedback. We hope that our additional experiments and clarifications have addressed your questions. Please let us know if you have any further questions.