- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes two changes to on-policy (self-)distillation. First, it argues that a forward KL objective is more informative as a reward signal on correct trajectories, and that reverse KL is better for incorrect ones. Second, it posits that on incorrect trajectories, the teacher signal is meaningless beyond a certain token number where the response has already strayed too far from a correct response. The two methods together lift the accuracy of Qwen3 1.7-8B models on AIME/HMMT by 2-3 points over vanilla OPSD. Ablation studies are given, and analysis shows that the method preserves accuracy on another domain somewhat better than vanilla OPSD.
Critical strengths
I see two strengths that merit acceptance:
- The claim that teacher supervision on some tokens is meaningless is indeed true and in my opinion could be useful in informing RL design.
- The experiments are well-conducted. I have a minor comment (see weaknesses and questions) but in general I think the hyperparameters and settings were chosen well.
Critical weaknesses
I have the following concerns:
- My biggest concern is perhaps with the framing of the paper and I have two points here. First of all, I think the paper is not really proposing a new method but performing a careful study of a few factors like which direction of KL divergence is better for learning. Indeed, such an analysis is helpful but the paper should be clearer that it is an analysis of several aspects of OPD.
- Second point from above: I don’t think on-policy self distillation is the right setting for these experiments. All of the proposed changes would apply and work well without the self-component. In fact, due to reasons like self-distillation penalizing thought tokens and suppressing exploration, I believe that the paper would be much more informative (and would perform better too) in a bigger-teacher distillation setting.
- Minor: In my own experiments, I have seen that even with 24-28K token budgets a lot of the gains from OPSD simply come from responses that used to be truncated, and now the model learned to output responses that fit in the token budget. I wonder how much of the (already small) gains come from this. The authors can remedy this concern by answering the question in the next section.
- Minor: Qwen models have gained infamy for being able to fit spurious supervision signals. It would be useful to have at least one non-Qwen model in the comparisons. TeX is supported
Questions for authors
- As mentioned in the weaknesses, I am interested in truncation-related metrics. Specifically, suppose you take the 100 problems with the largest gains in avg@32 accuracy, for the sake of argument (can be a different number). What was the truncation@32 rate for these problems before and after training with different methods with your token budgets?
- If time permits, do you have (preliminary) results with a non-Qwen model? Even a small model is fine for this part.
- I thought Qwen3-4B and 8B already get very high accuracies on MATH-500 and also on DeepScaleR and OpenThoughts, which are more suitable for earlier generations of models. Why not use harder datasets for training?
4 comments
Part II
W4: Minor: Qwen models have gained infamy for being able to fit spurious supervision signals. It would be useful to have at least one non-Qwen model in the comparisons. TeX is supported
Q2: If time permits, do you have (preliminary) results with a non-Qwen model? Even a small model is fine for this part.
We agree that evaluating a non-Qwen model would strengthen our empirical analysis. We are currently conducting additional experiments in this direction. However, due to our limited computational resources, we plan to incorporate the corresponding results into the paper as they become available.
Q3: I thought Qwen3-4B and 8B already get very high accuracies on MATH-500 and also on DeepScaleR and OpenThoughts, which are more suitable for earlier generations of models. Why not use harder datasets for training?
We use OpenThoughts as our training dataset in accordance with the established training paradigm adopted by existing OPSD methods, such as OPSD and PW-OPSD. By contrast, MATH-500 and DeepScaleR are used exclusively in our observational analysis of teacher continuation accuracy, providing diversity across datasets. Neither dataset is used for training. In our newly added OPD experiments, we use mathematical samples from DeepMath with difficulty scores of 6 or higher for training.
Thank you for the rewriting and results
Thank you to the authors for the rewriting and results. I still have some doubts regarding how exactly these insights may be leveraged in a practical setting (where mixing reverse/forward KL etc. may be a bit hacky). Nonetheless, I think the rebuttal has done a great job in directly addressing my two main concerns, i.e. writing (framing) and my most pressing concerns about truncation. I would strongly urge the authors to also include Qwen results in the next version as promised.
Based on these points, I will raise my score to "3: weak accept."
Dear Reviewer do2R,
Thank you for your response and for confirming that your main concerns have been addressed. We appreciate your time and consideration.
Best,
Authors
Part I
Thank you for this insightful comment. We agree that the analysis of key design choices in OPD is an important contribution of our work. To better reflect this perspective, we have added supplementary experiments under the OPD setting and revised the Abstract, Introduction, and Conclusion to clarify the framing of the paper. Specifically, we now explicitly emphasize our analysis of two key aspects of distillation, namely KL divergence selection and distillation position. We also reposition OG-OPSD as a practical instantiation of the insights gained from these analyses, rather than emphasizing methodological novelty alone. These revisions aim to better highlight the analytical contributions of our work while preserving its original structure.
Thank you very much for your valuable suggestion. We have added experiments in a conventional OPD setting. We use DeepMath as the training dataset, Qwen3-4B-Non-Thinking as the student model, and Qwen3-4B-Non-Thinking-RL-Math as the teacher model. Training and evaluation are conducted using the verl framework.
OPD w/ Trunc denotes the variant that applies a fixed RKL objective to all trajectories together with the truncation strategy, while OPD w/ OG-KL uses dynamic divergence selection without any truncation. These results provide some indication that our method can also yield benefits in the OPD setting. We will incorporate these results and the corresponding discussion into the experimental section of the paper.
Regarding this concern, we have computed the format_rate metric during evaluation, which measures the proportion of model-generated responses from which the answer can be successfully extracted. Since we set max_token to 38k during inference, the model has sufficient room to carry out its reasoning. We compare the Base model, vanilla OPSD, and our OG-OPSD, and observe that the format_rate remains consistently high across all three methods.
We do not observe any apparent correlation between the performance improvements achieved by our method and changes in format_rate. Therefore, under our evaluation setting, we believe that the observed performance differences are largely independent of the model learning to generate responses that fit within the token limit. We hope this helps address your concern.