- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies trajectory generation for distilling the reasoning capabilities of large language models. Teacher-generated trajectories are of higher quality but may be difficult for the student to learn from; student-generated trajectories are better aligned with the student’s capabilities, but errors in earlier reasoning steps may weaken subsequent teacher guidance. The authors attempt to balance these two issues by blending the teacher’s and student’s generation distributions, and refer to the phenomenon of entering regions where both models have low confidence during blended generation as “blending drift.” To mitigate this phenomenon, the authors propose ABPD: first, lower the temperature to concentrate the teacher and student distributions, and then adapt the blending ratio at each token. Subject to a KL-divergence budget relative to the student distribution, the method selects a blended distribution that is as close to the teacher as possible and uses it both to generate trajectories and to provide the training target. The paper evaluates the method on mathematical reasoning and code generation tasks, reports competitive overall performance, and analyzes the effects of temperature, the KL budget, and reward variability.
Critical strengths
-
The paper addresses a practically relevant problem, and the method has a clear motivation.
In distillation, teacher-generated trajectories are not necessarily suitable for the student to learn from, while erroneous prefixes generated by the student may weaken the effectiveness of teacher guidance. The paper attempts to adapt the degree of teacher involvement to the current prefix rather than always using a fixed ratio, which is a worthwhile direction. Using a KL budget to limit the distance between the blended distribution and the student also provides an explicit mechanism for controlling the strength of teacher guidance.
-
The experiments cover different model settings and tasks and include relevant analyses.
The authors evaluate the method on two mathematical reasoning model pairs and one code generation setting. In Tables 2 and 3, ABPD achieves better overall average scores than the compared methods, suggesting that the approach has potential. The paper also includes ablations on temperature and the KL budget, along with analyses of trajectory accuracy, diversity, and reward variability, rather than relying solely on comparisons of final performance.
Critical weaknesses
-
It remains unclear whether the blending drift metric measures harmful deviations in reasoning rather than normal uncertainty in generation.
I understand that measures the probability mass outside the union of the teacher’s and student’s Top- sets, rather than mass outside their strict probabilistic supports. However, at the same prefix and over the same common-tail set, linear blending satisfies:
Thus, a single blending step does not create additional tail probability mass. If blending does cause a more serious problem, the key mechanism should be that it changes the prefixes subsequently visited, making both the teacher and student more uncertain at those prefixes. The authors have already proposed this autoregressive explanation, but the current analysis does not sufficiently distinguish or connect “tail mass in a single-step distribution” and “a trajectory progressively deviating from sound reasoning.”
Furthermore, Figure 2 uses . When multiple valid continuations exist, perfectly valid tokens may fall outside both models’ Top-1 sets. Lowering the temperature reduces this probability mass, but this alone does not establish that reasoning errors are being reduced. Table 4 also shows that when the temperature is lowered to 0.5, trajectory accuracy and final performance decline even though drift is further suppressed. This indicates that the relationship between the metric and actual reasoning quality requires more direct validation.
-
The relationship between the training objective and its implementation is unclear, and the existing ablations do not sufficiently separate the contributions of the different modifications.
Section 4.3 explicitly treats the entire as a stop-gradient target distribution. Under this convention, Equation (7) is equivalent to cross-entropy optimization with as the target, with gradient:
This gradient contains no additional reward or advantage weighting. However, Section 4.4 states that only the sampling distribution and reward need to be replaced in an existing on-policy distillation pipeline. If the actual training procedure retains advantage weighting, importance ratios, or clipping, the resulting update is not necessarily equivalent to Equation (7). The main text does not specify the loss actually used for backpropagation, so I cannot currently determine whether the theoretical objective and experimental implementation are consistent.
At the same time, ABPD changes the temperature, the per-token blending ratio, and the supervision target together. The existing temperature and KL-budget ablations are helpful, but they do not establish whether the gains mainly come from more suitable sampled trajectories or from a blended supervision target that is closer to the student. This distinction directly affects how the paper’s contribution should be interpreted.
Questions for authors
-
How can harmful deviations in reasoning be distinguished from diversity among valid continuations?
Could you directly examine whether tokens in the common tail are more likely to correspond to incorrect reasoning steps or lower final-answer accuracy? In particular, when , how do you rule out the possibility that valid but non-top-ranked continuations are counted as drift?
Please also clarify the level at which the blending-specific problem occurs: is it in the single-step token distribution at a fixed prefix, or in the gradual trajectory deviation caused by changes in prefixes under blended sampling? If it is primarily the latter, I would appreciate comparisons that control for the prefix or concrete trajectory examples, as well as an evaluation of whether the conclusions are robust to different choices of .
-
What loss is actually used for backpropagation in the experiments?
Please provide the complete training loss or concise pseudocode and explain, component by component, how it relates to Equation (7). Specifically, which student snapshot is used to generate trajectories, where are gradients stopped through , are the relevant probabilities recomputed during training, and are importance ratios, advantages, and clipping used?
If the implementation uses cross-entropy with a fixed target, please explain how this corresponds to the statement in Section 4.4 that “only the sampling distribution and reward need to be replaced.” If it uses reward-weighted policy gradients, please explain their relationship to Equation (7).
-
How much do adaptive blending ratios and blended supervision targets each contribute to the gains?
With the sharpening temperature and training budget held constant, could you separately compare fixed versus adaptive blending ratios for trajectory generation, and pure-teacher versus blended distributions for supervision?
For the fixed-ratio baseline, I would suggest setting the teacher weight close to the average teacher weight of the adaptive method. This would help distinguish gains from per-token adaptation from those due simply to a different overall strength of teacher guidance. If adaptive blending still produces consistent gains after controlling for these factors, it would provide stronger support for the paper’s central design.
2 comments
Dear Reviewer oFUc, as the discussion period draws to a close, we would appreciate hearing if you have any remaining questions or concerns. We are truly grateful for your review and are fully prepared to provide any further clarification or details needed. Thank you very much for your time and thoughtful feedback!
Response to Additional Questions on Reasoning Quality, Training Objectives, and Ablation Studies
Response to Question 1: Distinguishing Harmful Reasoning Deviations from Valid Generation Diversity.
We thank the reviewer for raising this important concern. The key issue is not the single-step token distribution at a fixed prefix, but the trajectory-level accumulation of deviations under blended sampling. Under non-blending policies, low-confidence sampling typically leads to occasional and independent deviations that do not substantially disrupt reasoning, but rather contribute to generation diversity. On the other hand, blending drift can induce consecutive low-confidence sampling, causing errors to accumulate along the trajectory and eventually leading to severe hallucinations and semantically incoherent reasoning.
To further quantify the impact on rollout quality, we evaluate 10,000 sampled rollouts on DAPO-17K:
Notably, naive blending performs worse than both the teacher and student, whereas ABPD substantially mitigates this degradation. We also add a hallucination-rate analysis in Section 3.2, where blending-based policies exhibit higher hallucination rates alongside relatively large B1 values. Malformed generations, such as “it the five to roll sixth slipping is allowed exterior roll without figure”, further illustrate this reasoning collapse.
Regarding M, we use M=1 because it yields the largest common-tail region, providing a conservative measure of potential blending drift rather than a direct indicator of incorrect tokens. Please refer to our response to Pinned Concern 2 for further details.
Response to Question 2: Clarifying the training objective and its practical implementation.
We thank the reviewer for raising this question. We have provided a detailed clarification of the relationship between our theoretical objective and its practical implementation in our response to Pinned Concern 1.
For clarity, our experiments are conducted within the RL-based OPD framework of Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe, using PPO as the downstream optimizer. This experimental setup has already been explicitly stated in the original manuscript. ABPD modifies the rollout generation policy and reward construction while retaining the existing RL optimization pipeline. Accordingly, Equation (7) characterizes the theoretical objective induced by our reward design, rather than a standalone loss directly used for backpropagation.
We refer the reviewer to Pinned Concern 1 for further details.
Response to Question3: Contributions of Adaptive Blending Ratios and Blended Supervision Targets.
We thank the reviewer for this valuable suggestion. To disentangle the contributions of our core design components, we conducted comprehensive ablation studies across all three experimental settings (Qwen-math, DeepSeek-math, and Qwen-code). Specifically, we consider three variants:
(1) w/o Adaptive Mixing: removing the KL budget and fixing the teacher weight to β=0.8;
(2) w/o Sharpening: removing temperature scaling by setting τ=1;
(3) w/ Pure Teacher Supervision: replacing the adaptive blended supervision target $$pi^*$) with the pure teacher distribution (P_t$.
The results are summarized below.
Beyond final performance, we also investigate the underlying training dynamics. For the full method, Appendix G.1–G.3 provides a comprehensive analysis of training dynamics. Furthermore, Appendix G.4–G.6 examines the training dynamics of different ablation variants, including gradient norms, entropy, advantage statistics, and extreme reward behaviors. Together, these analyses provide further insights into the optimization dynamics, supervision quality, and training stability of ABPD.
We sincerely hope that our clarifications and additional experiments have addressed your concerns regarding trajectory-level blending drift and the contributions of ABPD's individual components. We would greatly appreciate your consideration of these results in your final assessment. Thank you again for your valuable feedback.