acceptodds
Under review as a conference paper at ICLR 2027

Mitigating Blending Drift: Adaptive Blended Policy Distillation for LLM Reasoning

Abstract

Large language model (LLM) knowledge distillation (KD) with reinforcement learning (RL) has recently shown promising results in enhancing the reasoning capabilities of student models by transferring knowledge from stronger teachers. Existing approaches predominantly fall into two pure policy paradigms. Off-policy distillation uses teacher-generated rollouts for high-quality supervision, but may exceed the student's learning capacity and induce severe train–test mismatch. On-policy distillation preserves learnability by using student-generated rollouts, but suffers from prefix failure, where erroneous student prefixes hinder effective teacher supervision. Policy blending offers a natural compromise by combining teacher and student policies for rollout generation. However, we identify a previously overlooked failure mode, termed blending drift, in which the blended policy assigns probability mass to regions unsupported by either the teacher or the student. Through theoretical analysis, we provide a theoretical guarantee that reducing the sampling temperature effectively mitigates this drift. Motivated by this finding, we introduce Adaptive Blended Policy Distillation (ABPD), a novel and simple sharpen-then-blend strategy that generates high-quality rollouts while remaining within the student's on-policy support. Extensive experiments on mathematical reasoning and code generation benchmarks demonstrate that ABPD consistently achieves the best overall performance among competing methods.

Primary area:
reinforcement learning
Keywords:
Knowledge Distillation, Reinforcement Learning, LLM Reasoning, Policy Blending, Blending Drift

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_oFUcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies trajectory generation for distilling the reasoning capabilities of large language models. Teacher-generated trajectories are of higher quality but may be difficult for the student to learn from; student-generated trajectories are better aligned with the student’s capabilities, but errors in earlier reasoning steps may weaken subsequent teacher guidance. The authors attempt to balance these two issues by blending the teacher’s and student’s generation distributions, and refer to the phenomenon of entering regions where both models have low confidence during blended generation as “blending drift.” To mitigate this phenomenon, the authors propose ABPD: first, lower the temperature to concentrate the teacher and student distributions, and then adapt the blending ratio at each token. Subject to a KL-divergence budget relative to the student distribution, the method selects a blended distribution that is as close to the teacher as possible and uses it both to generate trajectories and to provide the training target. The paper evaluates the method on mathematical reasoning and code generation tasks, reports competitive overall performance, and analyzes the effects of temperature, the KL budget, and reward variability.

Critical strengths
  1. The paper addresses a practically relevant problem, and the method has a clear motivation.

    In distillation, teacher-generated trajectories are not necessarily suitable for the student to learn from, while erroneous prefixes generated by the student may weaken the effectiveness of teacher guidance. The paper attempts to adapt the degree of teacher involvement to the current prefix rather than always using a fixed ratio, which is a worthwhile direction. Using a KL budget to limit the distance between the blended distribution and the student also provides an explicit mechanism for controlling the strength of teacher guidance.

  2. The experiments cover different model settings and tasks and include relevant analyses.

    The authors evaluate the method on two mathematical reasoning model pairs and one code generation setting. In Tables 2 and 3, ABPD achieves better overall average scores than the compared methods, suggesting that the approach has potential. The paper also includes ablations on temperature and the KL budget, along with analyses of trajectory accuracy, diversity, and reward variability, rather than relying solely on comparisons of final performance.

Critical weaknesses
  1. It remains unclear whether the blending drift metric measures harmful deviations in reasoning rather than normal uncertainty in generation.

    I understand that BMB_M measures the probability mass outside the union of the teacher’s and student’s Top-MM sets, rather than mass outside their strict probabilistic supports. However, at the same prefix and over the same common-tail set, linear blending satisfies:

BM ⁣(βPt+(1−β)Ps)=βBM(Pt)+(1−β)BM(Ps).B_M\!\left(\beta P_t+(1-\beta)P_s\right) =\beta B_M(P_t)+(1-\beta)B_M(P_s).

Thus, a single blending step does not create additional tail probability mass. If blending does cause a more serious problem, the key mechanism should be that it changes the prefixes subsequently visited, making both the teacher and student more uncertain at those prefixes. The authors have already proposed this autoregressive explanation, but the current analysis does not sufficiently distinguish or connect “tail mass in a single-step distribution” and “a trajectory progressively deviating from sound reasoning.”

Furthermore, Figure 2 uses M=1M=1. When multiple valid continuations exist, perfectly valid tokens may fall outside both models’ Top-1 sets. Lowering the temperature reduces this probability mass, but this alone does not establish that reasoning errors are being reduced. Table 4 also shows that when the temperature is lowered to 0.5, trajectory accuracy and final performance decline even though drift is further suppressed. This indicates that the relationship between the metric and actual reasoning quality requires more direct validation.

  1. The relationship between the training objective and its implementation is unclear, and the existing ablations do not sufficiently separate the contributions of the different modifications.

    Section 4.3 explicitly treats the entire π∗\pi^* as a stop-gradient target distribution. Under this convention, Equation (7) is equivalent to cross-entropy optimization with π∗\pi^* as the target, with gradient:

−Ey∼π∗[∑t∇θlog⁡Ps(yt∣x,y<t)].-\mathbb{E}_{y\sim\pi^*} \left[\sum_t\nabla_\theta\log P_s(y_t\mid x,y_{<t})\right].

This gradient contains no additional reward or advantage weighting. However, Section 4.4 states that only the sampling distribution and reward need to be replaced in an existing on-policy distillation pipeline. If the actual training procedure retains advantage weighting, importance ratios, or clipping, the resulting update is not necessarily equivalent to Equation (7). The main text does not specify the loss actually used for backpropagation, so I cannot currently determine whether the theoretical objective and experimental implementation are consistent.

At the same time, ABPD changes the temperature, the per-token blending ratio, and the supervision target together. The existing temperature and KL-budget ablations are helpful, but they do not establish whether the gains mainly come from more suitable sampled trajectories or from a blended supervision target that is closer to the student. This distinction directly affects how the paper’s contribution should be interpreted.

Questions for authors
  1. How can harmful deviations in reasoning be distinguished from diversity among valid continuations?

    Could you directly examine whether tokens in the common tail are more likely to correspond to incorrect reasoning steps or lower final-answer accuracy? In particular, when M=1M=1, how do you rule out the possibility that valid but non-top-ranked continuations are counted as drift?

    Please also clarify the level at which the blending-specific problem occurs: is it in the single-step token distribution at a fixed prefix, or in the gradual trajectory deviation caused by changes in prefixes under blended sampling? If it is primarily the latter, I would appreciate comparisons that control for the prefix or concrete trajectory examples, as well as an evaluation of whether the conclusions are robust to different choices of MM.

  2. What loss is actually used for backpropagation in the experiments?

    Please provide the complete training loss or concise pseudocode and explain, component by component, how it relates to Equation (7). Specifically, which student snapshot is used to generate trajectories, where are gradients stopped through π∗\pi^*, are the relevant probabilities recomputed during training, and are importance ratios, advantages, and clipping used?

    If the implementation uses cross-entropy with a fixed target, please explain how this corresponds to the statement in Section 4.4 that “only the sampling distribution and reward need to be replaced.” If it uses reward-weighted policy gradients, please explain their relationship to Equation (7).

  3. How much do adaptive blending ratios and blended supervision targets each contribute to the gains?

    With the sharpening temperature and training budget held constant, could you separately compare fixed versus adaptive blending ratios for trajectory generation, and pure-teacher versus blended distributions for supervision?

    For the fixed-ratio baseline, I would suggest setting the teacher weight close to the average teacher weight of the adaptive method. This would help distinguish gains from per-token adaptation from those due simply to a different overall strength of teacher guidance. If adaptive blending still produces consistent gains after controlling for these factors, it would provide stronger support for the paper’s central design.

2 comments
AuthorsOpenReview ↗

Response to Additional Questions on Reasoning Quality, Training Objectives, and Ablation Studies

Response to Question 1: Distinguishing Harmful Reasoning Deviations from Valid Generation Diversity.

We thank the reviewer for raising this important concern. The key issue is not the single-step token distribution at a fixed prefix, but the trajectory-level accumulation of deviations under blended sampling. Under non-blending policies, low-confidence sampling typically leads to occasional and independent deviations that do not substantially disrupt reasoning, but rather contribute to generation diversity. On the other hand, blending drift can induce consecutive low-confidence sampling, causing errors to accumulate along the trajectory and eventually leading to severe hallucinations and semantically incoherent reasoning.

To further quantify the impact on rollout quality, we evaluate 10,000 sampled rollouts on DAPO-17K:

Rollout PolicyAccuracy (%)
Teacher (Qwen3-4B-GRPO)47.1
Student (Qwen3-1.7B)29.2
Naive Blending (0.5PT+0.5PS0.5P_T+0.5P_S)21.4
ABPD (Ours)37.1

Notably, naive blending performs worse than both the teacher and student, whereas ABPD substantially mitigates this degradation. We also add a hallucination-rate analysis in Section 3.2, where blending-based policies exhibit higher hallucination rates alongside relatively large B1B_1 values. Malformed generations, such as “it the five to roll sixth slipping is allowed exterior roll without figure”, further illustrate this reasoning collapse.

Regarding MM, we use M=1M=1 because it yields the largest common-tail region, providing a conservative measure of potential blending drift rather than a direct indicator of incorrect tokens. Please refer to our response to Pinned Concern 2 for further details.

Response to Question 2: Clarifying the training objective and its practical implementation.

We thank the reviewer for raising this question. We have provided a detailed clarification of the relationship between our theoretical objective and its practical implementation in our response to Pinned Concern 1.

For clarity, our experiments are conducted within the RL-based OPD framework of Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe, using PPO as the downstream optimizer. This experimental setup has already been explicitly stated in the original manuscript. ABPD modifies the rollout generation policy and reward construction while retaining the existing RL optimization pipeline. Accordingly, Equation (7) characterizes the theoretical objective induced by our reward design, rather than a standalone loss directly used for backpropagation.

We refer the reviewer to Pinned Concern 1 for further details.

Response to Question3: Contributions of Adaptive Blending Ratios and Blended Supervision Targets.

We thank the reviewer for this valuable suggestion. To disentangle the contributions of our core design components, we conducted comprehensive ablation studies across all three experimental settings (Qwen-math, DeepSeek-math, and Qwen-code). Specifically, we consider three variants:

(1) w/o Adaptive Mixing: removing the KL budget and fixing the teacher weight to β=0.8\beta=0.8;

(2) w/o Sharpening: removing temperature scaling by setting τ=1\tau=1;

(3) w/ Pure Teacher Supervision: replacing the adaptive blended supervision target $$pi^*$) with the pure teacher distribution (P_t$.

The results are summarized below.

MethodQwen-math (Avg / Pass)DeepSeek-math (Avg / Pass)Qwen-code (Avg / Pass)
ABPD (Full)34.1 / 49.747.7 / 63.946.3 / 65.0
w/o Adaptive Mixing (�7�)�7�)33.3 (-0.8) / 46.9 (-2.8)46.0 (-1.7) / 61.7 (-2.2)45.3 (-1.0) / 63.8 (-1.2)
w/o Sharpening (�8��8�)33.9 (-0.2) / 48.4 (-1.3)45.5 (-2.2) / 60.9 (-3.0)44.0 (-2.3) / 61.8 (-3.2)
w/ Pure Teacher Supervision (�9��9�)33.1 (-1.0) / 47.2 (-2.5)45.9 (-1.8) / 61.4 (-2.5)45.5 (-0.8) / 64.1 (-0.9)

Beyond final performance, we also investigate the underlying training dynamics. For the full method, Appendix G.1–G.3 provides a comprehensive analysis of training dynamics. Furthermore, Appendix G.4–G.6 examines the training dynamics of different ablation variants, including gradient norms, entropy, advantage statistics, and extreme reward behaviors. Together, these analyses provide further insights into the optimization dynamics, supervision quality, and training stability of ABPD.

We sincerely hope that our clarifications and additional experiments have addressed your concerns regarding trajectory-level blending drift and the contributions of ABPD's individual components. We would greatly appreciate your consideration of these results in your final assessment. Thank you again for your valuable feedback.

Dear Reviewer oFUc, as the discussion period draws to a close, we would appreciate hearing if you have any remaining questions or concerns. We are truly grateful for your review and are fully prepared to provide any further clarification or details needed. Thank you very much for your time and thoughtful feedback!

Reviewer_CZWxOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Adaptive Blended Policy Distillation (ABPD) to balance rollout quality and student learnability in LLM distillation. It sharpens teacher and student distributions and adaptively selects their mixing coefficient under a student-relative KL constraint. Experiments show improved overall performance on mathematical reasoning and code generation.

Critical strengths
  1. The paper addresses a meaningful trade-off: teacher-generated responses provide stronger supervision but may be difficult for the student to learn, while student-generated responses can contain errors that weaken teacher guidance.
  2. Adjusting the mixing coefficient at each token is a sensible design that accounts for changing teacher–student disagreement along a rollout.
  3. ABPD achieves the best aggregate results across the evaluated math and code settings. The temperature and KL-budget ablations provide useful insights into the method’s behavior, including the trade-off between drift suppression and generation diversity.
Critical weaknesses
  1. The stability explanation is inconsistent with the stated training objective. Section 4.3 treats the blended policy as a stop-gradient target, making Equation 7 equivalent to cross-entropy training on its generated responses. For fixed rollouts, replacing the teacher log-probability with the blended-policy log-probability changes only a parameter-independent term and leaves the gradient unchanged. The lower reward variance in Figure 5 therefore does not establish the claimed gradient-stabilization mechanism. Please clarify the implemented update and explain how the proposed reward reformulation affects optimization.

  2. The theoretical analysis does not fully explain why blending creates the proposed failure mode. At a fixed prefix, the mixture’s common-tail probability is simply the weighted average of the teacher’s and student’s common-tail probabilities. The temperature bound shows that sharpening suppresses this probability, but does not establish the claimed accumulation of blending-induced errors across a trajectory. Please clarify how the local tail-probability analysis connects to the autoregressive degradation that motivates the paper, and what distinguishes this phenomenon from ordinary low-confidence sampling.

Questions for authors

Please answer the points in weakness.

2 comments
AuthorsOpenReview ↗

Response to Training Objective Clarification and Theoretical Analysis of Blending Drift

Response to Question: The stability explanation is inconsistent with the stated training objective.

We sincerely thank the reviewer for this insightful observation, and we agree that this concern highlights a potential misunderstanding regarding the relationship between our optimization objective and its practical implementation.

The confusion likely stems from viewing our method through the lens of standard cross-entropy (CE) training on fixed rollouts. In fact, our method operates within a standard RL pipeline: rollout generation, reward computation, advantage/return estimation, and policy optimization (e.g., PPO or GRPO).

Our design only modifies the first two stages (rollout and reward), leaving policy optimization untouched. This decoupling from the specific optimizer allows ABPD to be easily integrated into most existing RL distillation frameworks (as summarized in our top-level response to Concern 1). Our implementation follows the on-policy distillation baseline of <Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe>, which is based on PPO optimization.

The actual loss and gradient computation in our experiments is based on PPO optimization driven by our proposed reward, rather than differentiating through the reward itself. Replacing PPO with GRPO would change the gradient dynamics but would not change our reward design. Therefore, in our experiments we analyze reward stability rather than gradient-based metrics directly. Nevertheless, in Appendix G we visualize and analyze all training details, demonstrating that our method yields both more stable rewards and a consistently smaller and more stable Gradient Norm compared to pure-policy and blending-policy baselines, thereby contributing to stable training.

We have added Appendix F (Implementation Details of ABPD), which explicitly describes the differences from the baseline, the actual loss computation, and the core code. We have also updated Section 4.4 accordingly.

Response to Pinned Concern 2: The theoretical analysis does not fully establish trajectory-level blending degradation.

We thank the reviewer for this insightful observation. We agree that blending does not introduce new token-level support at a fixed prefix. The key distinction from ordinary low-confidence sampling lies in the temporal continuity of deviations. For non-blending policies, low-confidence tokens primarily reflect stochastic diversity and tend to occur as isolated deviations, without systematically disrupting the coherence of the reasoning trajectory. In contrast, blending drift can induce consecutive low-probability deviations: an early deviation leads to an unfamiliar prefix, where teacher–student disagreement may further increase, making subsequent deviations more likely. This creates a self-reinforcing process that progressively degrades reasoning coherence, potentially resulting in severe hallucinations and even semantically incoherent generations.

To investigate this phenomenon, we introduce a new Section 3.2, Blending Drift Collapses Reasoning under Autoregressive Generation. By analyzing 10,000 rollouts, we find that non-blending policies exhibit no systematic increase in drift across token positions, whereas blending policies show a pronounced accumulation of drift as generation proceeds. We further observe that hallucination rates remain low for non-blending policies but increase substantially with token position under blending. Moreover, blending can produce semantically incoherent reasoning, such as ``it the five to roll sixth slipping is allowed exterior roll without figure.'' These findings suggest that, unlike ordinary sampling diversity, blending drift introduces progressively accumulated noise that can undermine the semantic and logical coherence of entire reasoning trajectories. We provide further examples in Appendix E and refer the reviewer to our response to Concern 2 for additional details.

We sincerely hope that our clarifications and additional analyses have adequately addressed your concerns, particularly regarding the distinction between ordinary sampling diversity and trajectory-level blending drift. We would greatly appreciate it if you could take our revisions and new experimental evidence into consideration in your final assessment. Thank you again for your thoughtful feedback and constructive suggestions.

Dear Reviewer CZWx, as the discussion period draws to a close, we would appreciate hearing if you have any remaining questions or concerns. We are truly grateful for your review and are fully prepared to provide any further clarification or details needed. Thank you very much for your time and thoughtful feedback!

Reviewer_yvxtOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

ABPD sharpens teacher and student distributions, then adaptively mixes them while constraining divergence from the student. The resulting policy supplies both rollouts and distillation targets. Experiments show moderate improvements on mathematical reasoning and code generation.

Critical strengths
  • The combination of drift analysis, sharpening, and adaptive mixing is interesting and potentially useful.
  • ABPD achieves the best overall averages across both math settings and the code setting, alongside lower reported reward variance.
Critical weaknesses
  • The improvements are modest, and sharpening contributes little in the reported ablation. Overall Avg@4 gains over the strongest competing baseline are approximately 1.0–1.2 points. Sharpening improves Qwen-math from 33.9 to 34.1.

  • The claim of a blending-specific failure needs sharper justification. At a fixed prefix, a probability mixture’s mass outside both confident regions is simply the weighted average of the component masses. It cannot create support absent from both policies. The potentially distinctive issue is therefore how mixed trajectories change subsequent prefixes, which deserves clearer separation from ordinary low-confidence sampling.

Questions for authors

How much improvement comes from adaptive mixing, sharpening, and changing the supervision target separately?

2 comments
AuthorsOpenReview ↗

Response to Ablation Study and Clarification of Blending Drift

Response to Question: How much improvement comes from adaptive mixing, sharpening, and changing the supervision target separately?

To rigorously isolate the contribution of each component, we conducted comprehensive ablation studies across all three experimental settings (Qwen-math, DeepSeek-math, and Qwen-code). We decomposed our framework into three core components: (1) Adaptive Mixing (removing the KL budget and fixing β=0.8), (2) Sharpening (removing temperature scaling, τ=1), and (3) Supervision Target (replacing the adaptive blended target π* with the pure teacher P_t).

MethodQwen-math (Avg / Pass)DeepSeek-math (Avg / Pass)Qwen-code (Avg / Pass)
ABPD (Full)34.1 / 49.747.7 / 63.946.3 / 65.0
w/o Adaptive Mixing (β=0.8)33.3 (-0.8) / 46.9 (-2.8)46.0 (-1.7) / 61.7 (-2.2)45.3 (-1.0) / 63.8 (-1.2)
w/o Sharpening (τ=1)33.9 (-0.2) / 48.4 (-1.3)45.5 (-2.2) / 60.9 (-3.0)44.0 (-2.3) / 61.8 (-3.2)
w/ P_t Supervision33.1 (-1.0) / 47.2 (-2.5)45.9 (-1.8) / 61.4 (-2.5)45.5 (-0.8) / 64.1 (-0.9)

We have updated Table 2 and Table 3 in the revised paper and added a comprehensive discussion in Section 5.3. Furthermore, we provide detailed analyses of the training dynamics (Gradient Norm, Entropy, Advantage statistics, and Extreme Rewards) in Appendix G4, G5, and G6. These detailed statistical results support the following conclusions:

Sharpening mitigates blending drift. Its benefits are most pronounced on tasks with severe drift (e.g., DeepSeek-math); We admit that for inherently stable tasks, the gain may be marginal.

Adaptive budget and supervision jointly improve training metrics. Budget-based dynamic sampling enhances rollout learnability, while our supervision strategy avoids unreliable high-entropy teacher signals. Together, they yield consistently lower Gradient Norm, more stable rewards, and significantly lower Supervision Entropy (as validated in G5 and G6), translating into stable performance gains across all benchmarks.

Response to Weakness2: The claim of a blending-specific failure needs sharper justification.

We thank the reviewer for this insightful observation. We agree that at a fixed prefix, a mixture cannot create entirely new support. However, the blending-specific failure lies exactly in how mixed trajectories change subsequent prefixes via autoregressive compounding.

The key distinction is that, unlike occasional and isolated low-confidence samples under non-blending policies, which mainly contribute to generation diversity, blending can induce consecutive low-confidence sampling across successive prefixes. An early deviation can lead to increasingly unfamiliar prefixes, further amplifying teacher–student disagreement and causing deviations to accumulate autoregressively. This compounding effect, rather than the creation of new token-level support, is what makes blending drift a distinctive failure mode that can progressively undermine reasoning coherence.

As added in our new Section 3.2 Blending Drift Collapses Reasoning under Autoregressive Generation, we empirically observe that blending drift increases significantly as the token position grows, accompanied by a corresponding surge in hallucination rates. This is strong evidence that single-step drift compounds autoregressively, progressively amplifying hallucination until the reasoning rollout completely collapses. Furthermore, we demonstrate that our method, by suppressing this drift, achieves hallucination rates comparable to the Off-Policy in long rollouts. We also provide concrete examples of reasoning collapse in Section 3.1 and Appendix E to visually illustrate this process.

We sincerely hope that our responses and additional experiments have addressed your concerns and clarified the contributions of our work. We would be grateful if you could take these clarifications into consideration in your final assessment. Thank you again for your thoughtful feedback and valuable suggestions.

Dear Reviewer yvxt, as the discussion period draws to a close, we would appreciate hearing if you have any remaining questions or concerns. We are truly grateful for your review and are fully prepared to provide any further clarification or details needed. Thank you very much for your time and thoughtful feedback!

Comments on the paper

1 comment
AuthorsOpenReview ↗

## General Response to all Reviewers and AC for the common concerns

We thank all reviewers for their careful reading and constructive feedback. Before addressing individual comments, we first clarify three common concerns.

Concern 1. Relationship Between the Training Objective and Its Implementation

A general RL-based distillation pipeline typically consists of four components: rollout generation, reward computation, advantage/return estimation, and policy optimization (e.g., PPO or GRPO).

ABPD (ours) improves the first two components (rollout policy design and reward construction) and is agnostic to the downstream optimizer. This modular design allows ABPD to be easily adapted to most existing LLM RL distillation frameworks with only a few lines of code. All experiments are implemented using verl (v0.7.0) with the state-of-the-art on-policy distillation implementation from Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe, and PPO as the downstream optimizer. If the paper could be accepted, we will open-source the complete codebase. We have updated Section 4.4 (Algorithm Summary) in the paper to provide a detailed explanation of this aspect. In the appendix, we also provide the key source code.

Equation 7 (Training Objective) in the original manuscript characterizes the theoretical objective induced by our reward design, rather than the loss directly optimized during policy updates. It establishes a theoretical connection between our reward formulation and the standard forward-KL distillation objective, while the actual policy updates are performed through the downstream RL optimization algorithm, e.g., PPO or GRPO. We have clarified this distinction and provided further explanations in Section 4.3 (Token-level Reward) of the revised manuscript.

Concern 2. How blending drift damages reasoning rollouts?

Unlike low-confidence sampling under non-blending policies, which introduces occasional & isolated deviations that contribute to generation diversity, blending drift leads to consecutive low-confidence sampling during autoregressive generation, progressively accumulating errors and undermining reasoning coherence.

We have added a new subsection, Section 3.2: Blending Drift Collapses Reasoning under Autoregressive Generation, to fully explain this in detail. In this section, we analyze 10,000 rollouts to examine how blending drift and hallucination rates evolve across token positions under blended and non-blended policies. We also provide illustrative examples in the main text and Appendix E to further validate these findings.

Notably, we find that ABPD (ours) substantially reduces the hallucination rate to a level comparable to that of off-policy distillation by mitigating blending drift, providing further empirical evidence of the important role blending drift plays in hallucination.

Concern 3. Performance and Ablation Studies

We supplement the ablation studies across all experimental settings with three additional variants: ABPD w/o Adaptive Mixing, ABPD w/o Sharpening, and ABPD w/ Pure Teacher Supervision, which together examine the contributions of the core design components of our method.

MethodQwen-math (Avg / Pass)DeepSeek-math (Avg / Pass)Qwen-code (Avg / Pass)
ABPD (Full)34.1 / 49.747.7 / 63.946.3 / 65.0
w/o Adaptive Mixing (β=0.8)(\beta=0.8)33.3 (-0.8) / 46.9 (-2.8)46.0 (-1.7) / 61.7 (-2.2)45.3 (-1.0) / 63.8 (-1.2)
w/o Sharpening (τ=1)(\tau=1)33.9 (-0.2) / 48.4 (-1.3)45.5 (-2.2) / 60.9 (-3.0)44.0 (-2.3) / 61.8 (-3.2)
w/ Pure Teacher Supervision �0��0�33.1 (-1.0) / 47.2 (-2.5)45.9 (-1.8) / 61.4 (-2.5)45.5 (-0.8) / 64.1 (-0.9)

Beyond final performance, we further investigate the underlying training dynamics. Appendix G.1–G.3 provides a comprehensive analysis of the full ABPD method, while Appendix G.4–G.6 examines the training dynamics of the ablation variants. These analyses cover gradient norms, policy entropy, advantage statistics, and extreme reward behaviors, offering complementary insights into optimization dynamics, supervision quality, and training stability.

We have updated Tables 2 and 3 and added the corresponding discussions in Section 5.3 and Appendix G.

Summary of Manuscript Revisions

  • Section 3.2: Added an analysis of blending drift and its impact on reasoning.
  • Section 4.4: Added an algorithm overview to clarify the implementation of ABPD.
  • Section 5.3: Added comprehensive ablation studies.
  • Appendix E: Added examples of reasoning collapse caused by blending drift.
  • Appendix F: Added implementation details and key code snippets.
  • Appendix G: Added comprehensive training dynamics analyses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.