acceptodds
Under review as a conference paper at ICLR 2027

On-Policy Distillation with Negative-Policy Rollouts

Abstract

On-policy distillation (OPD) has been widely studied as a post-training method in which a student model obtains token-level supervision from a stronger teacher on its own rollouts. Recent studies have improved OPD through alternative distillation reward formulations and teacher configurations, while the objective of distillation remains centered on mimicking the teacher. However, when a stronger teacher has limited distributional overlap with the student, such positive guidance can provide insufficient learning signals. In this work, we introduce **N**egative-**P**olicy **OPD** (**NP-OPD**), which complements teacher supervision with rollouts from a lower-performing, lower-capability negative policy that serves as a negative reference for the student. Rather than modifying the distillation reward formulation, NP-OPD introduces the negative policy at the rollout stage, continuously supplying tokens preferred by the negative policy over the teacher so that they remain exposed to teacher supervision throughout training. This provides an explicit negative signal through negative-policy rollouts while preserving the positive teacher supervision used in OPD. Through extensive experiments, we show that NP-OPD improves OPD across model scales, generation modes, reasoning domains, and different OPD variants. Furthermore, our analyses show that NP-OPD effectively suppresses tokens preferred by the negative policy over the teacher and moves the student away from the negative policy. These results support our design of introducing negative signals through negative-policy rollouts and provide new insight into the role of the rollout policy in OPD.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
On-policy distillation, post training, reasoning model, negative-policy rollouts

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_pJa9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper takes a preference optimization view on on-policy distillation by aiming to learn from teacher and "negative teacher" rollouts. A "negative teacher" is defined as a weaker model (smaller or weaker model). The central idea is that in the proposed setup, the teacher has a positive influence on the student improvement and the negative teacher still helps student improve by clarifying which direction to avoid. There is no reward formulation modification and no sparse OPD setup where one would then need to select which tokens to grade. The paper proposes two metrics to track learning - NSP (negative policy suppression precision, higher is better) and NOR (negative overlap reduction). These metrics track the effect of the negative rollouts in comparison to the teacher rollouts.

While the idea is simple - if learning from positive and negative samples is common in ML, why not use two policies to bootstrap on policy distillation - it is developed well with supporting metrics and experimental setup. The KL formulation assumption is a little non-standard but it seems to work in the settings tested. The gains are not always noticeable, but overall the method provides a net benefit. The cost of implementing this in practice could be of slight concern, but the authors offer that the negative rollouts could be done once and stored. Additionally, if the distillation converges quicker, the cost balances out. One concern is whether the method is general enough and how one might choose the negative rollout model given a task and a student, teacher pair.

Critical strengths

Clearly written paper (slightly slow to get going) with good development of a core scientific concept of preference optimization or labelled learning in an OPD setting, with clear metrics that track if the two learning objectives are being satisfied (is the teacher effective at suppressing negative policy tokens & is the student moving away from the negative policy)

Experiments justify the design choices cleanly.

Exhaustive experimentation across model sizes and task sets to demonstrate gains over a number of OPD settings.

Critical weaknesses

It is assumed that the KL div between the teacher and negative policy is greater than the KL div between student and negative policy. This may not always hold, especially on a sequence level or potentially even task-level.

The student-teacher pairs are from the same model family. Presumably this was chosen as such to keep the same tokenizers. However, in this setting the teacher, student and negative rollout models differ only in size (very likely there is a large overlap in their pre/post training tokens) which would then support the assumption that the teacher is further away from the negative rollout model compared to the student. This clean set of size controls on models may not always be available, so in practice, the teacher, student and negative rollout model may need to be from a different model family and architecture.

Questions for authors

How do you recommend ML practitioners select a weaker model to implement N-OPD? 'Weaker' could mean scoring lower on certain benchmarks or smaller in size, but it's not clear that small is always weaker or that a teacher model is always more correct than a weaker model.

3 comments

We thank the reviewer for recognizing the clear presentation and conceptual development of NP-OPD, as well as the metrics used to examine its learning effects. We also appreciate the positive assessment of our experiments supporting the design choices and our extensive evaluation across model sizes, tasks, and OPD variants.


The KL ordering assumed in Equation 11 may not hold for all sequences or tasks. (Weakness 1)

As the reviewer notes and as stated in our paper, Equation 11 is an assumption. In our setting, we use a lower-capacity model from the same family as the negative policy because its rollouts are expected to contain more tokens requiring teacher-guided suppression, but this choice does not guarantee that Equation 11 always holds. However, because OPD uses token-level learning signals, it can still selectively suppress tokens requiring teacher-guided correction when they occur in a rollout, even if Equation 11 does not hold for every sequence. Our NSP/NOR analyses provide empirical evidence of such selective correction, alongside the performance improvements observed in our experiments.

At the task level, our current approach uses the same negative policy across tasks because the selected model generally performs worse than the student across the evaluated tasks. Selecting different negative policies for different tasks may provide more informative suppressive signals, depending on the weaknesses of each model. We will consider this possibility in future work.


The constraint that the teacher, student, and negative policy belong to the same model family. (Weakness 2)

We retain the original OPD reward formulation and use teacher and student models from the same family, allowing direct comparisons of their token-level log-probabilities with a shared tokenizer. Using teacher-student pairs with different tokenizers requires additional alignment or cross-tokenizer likelihood scoring methods. Extending the OPD supervision to this setting is beyond the scope of our work.

However, the negative policy could come from a different model family. In our additional experiment, we observed that trajectories from Qwen3-0.6B can also benefit the training of the Gemma-E4B student. As shown in the table, Qwen3-0.6B rollouts improve overall average accuracy from 57.91 to 58.65.

Nevertheless, our original NP-OPD using Gemma-E2B rollouts achieves a larger improvement to 60.00. We use same-family models by default because we expect fewer mismatches in tokenization, reasoning templates, and training recipes. Following the reviewer’s suggestion, we will consider broader combinations of teacher, student, and negative-policy model families in future work.

MethodMathCodeScienceAvg.
OPD62.9654.9149.1257.91
NP-OPD (Qwen3-0.6B rollouts)64.7753.8449.1858.65
NP-OPD (Gemma-E2B rollouts)66.4355.1849.8360.00

Practical use of NP-OPD when a suitable smaller model is unavailable. (Weakness 2)

We also consider practical settings where a suitable smaller model is unavailable. As discussed in Section 5 ("Can negative rollouts be synthesized from the student?"), we examine whether negative rollouts can be generated from the student itself by modifying its rollout process, such as increasing the sampling temperature or changing the persona prompt. For Qwen3-1.7B, these approaches improve average accuracy across four benchmarks from 42.9 to 43.8 and 43.9, respectively, as shown in Table 4 of the paper. Although these gains are smaller than those of our default NP-OPD setting (46.3), the results suggest that modifying the student’s rollout process can provide a practical alternative when a separate smaller model is unavailable.


Selection guideline of a weaker model for negative-policy rollouts. (Question 1)

We recommend starting with a smaller model from the same family and checking that its performance is lower than the student’s on task-relevant validation data. Model size provides a practical starting point. This choice is consistent with the use of smaller models to generate rejected responses in preference-based learning methods, such as delta learning [1] and the DPO stage of Olmo 3 [2]. In our Qwen3 and Gemma experiments, smaller models generally show lower benchmark performance, and our rollout comparisons support their use as negative policies.

When a suitable smaller model is unavailable, the alternatives discussed in our response above can also be considered. Beyond these alternatives, unlearning could also be explored as a way to construct negative policies with task-specific weaknesses in future work.

[1] The Delta Learning Hypothesis: Preference Tuning on Weak Data can Yield Strong Gains, COLM 2025 [2] Olmo 3, 2025

Thank you for the detailed response and providing the additional cross-family ablation.

The new result for Qwen3-0.6B negative rollouts with Gemma-E4B student is illustrative, showing that negative policy does not need to share the student's family or tokenizer. As the gains are modest (57.9->58.65) with a slight dip on code, compared to same family (Gemma-E2B), it appears to reinforce the efficacy of the method primarily for same family OPD with negative rollouts (i.e. method benefits from family alignment).

Thank you for the explanations on token-level behaviour and self-rollout option which clarifies the existing framing but does not provide any new evidence on generality or how the weakness of the negative policy impacts the setup. Therefore, my overall assessment remains positive but unchanged.

Thank you for your comment on our rebuttal and for actively participating in the discussion.

The reviewer's constructive feedback and positive assessment are greatly appreciated, and we will sincerely consider the suggestions in our future works. If any additional discussion would be helpful, we would be happy to provide further clarification.

Reviewer_Qqx4OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies using a weaker model as a rollout source for on-policy distillation. NP-OPD mixes these trajectories with student rollouts while retaining the original teacher–student reward. Experiments cover three Qwen student sizes, thinking and non-thinking modes, Gemma, and three reward formulations. The authors also analyze which tokens are suppressed and how student predictions shift relative to the weaker model and teacher.

Critical strengths
  1. The empirical improvement is substantial in several settings. In Table 1, Qwen3-4B improves from 62.2 to 70.5 on the math average and from 40.2 to 51.1 on code. The full mixing-ratio results in Appendix C.2 show that the benefit extends beyond a single selected setting. Compatibility with ExOPD and OPD² makes this a useful modification to the rollout stage.

  2. The analysis includes meaningful alternatives: stronger rollout models, a fixed model at the student's size, and degraded student rollouts. The held-out NOR analysis in Appendix C.4 also extends the observed probability shift beyond training prompts. These checks give the proposed rollout source more support than the headline accuracy comparison alone.

Critical weaknesses
  1. Equation 12 establishes a negative expected reward under the KL ordering in Equation 11, but does not by itself show that the parameter update reduces the likelihood of negative-policy samples. Lower task performance also does not establish that KL ordering. The NSP and NOR results provide useful empirical evidence for the proposed mechanism; this evidence should be separated from the stronger update-direction claim.

  2. Appendix C.2 selects the displayed mixing ratio using domain improvements and the overall evaluation average, but does not identify a separate validation set. The full sweep supports the broad improvements. The remaining issue is how to interpret the selected headline values, especially smaller differences that the authors already describe as comparable. This is a clarification of the selection protocol, not evidence that the reported gains are invalid.

Questions for authors
  1. Can the conclusion following Equation 12 be stated as a conditional motivation, with the NSP/NOR measurements providing the empirical evidence? If a stronger update-direction claim is intended, what additional assumptions make it follow?

  2. Was the displayed mixing ratio chosen using a separate validation set? If the reported benchmarks were used for selection, please make that explicit.

5 comments

We thank the reviewer for recognizing the empirical improvements across settings and mixing ratios, as well as the compatibility of NP-OPD with ExOPD and OPD$^2$. We also appreciate the positive assessment of our comparisons with alternative rollout sources and in-depth analysis, which provide supporting evidence beyond the headline accuracy results.


Can the conclusion following Equation 12 be stated as conditional motivation, with the NSP/NOR measurements providing the empirical evidence? (Weakness 1&Question 1)

Thank you for your suggestion and we revise our paper as follows the reviewer's suggestion. As highlighted in blue in Section 3.2 of the revised paper, we explicitly state that Equation 11 is an assumption, and that the mathematical explanation through Equations 11 and 12 provides a motivation for our method, NP-OPD. We then refer to the NSP/NOR analyses in Section 4.3 as the empirical evidence for the proposed mechanism.

In addition to the revision above, we would like to further discuss the two points raised by the reviewer and provide additional intuition on how NP-OPD works.

On the update direction. We agree that negative expected rewards alone do not guarantee a reduction in the likelihood of negative-policy samples, and our mechanism does not depend solely on this implication. Equation 10 shows that negative-policy rollouts amplify the gradient contributions of token sequences that are more likely under the negative policy than under the student; when their rewards are negative, these are suppressive contributions. NP-OPD thus provides additional opportunities for selective suppression without modifying the original OPD reward. Furthermore, under Equation 11, Equation 12 states that the teacher assigns a lower expected log-likelihood to negative-policy samples than the student does, i.e., Eπn[log⁡π∗−log⁡πθ]<0\mathbb{E}_{\pi_n}[\log\pi^*-\log\pi_\theta]<0. Sufficient alignment toward the teacher would therefore reduce the expected log-likelihood of negative-policy samples, although such a reduction is not guaranteed during training.

On the KL ordering. Lower performance does not mathematically guarantee the KL ordering in Equation 11, and we now state it as an assumption. We expect it to hold in our setting because the negative policy is a smaller model from the same family as the student, whereas the teacher is substantially stronger, so the negative policy is expected to be distributionally closer to the student than to the teacher. Empirically, for example, the mean OPD reward on negative-policy rollouts at initialization is -0.404 for the Qwen3-1.7B, which is the quantity in Equation 12. The rollout-capability comparisons in Figs. 2–3 further support the choice of weaker models as negative references, although they do not directly verify the KL ordering.

Additional intuition. We would like to provide further intuition on the mechanism of NP-OPD. As discussed in lines 81–85 of our paper, our motivation shares some similarities with DPO, in that suppressing undesirable behaviors can be as important as reinforcing desirable ones. From this perspective, negative-policy rollouts can both expose more tokens requiring suppression and sustain this exposure throughout training. In standard OPD, as the student learns to suppress certain behaviors, these behaviors become less likely to appear in subsequent student rollouts. Consequently, the teacher has fewer opportunities to provide suppressive signals for them, even when further suppression may still be beneficial. NP-OPD addresses this limitation by introducing rollouts from a fixed negative policy, which continuously exposes the student to undesirable behaviors regardless of how its own rollout distribution evolves.

We hope that this intuition also helps explain why NP-OPD works, alongside the conditional theoretical motivation from Equations 11 and 12 and the empirical evidence from NSP/NOR.

Was the displayed mixing ratio selected using a separate validation set or the reported evaluation benchmarks? (Weakness 2 & Question 2)

Thank you for pointing this out. The displayed mixing ratios (α\alpha) were selected using the reported evaluation benchmarks and the full results with all evaluated α\alphas are reported in Tables C.2-C.14. We explicitly clarify this selection protocol in lines 269-293 of the revised version.

In addition, the performance gains are broadly observed across benchmarks, and the overall improvement trend is maintained when a separate subset of benchmarks is used to select α\alpha. For example, selecting α\alpha among the evaluated non-zero values using AIME24, LCB, and GPQA, one benchmark from each domain (math, coding, science), yields higher average accuracy than the corresponding baselines on the remaining 10 benchmarks in all 18 settings (three student models (Qwen3-1.7B, Qwen3-4B, and Qwen3-8B), thinking and non-thinking modes, and three OPD variants), with an average improvement of 2.47 points. Furthermore, the full sweep also shows broad improvements across multiple positive α\alpha values. These results support the consistency of the overall gains from negative-policy rollouts. Nevertheless, following the reviewer’s suggestion, we explicitly describe how the displayed mixing ratios were selected in the revision.

Dear Reviewer Qqx4,

Thank you again for your constructive review. As the discussion period is coming to an end, we would like to kindly follow up on our response. Following your suggestions, we revised Section 3.2 to present Equations 11-12 as a conditional motivation and provided further explanation of NP-OPD's mechanism. We also clarified how the displayed mixing ratios were selected, with additional results on a separate subset of benchmarks.

We would appreciate it if you could let us know whether our revision and response address your concerns. We are happy to provide further clarification on any remaining points during the discussion period.

We sincerely appreciate your time and valuable feedback.

Reviewer_RMsWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a negative policy into standard On-Policy Distillation (OPD) to generate rollouts for mix training, aiming to provide negative learning signals and address the problem of insufficient positive supervision in OPD. The proposed method does not modify the token-level distillation reward function. Instead, it changes the rollout distribution so that tokens preferred by the negative policy are continuously exposed to teacher supervision and included in training. The authors argue that this mechanism provides additional negative learning signals, thereby pushing the student model away from the negative policy. Experimentally, the authors evaluate the proposed method on the Qwen3 and Gemma model families, covering different model scales, thinking and non-thinking modes, and 13 benchmarks across mathematics, coding, and science. The results show that NP-OPD improves standard OPD performance in various settings.

Critical strengths

1. Intuitively, introducing negative-policy rollouts for mixed training seems to be a relatively effective approach.

2. The method is simple, has good compatibility, and achieves relatively significant performance improvements in experiments.

Critical weaknesses

1. Limited explanation of why the mechanism works and limited support from ablation experiments. The paper does not sufficiently explain why Negative-Policy Rollouts can improve OPD. In Section 3.2, the authors use Equations (11) and (12) to show that tokens generated by the negative policy have negative expected OPD rewards, and conclude that the student model will move away from the negative policy. However, a negative expected reward does not necessarily mean that the actual gradient will push the student away from the negative policy. At the same time, rollouts generated by a weaker model may only contain errors at a few critical tokens, while the remaining tokens are still reasonable. The authors do not sufficiently explain why increasing the sampling probability of these trajectories can provide effective negative learning signals. Therefore, the theoretical analysis does not sufficiently support the authors' core explanation of why the method works.

2. Missing key ablation experiments, and the existing ablations are insufficient to validate the core mechanism. For example, the authors could conduct an experiment using only tokens with positive rewards from Negative-Policy Rollouts for training. Would this still achieve similar improvements?

3. The paper does not sufficiently analyze the role of the negative policy at different stages of training. As OPD training progresses, the student may gradually approach the teacher. At this point, the distributional difference between the student and the negative policy may become increasingly large. The authors should analyze whether Negative-Policy Rollouts can consistently provide effective learning signals throughout training. Would Negative-Policy Rollouts still be helpful when the student is already close to the teacher? Would different training stages require different negative policies or mixing ratios?

Questions for authors

1. In Section 3.2, the authors use Equations (11) and (12) to show that tokens generated by the negative policy have negative expected OPD rewards, and conclude that the student will move away from the negative policy. However, a negative expected reward does not necessarily mean that the actual gradient update will push the student away from the negative policy. Could the authors further explain or prove this conclusion?

2. The paper argues that the performance improvements come from the additional negative learning signals provided by Negative-Policy Rollouts. Could the authors conduct an ablation experiment that uses only tokens with positive OPD rewards from Negative-Policy Rollouts for training, and compare it with the full NP-OPD? If similar performance improvements can still be achieved, would this indicate that the improvements do not necessarily come from negative learning signals?

3. As training progresses, the student may gradually approach the teacher, while the distributional difference between the student and the negative policy may further increase. Have the authors analyzed the effectiveness of Negative-Policy Rollouts at different stages of training? In particular, can these rollouts still provide useful learning signals when the student is already close to the teacher?

4. Rollouts generated by a weaker model may only contain errors at a few critical tokens, while most of the remaining tokens are still reasonable. In this case, could increasing the sampling proportion of weaker-model rollouts negatively affect these correct tokens? Have the authors analyzed this situation?

5. The paper uses a fixed Negative Policy and controls the number of Negative-Policy Rollouts through the mixing ratio α\alpha. Considering that the student's distribution continuously changes during training, have the authors investigated using different negative policies or dynamically adjusting α\alpha at different training stages? Could a fixed negative policy and mixing ratio limit the long-term effectiveness of the method?

4 comments

We thank the reviewer for finding our approach intuitive. We also appreciate that they highlighted the simplicity of NP-OPD, its compatibility with existing OPD methods, and the performance improvements demonstrated in our experiments.

Why NP-OPD works

A negative expected reward does not necessarily mean that the actual gradient will push the student away from the negative policy. At the same time, rollout generated by a weaker model may only contain errors at a few critical tokens. Could the authors further explain why NP-OPD works? (Weakness 1 & Question 1)

To our understanding, the reviewer asks for additional explanation of why NP-OPD works, because the existing explanation may not hold given two concerns: (1) the reward may not be directly connected to the actual update, considering the update rule of OPD, and (2) if there are only a few critical tokens to suppress in negative-policy rollouts, our method may not work as intended. We would like to address these two concerns first to complement our explanation, and then provide additional intuition on the mechanism of NP-OPD.

From reward to update. OPD is based on token-level signals, following Equation 2. Note that the reward is stop-gradiented, so each sampled token contributes R(yt)∇θlog⁡πθ(yt)R(y_t)\nabla_\theta\log\pi_\theta(y_t), and this is directly connected to the update: a token with a negative reward provides a gradient contribution directed toward reducing its log-probability. In the case of on-policy distillation using student (on-policy) rollouts, the negative expected reward could not affect the update, because the mean-reward contribution cancels in expectation, since Eπθ[∇θlog⁡πθ]=0\mathbb{E}_{\pi_\theta}[\nabla_\theta\log\pi_\theta]=0. However, we utilize negative-policy rollouts, which are a kind of off-policy rollouts with respect to the student. In this case, the mean-reward contribution does not cancel, since Eπn[∇θlog⁡πθ]=−∇θDKL(πn∥πθ)\mathbb{E}_{\pi_n}[\nabla_\theta\log\pi_\theta]=-\nabla_\theta D_{\mathrm{KL}}(\pi_n\Vert\pi_\theta), and the negative expected reward in Equation 12 is connected to the update in expectation: on average over negative-policy rollouts, this term increases DKL(πn∥πθ)D_{\mathrm{KL}}(\pi_n\Vert\pi_\theta).

Few critical tokens. Even if we choose the negative-policy model carefully, not all tokens in its rollouts are judged by the teacher as tokens to suppress. However, as mentioned above, the signal is token-level, and a token with negative reward has its probability reduced. Through these token-level rewards and updates, the student becomes less likely to generate the suppressed tokens, which makes it less likely to reproduce the corresponding tendencies of the negative policy. By the expression "moving away from the negative policy" in our paper, we mean selectively reducing behaviors represented in its rollouts that require teacher-guided suppression, rather than uniformly suppressing all tokens generated by that policy. Thus, as long as there are tokens that clearly receive such a token-level signal and they are suppressed, our method works as intended.

Additional intuition. Additionally, we would like to provide further intuition on the mechanism of NP-OPD. As discussed in lines 81–85 of our paper, our motivation shares some similarities with DPO, in that suppressing undesirable behaviors can be as important as reinforcing desirable ones. From this perspective, negative-policy rollouts can both expose more tokens requiring suppression and sustain this exposure throughout training. In standard OPD, as the student learns to suppress certain behaviors, these behaviors become less likely to appear in subsequent student rollouts. Consequently, the teacher has fewer opportunities to provide suppressive signals for them, even when further suppression may still be beneficial. NP-OPD addresses this limitation by introducing rollouts from a fixed negative policy, which continuously exposes the student to undesirable behaviors regardless of how its own rollout distribution evolves.

Empirical support. Several experimental results provide empirical evidence for our explanation of NP-OPD. As shown below, the positive-reward-only ablation falls below the standard OPD, which is consistent with the importance of the tokens with negative reward. The NSP/NOR analyses in Section 4.3 of the paper also provide empirical evidence for our explanation of suppression, showing how the student’s token probabilities shift relative to the negative policy and the teacher.

We hope our explanation of the connection between the reward and the update, the token-level signals, the additional intuition on NP-OPD, and the empirical results help address the reviewer’s concerns, and we welcome further discussion.

Would training with only positive OPD rewards from negative-policy rollouts achieve similar improvements to NP-OPD? (Weakness 2 & Question 2)

We conducted the suggested positive-only ablation on the Qwen3-1.7B student model in thinking mode. This variant follows the experimental settings as NP-OPD (α=1\alpha=1), except that negative-reward tokens are excluded from the update. The positive-only variant achieves an average accuracy of 44.82 across 13 benchmarks, compared to 51.65 for NP-OPD and 49.67 for standard OPD. Thus, positive learning signals from negative-policy rollouts alone do not reproduce the improvements under the same training setup. These results support the importance of retaining suppressive signals in NP-OPD and are consistent with our explanation of the core mechanism.

MathCodeSciAvg.
OPD60.034.340.949.67
NP-OPD61.837.641.951.65
NP-OPD (positive-only)55.229.336.244.82

Effectiveness of NP-OPD across training stages

Would Negative-Policy Rollouts still be helpful when the student is already close to the teacher? (Weakness 3 & Question 3)

To examine how the effectiveness of NP-OPD changes over training, we evaluate the Qwen3-1.7B student’s average accuracy across 13 benchmarks at 50, 100, and 200 training steps. Since OPD training trains the student to follow the teacher, a student trained for more steps is expected to be closer to the teacher. Both NP-OPD settings, with fixed α=0.5\alpha=0.5 and α=1\alpha=1, outperform OPD with all training steps. From 100 to 200 training steps, OPD loses approximately 0.60 points in average accuracy, whereas NP-OPD with α=0.5\alpha=0.5 gains 0.24 points and NP-OPD with α=1\alpha=1 shows a smaller decrease of 0.43 points. Furthermore, the mixing-ratio schedule ablations presented below show that increasing α\alpha during training achieves higher performance than decreasing it. In particular, replacing the last 50 steps of 100 step OPD training with NP-OPD (No. 7 in the table below) improves the average accuracy, suggesting that negative-policy rollouts remain helpful for a student that has already been trained with OPD. These complementary results support maintaining exposure to negative-policy rollouts in later training stages.

StepOPD (α=0)NP-OPD (α=0.5)NP-OPD (α=1)
5049.2850.6151.52
10049.6750.3551.65
20049.0750.5951.22

Could a fixed negative policy with a fixed mixing ratio limit the long-term effectiveness of the method? (Weakness 3 & Question 5)

We evaluated fixed and dynamic mixing-ratio schedules over 100 training steps while keeping the negative policy unchanged. Here, we show the results of various mixing-ratio schedules on the Qwen3-1.7B student in the thinking mode. The dynamic schedules include linear changes in α\alpha and two-stage switches between OPD and NP-OPD after 50 steps. Among the tested schedules, fixed α=1\alpha=1 achieves the highest average accuracy across 13 benchmarks (51.65), compared with 49.67 for standard OPD. Thus, dynamically adjusting α\alpha does not improve upon the fixed setting in these experiments.

The schedule comparisons also support maintaining negative-policy rollouts in later training stages. Increasing α\alpha from 0 to 1 achieves higher accuracy than decreasing it from 1 to 0 (50.98 vs. 49.94). Similarly, using OPD for the first 50 steps followed by NP-OPD for the remaining 50 steps outperforms the reverse order (50.78 vs. 49.93).

Training scheduleAvg. (13 benchmarks)
1OPD, fixed α=0\alpha=049.67
2NP-OPD, fixed α=1\alpha=151.65
3Linear change α\alpha 0 → 150.98
4Linear change α\alpha 1 → 049.94
5Linear change α\alpha 0.5 → 151.10
6Linear change α\alpha 0.5 → 049.77
7α\alpha=0 50 steps → α\alpha=1 50 steps50.78
8α\alpha=1 50 steps → α\alpha=0 50 steps49.93

We investigated dynamic mixing ratios and found that scheduling α\alpha could sometimes provide additional gains in few settings across various model scales and reward formulations (e.g., OPD$^2$ on Qwen3-8B with increasing α\alpha from 0 to 1). However, gains are not consistent, so we presented our main experiments and results focusing on fixed mixing ratios to show the contribution of negative-policy rollouts under a consistent experimental protocol. We include these ablation studies in the revised paper.

Could increasing the proportion of weaker-model rollouts negatively affect the reasonable tokens within them? (Question 4)

As explained in "Why NP-OPD works" above, increasing the proportion of negative-policy rollouts does not automatically assign negative rewards to all tokens in those rollouts. The original OPD reward continues to determine the learning signal for each token. Indeed, negative-policy rollouts contain many tokens with positive or near-zero OPD rewards.

To examine the actual probability changes, we group tokens in fixed negative-policy rollouts by their initial reward (R0R^0) before training, using the Qwen3-1.7B student in thinking mode. The table below reports the change (Δ\Delta) in log-probability from the initial student under NP-OPD and OPD at the same prefixes.

GroupToken shareNP-OPD Δ\DeltaOPD Δ\DeltaNP-OPD − OPD [95% CI]
Negative (R0<−0.001R^0 < -0.001)47.4%−0.028−0.007−0.021 [−0.024, −0.019]
Near-zero (∣R0∣<0.001\lvert R^0\rvert < 0.001)25.6%−0.0001+0.0002−0.0003 [−0.0005, −0.0002]
Positive (R0≥0.001R^0 \ge 0.001)27.0%+0.591+0.414+0.177 [+0.170, +0.185]

Tokens on which the teacher and the student agree (near-zero in the table) are unchanged under both OPD and NP-OPD. Tokens with positive rewards are not suppressed, and are reinforced more strongly than under OPD, while tokens with negative rewards are suppressed more than under OPD. This indicates that NP-OPD does not harm the generation of reasonable tokens.

Dear Reviewer RMsW,

Thank you again for your detailed review. As the discussion period is coming to an end, we would like to kindly follow up on our response. Following your suggestions, we provided additional explanation and intuition on how NP-OPD works, and conducted the positive-reward-only ablation. We also analyzed NP-OPD across training stages and mixing-ratio schedules, and examined token-level probability changes on negative-policy rollouts.

We would appreciate it if you could let us know whether our response addresses your concerns. We are happy to provide further clarification on any remaining points during the discussion period.

We sincerely appreciate your time and valuable feedback.

Comments on the paper

6 comments
AuthorsOpenReview ↗

General Response

We thank the reviewers for their thoughtful and valuable feedback. We are encouraged that they considered our approach intuitive (Reviewer RMsW) and simple (Reviewer RMsW, Reviewer pJa9), with good compatibility with existing OPD variants such as ExOPD and OPD$^2$ (Reviewer RMsW, Reviewer Qqx4). We are pleased they found the performance improvements significant (Reviewer RMsW, Reviewer Qqx4) and our experiments extensive (Reviewer Qqx4, Reviewer pJa9). They also found that our analyses support the method beyond the accuracy improvements (Reviewer Qqx4, Reviewer pJa9), and that the paper is clearly written and presented (Reviewer pJa9).

We have tried to address all concerns raised by the reviewers. Our responses and revisions are summarized as follows:

  • Clarified how the token-level reward connects to the update and how NP-OPD selectively suppresses tokens within negative-policy rollouts, and conducted the suggested positive-reward-only ablation (W1/Q1 and W2/Q2 by Reviewer RMsW)
  • Provided additional intuition on why NP-OPD works (W1/Q1 by Reviewer RMsW, W1/Q1 by Reviewer Qqx4)
  • Clarified the conditional motivation in Section 3.2 and linked it to the empirical evidence from the NSP/NOR analyses (W1/Q1 by Reviewer Qqx4, W1 by Reviewer pJa9)
  • Analyzed the effectiveness of NP-OPD across training stages, with results at different training steps and a comparison of fixed and dynamic mixing-ratio schedules (W3/Q3 and W3/Q5 by Reviewer RMsW)
  • Provided the token-level analysis of log-probability changes on negative-policy rollouts (Q4 by Reviewer RMsW)
  • Clarified how the displayed mixing ratios were selected, and provided results with α\alpha selected on a separate subset of benchmarks (W2/Q2 by Reviewer Qqx4)
  • Provided the cross-family experiment using Qwen3-0.6B rollouts for the Gemma-4-E4B student, and a guideline for selecting a negative policy (W2 and Q1 by Reviewer pJa9)

Throughout the revised paper, newly added or edited parts are highlighted in blue. Detailed responses to the comments are provided in the individual comments.

Hi authors. Could you please discuss NP-OPD and TOP-D [1] in more detail, and perhaps add more experimental comparisons?

[1] Z Xie et al. Trust Region Policy Distillation. NeurIPS 2026

Hi, thank you for introducing your interesting work, TOP-D.

To our understanding, TOP-D and NP-OPD modify different components of OPD: TOP-D modifies the reward and the update rule, whereas NP-OPD modifies the rollout source. TOP-D addresses the optimization instability and low sample efficiency of standard OPD through an external proximal teacher and internal trust region iterations over on-policy rollouts. NP-OPD keeps the reward formulation unchanged and mixes in rollouts from a lower-capability negative policy that serves as a negative reference. We think that TOP-D belongs to the line of work that modifies the distillation signal, such as ExOPD and OPD$^2$, whereas NP-OPD operates at the rollout stage.

We will consider citing this work in the final version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.