- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper takes a preference optimization view on on-policy distillation by aiming to learn from teacher and "negative teacher" rollouts. A "negative teacher" is defined as a weaker model (smaller or weaker model). The central idea is that in the proposed setup, the teacher has a positive influence on the student improvement and the negative teacher still helps student improve by clarifying which direction to avoid. There is no reward formulation modification and no sparse OPD setup where one would then need to select which tokens to grade. The paper proposes two metrics to track learning - NSP (negative policy suppression precision, higher is better) and NOR (negative overlap reduction). These metrics track the effect of the negative rollouts in comparison to the teacher rollouts.
While the idea is simple - if learning from positive and negative samples is common in ML, why not use two policies to bootstrap on policy distillation - it is developed well with supporting metrics and experimental setup. The KL formulation assumption is a little non-standard but it seems to work in the settings tested. The gains are not always noticeable, but overall the method provides a net benefit. The cost of implementing this in practice could be of slight concern, but the authors offer that the negative rollouts could be done once and stored. Additionally, if the distillation converges quicker, the cost balances out. One concern is whether the method is general enough and how one might choose the negative rollout model given a task and a student, teacher pair.
Critical strengths
Clearly written paper (slightly slow to get going) with good development of a core scientific concept of preference optimization or labelled learning in an OPD setting, with clear metrics that track if the two learning objectives are being satisfied (is the teacher effective at suppressing negative policy tokens & is the student moving away from the negative policy)
Experiments justify the design choices cleanly.
Exhaustive experimentation across model sizes and task sets to demonstrate gains over a number of OPD settings.
Critical weaknesses
It is assumed that the KL div between the teacher and negative policy is greater than the KL div between student and negative policy. This may not always hold, especially on a sequence level or potentially even task-level.
The student-teacher pairs are from the same model family. Presumably this was chosen as such to keep the same tokenizers. However, in this setting the teacher, student and negative rollout models differ only in size (very likely there is a large overlap in their pre/post training tokens) which would then support the assumption that the teacher is further away from the negative rollout model compared to the student. This clean set of size controls on models may not always be available, so in practice, the teacher, student and negative rollout model may need to be from a different model family and architecture.
Questions for authors
How do you recommend ML practitioners select a weaker model to implement N-OPD? 'Weaker' could mean scoring lower on certain benchmarks or smaller in size, but it's not clear that small is always weaker or that a teacher model is always more correct than a weaker model.
3 comments
Thank you for the detailed response and providing the additional cross-family ablation.
The new result for Qwen3-0.6B negative rollouts with Gemma-E4B student is illustrative, showing that negative policy does not need to share the student's family or tokenizer. As the gains are modest (57.9->58.65) with a slight dip on code, compared to same family (Gemma-E2B), it appears to reinforce the efficacy of the method primarily for same family OPD with negative rollouts (i.e. method benefits from family alignment).
Thank you for the explanations on token-level behaviour and self-rollout option which clarifies the existing framing but does not provide any new evidence on generality or how the weakness of the negative policy impacts the setup. Therefore, my overall assessment remains positive but unchanged.
Thank you for your comment on our rebuttal and for actively participating in the discussion.
The reviewer's constructive feedback and positive assessment are greatly appreciated, and we will sincerely consider the suggestions in our future works. If any additional discussion would be helpful, we would be happy to provide further clarification.
We thank the reviewer for recognizing the clear presentation and conceptual development of NP-OPD, as well as the metrics used to examine its learning effects. We also appreciate the positive assessment of our experiments supporting the design choices and our extensive evaluation across model sizes, tasks, and OPD variants.
As the reviewer notes and as stated in our paper, Equation 11 is an assumption. In our setting, we use a lower-capacity model from the same family as the negative policy because its rollouts are expected to contain more tokens requiring teacher-guided suppression, but this choice does not guarantee that Equation 11 always holds. However, because OPD uses token-level learning signals, it can still selectively suppress tokens requiring teacher-guided correction when they occur in a rollout, even if Equation 11 does not hold for every sequence. Our NSP/NOR analyses provide empirical evidence of such selective correction, alongside the performance improvements observed in our experiments.
At the task level, our current approach uses the same negative policy across tasks because the selected model generally performs worse than the student across the evaluated tasks. Selecting different negative policies for different tasks may provide more informative suppressive signals, depending on the weaknesses of each model. We will consider this possibility in future work.
We retain the original OPD reward formulation and use teacher and student models from the same family, allowing direct comparisons of their token-level log-probabilities with a shared tokenizer. Using teacher-student pairs with different tokenizers requires additional alignment or cross-tokenizer likelihood scoring methods. Extending the OPD supervision to this setting is beyond the scope of our work.
However, the negative policy could come from a different model family. In our additional experiment, we observed that trajectories from Qwen3-0.6B can also benefit the training of the Gemma-E4B student. As shown in the table, Qwen3-0.6B rollouts improve overall average accuracy from 57.91 to 58.65.
Nevertheless, our original NP-OPD using Gemma-E2B rollouts achieves a larger improvement to 60.00. We use same-family models by default because we expect fewer mismatches in tokenization, reasoning templates, and training recipes. Following the reviewer’s suggestion, we will consider broader combinations of teacher, student, and negative-policy model families in future work.
We also consider practical settings where a suitable smaller model is unavailable. As discussed in Section 5 ("Can negative rollouts be synthesized from the student?"), we examine whether negative rollouts can be generated from the student itself by modifying its rollout process, such as increasing the sampling temperature or changing the persona prompt. For Qwen3-1.7B, these approaches improve average accuracy across four benchmarks from 42.9 to 43.8 and 43.9, respectively, as shown in Table 4 of the paper. Although these gains are smaller than those of our default NP-OPD setting (46.3), the results suggest that modifying the student’s rollout process can provide a practical alternative when a separate smaller model is unavailable.
We recommend starting with a smaller model from the same family and checking that its performance is lower than the student’s on task-relevant validation data. Model size provides a practical starting point. This choice is consistent with the use of smaller models to generate rejected responses in preference-based learning methods, such as delta learning [1] and the DPO stage of Olmo 3 [2]. In our Qwen3 and Gemma experiments, smaller models generally show lower benchmark performance, and our rollout comparisons support their use as negative policies.
When a suitable smaller model is unavailable, the alternatives discussed in our response above can also be considered. Beyond these alternatives, unlearning could also be explored as a way to construct negative policies with task-specific weaknesses in future work.
[1] The Delta Learning Hypothesis: Preference Tuning on Weak Data can Yield Strong Gains, COLM 2025 [2] Olmo 3, 2025