- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Standard GRPO generates multiple rollouts for each prompt and trains the model using advantage-weighted updates based on their rewards. MOPD starts from the same kind of rollout group, but uses it to construct a self-teacher: the model receives a few successful rollouts, when available, together with a failed rollout labeled as a mistake to avoid. This extra context gives the same model a better-informed next-token distribution, which becomes the target for on-policy distillation along the student’s own rollouts. The student learns without seeing the peer examples itself, and no separate, larger teacher is needed. The paper reports improvements over SDPO across the evaluated tasks and over GRPO on most benchmarks, with mixed results against GRPO on mathematics.
Critical strengths
The main strength is a distinct way to construct a self-teacher from existing rollouts. MOPD jointly conditions the teacher on successful and failed peer attempts, using their contrast to provide richer supervision.
The paper shows broad empirical gains against the benchmarks considered. Improvements span coding, mathematics, science QA, and tool use, with multiple model sizes and families; broad evaluation is a plus.
The paper has useful ablations. Comparisons with success-only, random, and unrelated peer contexts support the value of the proposed teacher construction.
The paper is clearly written.
Critical weaknesses
The two main weaknesses of this paper are
(1) Limited novelty: there are now many different ways to make a "teacher" by reusing the rollouts; this is a fairly obvious thing to try, and its also fairly obvious it should work better than, eg, GRPO. The evaluations are solid, but the idea has limited novelty / aha factor.
(2) The paper misses comparisons against several methods in exactly the same vein that appeared at least 2-3 months before the submission deadline. these include:
OGLS-SD: https://arxiv.org/html/2605.12400v1 (also makes token-level guidance from successful and failed rollouts, but in a different way from current paper)
HSD: https://arxiv.org/html/2606.15576v1 (like current paper, makes teacher by conditioning on correct rollout, but also plus original ground truth answer - and no failures)
The paper would be better positioned if it was a comparison of the many different ways to make a teacher from a bunch of rollouts.
And of course the real problem to solve here is what happens when positive rollouts are hard to obtain; this paper does not (afaik) have any implications for that
Questions for authors
It would be good to include a short example of how exactly the teacher was made (i.e. how exactly successful rollouts were compressed back into context). Did it use another powerful LLM for summarization ? (if so, that makes the whole paper that much weaker)
2 comments
Q1: A short example of how exactly the teacher was made (i.e. how exactly successful rollouts were compressed back into context). Did it use another powerful LLM for summarization ?
Response. No. No external model is used at any stage, and nothing is compressed or rewritten. The teacher context consists of the peer rollouts themselves, copied verbatim into fixed text templates (Eqs. 7–9). The teacher is the same model as the student (an EMA of the student's weights), and the review's own summary notes that "no separate, larger teacher is needed".
The templates are defined verbatim in verl/trainer/config/actor/actor.yaml (lines 158–185, self_distillation block), and the teacher prompt is assembled in _maybe_build_self_distillation_batch in verl/trainer/ppo/ray_trainer.py.
We thank the reviewer for their valuable time and constructive feedback.
W1: Novelty. W2 comparison.** From the reviewer's perspective, MOPD may look like simply adding failed peer rollouts to the teacher context. The design, however, comes from a specific observation about the implicit reward in on-policy distillation.
For a success-conditioned teacher, the ideal token-level reward is a likelihood ratio between success and failure, so a teacher shown only successes also rewards structure that correct and incorrect solutions share. Adding a failed peer to the same context cancels that shared component, so the implicit reward credits what is relatively correct and penalizes what is relatively wrong.
To our knowledge, this analysis of positive-only self-teachers, and the choice to resolve it inside the teacher's context rather than through logit arithmetic (as in OGLS-SD), have not appeared in prior on-policy distillation work.
In short, writing a demo's match as shared structure c (common to the prompt's correct and incorrect solutions) plus an outcome-specific part d, the implicit reward under each context is below; the derivation and the model are in the new appendix.
OGLS-SD reaches the same cancellation by averaging logits from separate single-demo passes (z+λ(zˉ+−zˉ−)), which supports the motivation above: rollout-built teachers need failure evidence. Under the additive model, the two constructions coincide. They differ in two ways the model does not capture. First, with both peers in one context, the teacher can compare them and locate where they diverge, which is not possible when each demo is read alone. Second, MOPD needs one teacher pass per target rather than one per demo plus one.
Neither HSD nor OGLS-SD has released code. We therefore reimplement both from their papers within our trainer, keeping every other setting identical to MOPD: HSD as the ground-truth answer plus one successful sibling (answer only when no sibling succeeded), and OGLS-SD as logit steering z+λ(zˉ+−zˉ−) over separate single-trace teacher passes.
W2 And of course the real problem to solve here is what happens when positive rollouts are hard to obtain; this paper does not (afaik) have any implications for that.###
Response. This concern is addressed directly in the paper: Section 5.5 is titled "Behavior when successes are scarce", and Appendix C is a dedicated scarce-success experiment.
W2 The paper would be better positioned if it was a comparison of the many different ways to make a teacher from a bunch of rollouts.###
Response. It is what the paper's analysis sections already contain. We compare the ways of building a teacher from the same rollout group at multiple levels:
Peer composition, after training (Table 6).
Selection rule and objective at matched budget (Tables 7–8, LCBv6, Qwen3-4B). With the same rollouts, three peer slots and the same number of updates: