acceptodds
Under review as a conference paper at ICLR 2027

MOPD: Multi-Rollout On-Policy Distillation via Peer Successes and Failures

Abstract

Large language models are often post-trained with sparse verifier rewards, which indicate whether a sampled trajectory succeeds but provide limited guidance about where reasoning succeeds or fails. On-policy distillation (OPD) offers denser token-level supervision by training on student-generated trajectories, yet existing methods typically distill each rollout independently and ignore the other attempts sampled for the same prompt. We introduce Multi-Rollout On-Policy Distillation (MOPD), a peer-conditioned distillation framework that uses the student's local rollout group to construct more informative teacher signals. MOPD conditions the teacher on both successful and failed peer rollouts: successes provide positive evidence for valid reasoning patterns, while failures provide structured negative evidence about plausible mistakes to avoid. Experiments on coding, mathematical reasoning, scientific question answering, and tool-use benchmarks show that MOPD improves over standard on-policy baselines on most benchmarks, and also improves over baselines that reuse the same rollout group through verified-success SFT or preference optimization. Under matched demonstration budgets, random peers, peers from unrelated prompts, and success-only contexts all fall short of the mixed context, indicating that the gain comes from the same-prompt success–failure contrast rather than from a longer teacher context. Teacher-signal analysis shows the same ordering: mixed contexts align teacher scores with verifier rewards better than any single-type context. These results indicate that effective on-policy distillation should exploit the student's multi-rollout trial-and-error behavior rather than treating rollouts as isolated samples. Code is available at https://anonymous.4open.science/r/mopd_code-C7BC/.

Primary area:
reinforcement learning
Keywords:
on policy distillation, peer information, self-distillation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_RSfqOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

Standard GRPO generates multiple rollouts for each prompt and trains the model using advantage-weighted updates based on their rewards. MOPD starts from the same kind of rollout group, but uses it to construct a self-teacher: the model receives a few successful rollouts, when available, together with a failed rollout labeled as a mistake to avoid. This extra context gives the same model a better-informed next-token distribution, which becomes the target for on-policy distillation along the student’s own rollouts. The student learns without seeing the peer examples itself, and no separate, larger teacher is needed. The paper reports improvements over SDPO across the evaluated tasks and over GRPO on most benchmarks, with mixed results against GRPO on mathematics.

Critical strengths

The main strength is a distinct way to construct a self-teacher from existing rollouts. MOPD jointly conditions the teacher on successful and failed peer attempts, using their contrast to provide richer supervision.

The paper shows broad empirical gains against the benchmarks considered. Improvements span coding, mathematics, science QA, and tool use, with multiple model sizes and families; broad evaluation is a plus.

The paper has useful ablations. Comparisons with success-only, random, and unrelated peer contexts support the value of the proposed teacher construction.

The paper is clearly written.

Critical weaknesses

The two main weaknesses of this paper are

(1) Limited novelty: there are now many different ways to make a "teacher" by reusing the rollouts; this is a fairly obvious thing to try, and its also fairly obvious it should work better than, eg, GRPO. The evaluations are solid, but the idea has limited novelty / aha factor.

(2) The paper misses comparisons against several methods in exactly the same vein that appeared at least 2-3 months before the submission deadline. these include:

OGLS-SD: https://arxiv.org/html/2605.12400v1 (also makes token-level guidance from successful and failed rollouts, but in a different way from current paper)

HSD: https://arxiv.org/html/2606.15576v1 (like current paper, makes teacher by conditioning on correct rollout, but also plus original ground truth answer - and no failures)

The paper would be better positioned if it was a comparison of the many different ways to make a teacher from a bunch of rollouts.

And of course the real problem to solve here is what happens when positive rollouts are hard to obtain; this paper does not (afaik) have any implications for that

Questions for authors

It would be good to include a short example of how exactly the teacher was made (i.e. how exactly successful rollouts were compressed back into context). Did it use another powerful LLM for summarization ? (if so, that makes the whole paper that much weaker)

2 comments

We thank the reviewer for their valuable time and constructive feedback.

W1: Novelty. W2 comparison.** From the reviewer's perspective, MOPD may look like simply adding failed peer rollouts to the teacher context. The design, however, comes from a specific observation about the implicit reward in on-policy distillation.

For a success-conditioned teacher, the ideal token-level reward is a likelihood ratio between success and failure, so a teacher shown only successes also rewards structure that correct and incorrect solutions share. Adding a failed peer to the same context cancels that shared component, so the implicit reward credits what is relatively correct and penalizes what is relatively wrong.

To our knowledge, this analysis of positive-only self-teachers, and the choice to resolve it inside the teacher's context rather than through logit arithmetic (as in OGLS-SD), have not appeared in prior on-policy distillation work.

In short, writing a demo's match as shared structure cc (common to the prompt's correct and incorrect solutions) plus an outcome-specific part dd, the implicit reward under each context is below; the derivation and the model are in the new appendix.

Teacher contextImplicit reward A(a)A(a)Effect
Successes only (SDPO, HSD)β(c+dS)\beta(c + d_S)Shared structure is rewarded: credit for resembling the demo, not for being right
Failure only−γ(c+dF)-\gamma(c + d_F)The correct skeleton is penalized along with the error
Logit steering over separate passes (OGLS-SD)λ(dˉS−dˉF)\lambda(\bar d_S - \bar d_F)Shared structure cancels in logit space, but each demo is read in its own pass: the teacher never sees a success and a failure together, and needs one extra pass per demo
Successes + failure in one context (MOPD, γ≈β\gamma \approx \beta)β(dS−dF)\beta(d_S - d_F)Shared structure cancels within a single pass: credit for what is relatively correct, penalty for what is relatively wrong

OGLS-SD reaches the same cancellation by averaging logits from separate single-demo passes (z+λ(zˉ+−zˉ−)z + \lambda(\bar z^+ - \bar z^-)), which supports the motivation above: rollout-built teachers need failure evidence. Under the additive model, the two constructions coincide. They differ in two ways the model does not capture. First, with both peers in one context, the teacher can compare them and locate where they diverge, which is not possible when each demo is read alone. Second, MOPD needs one teacher pass per target rather than one per demo plus one.

Neither HSD nor OGLS-SD has released code. We therefore reimplement both from their papers within our trainer, keeping every other setting identical to MOPD: HSD as the ground-truth answer plus one successful sibling (answer only when no sibling succeeded), and OGLS-SD as logit steering z+λ(zˉ+−zˉ−)z + \lambda(\bar z^+ - \bar z^-) over separate single-trace teacher passes.

W2 And of course the real problem to solve here is what happens when positive rollouts are hard to obtain; this paper does not (afaik) have any implications for that.###

Response. This concern is addressed directly in the paper: Section 5.5 is titled "Behavior when successes are scarce", and Appendix C is a dedicated scarce-success experiment.

W2 The paper would be better positioned if it was a comparison of the many different ways to make a teacher from a bunch of rollouts.###

Response. It is what the paper's analysis sections already contain. We compare the ways of building a teacher from the same rollout group at multiple levels:

Peer composition, after training (Table 6).

Teacher contextSciQA-Chem mean@8LCBv6 mean@8
1 failure35.6746.72
1 success51.9049.49
2 successes52.6255.62
1 success + 1 failure60.5058.90
2 successes + 1 failure (MOPD)74.2961.82

Selection rule and objective at matched budget (Tables 7–8, LCBv6, Qwen3-4B). With the same rollouts, three peer slots and the same number of updates:

Way to build the teacher / use the groupmean@8
Verified-success SFT on the group37.36
Unrelated-prompt successes37.10
DPO on same-prompt success–failure pairs40.26
Random same-prompt peers (labels ignored)45.83
Success-only, 3 successes49.61
Verifier-ranked top-351.41
MOPD: 2 successes + 1 failure57.01

Q1: A short example of how exactly the teacher was made (i.e. how exactly successful rollouts were compressed back into context). Did it use another powerful LLM for summarization ?

Response. No. No external model is used at any stage, and nothing is compressed or rewritten. The teacher context consists of the peer rollouts themselves, copied verbatim into fixed text templates (Eqs. 7–9). The teacher is the same model as the student (an EMA of the student's weights), and the review's own summary notes that "no separate, larger teacher is needed".

The templates are defined verbatim in verl/trainer/config/actor/actor.yaml (lines 158–185, self_distillation block), and the teacher prompt is assembled in _maybe_build_self_distillation_batch in verl/trainer/ppo/ray_trainer.py.

Reviewer_X6UUOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MOPD constructs teacher feedback from up to two successful peers and one failed peer in the student's rollout group, excluding the target response. It applies on-policy distillation across mathematics, science, coding, and tool use.

Critical strengths
  • The method reuses existing rollouts, with explicit peer-selection and fallback rules.
  • LiveCodeBench gains over SDPO are 8.17 and 12.33 mean@8 points for the two model sizes.
  • Repeated science/peer-selection runs, bootstrap intervals, verifier-noise tests, and runtime measurements strengthen the evaluation.
Critical weaknesses
  1. Loss specification: Equation (6) uses mα=(1−α)πθ+αqm_\alpha=(1-\alpha)\pi_\theta+\alpha q and Dα=(1−α)KL(πθ∣mα)+αKL(q∣mα).D_\alpha=(1-\alpha)\mathrm{KL}(\pi_\theta\\|m_\alpha)+\alpha\mathrm{KL}(q\\|m_\alpha).

Thus D0=D1=0D_0=D_1=0, although mathematics reportedly uses α=1\alpha=1. An endpoint branch could explain the implementation, this is a written inconsistency.

  1. Attribution: The mixed-versus-success-only comparison does not establish matched eligible prompts or realized token budgets. Some central results remain single-run despite useful repeated science experiments.
  2. General lack of details for all experiments, I appreciate the code availability, but all configurations should be explicit and easy to run.
Questions for authors
  1. What exact executed loss and endpoint handling produced the mathematics results?
  2. Please provide coding problem IDs, train/test lists, and main-versus-diagnostic run details.
  3. Can you compare peer mixtures on common eligible prompts with matched tokens and paired seeds, document the SDPO configuration, and report matched-time results?
3 comments

We thank the reviewer for their valuable time and constructive feedback.

Weakness 1: written inconsistency of the loss specification.

Equation (6) uses mα=(1−α)πθ+αqm_\alpha = (1-\alpha)\pi_\theta + \alpha q and Dα=(1−α)KL(πθ∥mα)+αKL(q∥mα)D_\alpha = (1-\alpha)\mathrm{KL}(\pi_\theta\|m_\alpha) + \alpha\mathrm{KL}(q\|m_\alpha). Thus D0=D1=0D_0 = D_1 = 0, although mathematics reportedly uses α=1\alpha = 1. An endpoint branch could explain the implementation, this is a written inconsistency.

Response. There's a written inconsistency. We added one sentence in the PDF stating that the code handles the endpoints separately: when α\alpha is 0 or 1, it does not use the mixture form of Eq. (6) at all, but takes two dedicated branches.

Weakness 2 and Q3: Attribution (matched eligible prompts and realized token budgets). Common eligible prompts, matched tokens.

The mixed-versus-success-only comparison does not establish matched eligible prompts or realized token budgets. Some central results remain single-run despite useful repeated science experiments.

Response. In Table 7 we already matched the number of peer rollouts (eligible prompts) in the teacher context: every row places exactly three peers (unrelated-prompt peers: 3 successes; random peers: 3, labels ignored; success-only: 3 successes; verifier-ranked: top-3 by reward; MOPD: 2 successes + 1 failure).

**Weakness 3 and Q3: General lack of details for all experiments, I appreciate the code availability, but all configurations should be explicit and easy to run. **

Response. The repository includes step-by-step instructions for launching every experiment, and we have verified that the released configurations run end to end. All configurations are in trainer/config/; the main training configuration, including the peer-context construction, divergence choice (α\alpha), top-kk and sequence budgets, is trainer/config/actor/actor.yaml.

Q1, Q2, Q3 Response. It seems like these 3 questions are all implementation questions. Here is the detailed version in response to your questions, as well as where in the code.

Q1. What exact executed loss and endpoint handling produced the mathematics results?

The executed loss is in verl/trainer/ppo/core_algos.py (lines 1139–1161). The endpoints take dedicated branches :

if alpha == 0.0:    # forward KL: KL(q || pi_theta)
    kl_loss = F.kl_div(student_logp, teacher_logp, reduction="none", log_target=True)
elif alpha == 1.0:  # reverse KL: KL(pi_theta || q)
    kl_loss = F.kl_div(teacher_logp, student_logp, reduction="none", log_target=True)
else:               # generalized JSD with m = (1-alpha) * pi_theta + alpha * q
    ...
    kl_loss = torch.lerp(kl_student, kl_teacher, alpha)

The mathematics runs use alpha = [[1.0]] (run names encode alpha${ALPHA}), i.e., the per-token reverse KL KL(πθ ∥ qpeer)\mathrm{KL}(\pi_\theta \,\|\, q^{\text{peer}}). Both distributions are restricted to the teacher's top-100 tokens plus one tail bucket holding the remaining probability mass (distillation_add_tail: True, lines 1113–1129). The teacher is an EMA of the student (teacher_update_rate: 0.05) under stop-gradient. The revision states Eq. (6) piecewise and corrects the description of the top-kk step in Appendix B.

Q2. Coding problem IDs, train/test lists, and main-versus-diagnostic run details.

Q2 (splits and runs). The LCBv6 split is temporal and is defined in data/utils/livecodebench.py:

LCB_TEST_CUTOFF = datetime(2025, 2, 1)
LCB_TRAIN_CUTOFF = datetime(2025, 2, 1)

if dataset_split == "train":
    ds = ds.filter(lambda ex: ex["contest_date"] < LCB_TRAIN_CUTOFF)
else:
    ds = ds.filter(lambda ex: ex["contest_date"] >= LCB_TEST_CUTOFF)
  • Train: LiveCodeBench problems with contest dates before Feb 1, 2025 ([[N]] problems).
  • Test: the 131 problems with contest dates from Feb 1 to Apr 30, 2025 (until=datetime(2025, 5, 1)), none of which appear in training.

The authors answered most of my concerns clearly. I'm raising the score to 3.

Thank you for reconsidering our work and raising your score!

Reviewer_v7woOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MOPD to better exploit multiple student-generated attempts at the same problem during on-policy distillation. It conditions a self-teacher on both successful and failed peer trajectories to produce token-level supervision. Experiments across coding, math, science QA, and tool use show gains over several on-policy and rollout-reuse baselines. Ablations and teacher-signal analyses support the value of combining successful and failed peers.

Critical strengths
  1. A simple and practically appealing design.
    MOPD uses successful and failed peer rollouts to construct teacher supervision, reusing existing trajectories and verifier outcomes without additional process annotations. Its limited changes to the training framework make it a practical extension of on-policy self-distillation.

  2. Broad evaluation with substantial reported gains.
    Experiments cover four task families and two model families. On Qwen3-8B, MOPD improves LiveCodeBench-v6 mean@8 from 49.49 to 61.82 over SDPO and science QA chemistry accuracy from 62.91 to 74.29. Three-run statistics support the science QA gains, strengthening the evidence for the method’s empirical value.

Critical weaknesses
  1. Limited methodological novelty and inaccurate positioning of related work.
    MOPD mainly modifies the teacher context in on-policy self-distillation by jointly incorporating successful and failed peer rollouts. However, SDPO already uses successful trajectories from the same rollout group, contrary to the appendix’s characterization of its context as external to that group. The closely related HSD paper, publicly released in June 2026, is discussed only in the appendix as “subsequent work,” without a clear chronological reference. The main text should clarify these overlaps and establish MOPD’s incremental contribution.

  2. Unclear context construction and an incorrect example in Figure 1.
    “Summary of another solution” lacks an explanation of its source and generation procedure, whereas the method describes concatenating raw peer rollouts. The figure should clarify whether summarization is actually used. Moreover, the example labeled successful calls .count() on a dict_values object, which raises an AttributeError in Python 3 and should be corrected.

Questions for authors

please see Weaknesses

1 comment

We thank the reviewer for their valuable time and constructive feedback.

Weakness 1: Limited methodological novelty and inaccurate positioning of related work.

MOPD mainly modifies the teacher context in on-policy self-distillation by jointly incorporating successful and failed peer rollouts. However, SDPO already uses successful trajectories from the same rollout group, contrary to the appendix's characterization of its context as external to that group. The closely related HSD paper, publicly released in June 2026, is discussed only in the appendix as "subsequent work," without a clear chronological reference. The main text should clarify these overlaps and establish MOPD's incremental contribution.

Response. The reviewer mainly has 3 concerns, and we politely address them.

  1. SDPO already uses successful trajectories. Yes. Our sentence in Appendix A has been corrected. And SDPO using successful peers doesn't affect our novelty.

  2. HSD, clarify these overlaps. HSD never places a failed rollout in the teacher's context. The teacher would only see the successful rollouts.

  3. Novelty From the reviewer's perspective, MOPD may look like simply adding failed peer rollouts to the teacher context. The design, however, comes from a specific observation about the implicit reward in on-policy distillation.

For a success-conditioned teacher, the ideal token-level reward is a likelihood ratio between success and failure, so a teacher shown only successes also rewards structure that correct and incorrect solutions share. Adding a failed peer to the same context cancels that shared component, so the implicit reward credits what is relatively correct and penalizes what is relatively wrong.

To our knowledge, this analysis of positive-only self-teachers, and the choice to resolve it inside the teacher's context rather than through logit arithmetic (as in OGLS-SD), have not appeared in prior on-policy distillation work.

In short, writing a demo's match as shared structure cc (common to the prompt's correct and incorrect solutions) plus an outcome-specific part dd, the implicit reward under each context is below; the derivation and the model are in the new appendix.

Teacher contextImplicit reward A(a)A(a)Effect
Successes only (SDPO, HSD)β(c+dS)\beta(c + d_S)Shared structure is rewarded: credit for resembling the demo, not for being right
Failure only−γ(c+dF)-\gamma(c + d_F)The correct skeleton is penalized along with the error
Successes + failure (MOPD, γ≈β\gamma \approx \beta)β(dS−dF)\beta(d_S - d_F)Shared structure cancels: credit for what is relatively correct, penalty for what is relatively wrong

Weakness 2: Unclear context construction and an incorrect example in Figure 1.

“Summary of another solution” lacks an explanation of its source and generation procedure, whereas the method describes concatenating raw peer rollouts. The figure should clarify whether summarization is actually used. Moreover, the example labeled successful calls .count() on a dict_values object, which raises an AttributeError in Python 3 and should be corrected.

Response. We have updated Figure 1 accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.