MOPD: Multi-Rollout On-Policy Distillation via Peer Successes and Failures

Large language models are often post-trained with sparse verifier rewards, which indicate whether a sampled trajectory succeeds but provide limited guidance about where reasoning succeeds or fails. On-policy distillation (OPD) offers denser token-level supervision by training on student-generated trajectories, yet existing methods typically distill each rollout independently and ignore the other attempts sampled for the same prompt. We introduce Multi-Rollout On-Policy Distillation (MOPD), a peer-conditioned distillation framework that uses the student's local rollout group to construct more informative teacher signals. MOPD conditions the teacher on both successful and failed peer rollouts: successes provide positive evidence for valid reasoning patterns, while failures provide structured negative evidence about plausible mistakes to avoid. Experiments on coding, mathematical reasoning, scientific question answering, and tool-use benchmarks show that MOPD improves over standard on-policy baselines on most benchmarks, and also improves over baselines that reuse the same rollout group through verified-success SFT or preference optimization. Under matched demonstration budgets, random peers, peers from unrelated prompts, and success-only contexts all fall short of the mixed context, indicating that the gain comes from the same-prompt success–failure contrast rather than from a longer teacher context. Teacher-signal analysis shows the same ordering: mixed contexts align teacher scores with verifier rewards better than any single-type context. These results indicate that effective on-policy distillation should exploit the student's multi-rollout trial-and-error behavior rather than treating rollouts as isolated samples. Code is available at https://anonymous.4open.science/r/mopd_code-C7BC/.