acceptodds
Under review as a conference paper at ICLR 2027

Self-Supervised On-Policy Distillation from Contrasting Reasoning Trajectories

Abstract

On-policy RLVR trains reasoning models from groups of sampled attempts but assigns token updates using terminal rewards. A mixed group contains a richer signal: a correct attempt shows how the current policy can solve a prompt, while an incorrect attempt exposes prefixes it visits before failing. The two trajectories usually diverge, so the successful tokens cannot directly supervise the failed prefixes. We introduce Self-Supervised On-Policy Distillation (SSOPD), which uses a successful completion as privileged hindsight for a teacher at prefixes of a failed completion, then distills the teacher's local distribution into a student that sees only the prefix. This converts within-group outcome contrast into dense on-policy supervision without external solution traces. We connect the method to the policy's success-conditioned action distribution and show how the frontier weight tracks available correct–wrong pairs. Across AIME 2024, AIME 2025, and HMMT 2025, SSOPD improves over GRPO in all nine model-benchmark settings. On Qwen3-8B, it reaches a macro Avg@12 of 65.6, exceeding GRPO by 1.6 points and the solution-conditioned OPSD baseline by 0.8 points.

Primary area:
foundation or frontier models, including LLMs
Keywords:
on policy distillation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_8VEbOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

SSOPD augments GRPO with an auxiliary distillation loss that activates only when a sampled group contains both correct and incorrect completions. For such a mixed group, the method selects the shortest correct completion as a witness and the longest incorrect completion as the failure. At each of the failure's first K prefixes, a stop-gradient teacher—conditioned on the witness as privileged context—forms a next-action distribution, and the student trains to match it via a pointwise-clipped forward-KL surrogate. The frontier weight scales the auxiliary loss by the number of correct–wrong pairs in the group, concentrating updates where the policy is most uncertain. Proposition 3.1 establishes the success-conditioned policy q⁺ as the exact process target implicit in binary verifier feedback. Theorem 3.2 gives a local improvement bound separating the teacher's witness-approximation error from the student's distillation error. Experiments on Qwen3-1.7B, 4B, and 8B on AIME 2024/2025 and HMMT 2025 show consistent improvements over GRPO at all scales, with SSOPD at 8B matching or exceeding SFT and solution-conditioned OPSD while using only on-policy samples and answer verification.

Critical strengths
  • The theoretical motivation is unusually careful for this subfield. Proposition 3.1 identifies q⁺ as a well-defined target that is implicit in binary outcome feedback but never directly accessible, which explains concretely why GRPO leaves process-level signal on the table. This is a cleaner characterization of the problem than prior work typically provides.

  • The frontier weight design is principled and its properties are well-analyzed. Proposition 3.3 connects p_x(1−p_x) to branching variance, making the weight's behavior interpretable: it naturally rises as the policy begins solving a prompt and falls again once the prompt is mastered, without requiring a separate difficulty model.

  • The 8B result is the most compelling number in the paper. Exceeding solution-conditioned OPSD in macro Avg@12 (65.6 vs 64.8) under an information-controlled comparison—where SSOPD uses no reference solutions while OPSD does—is a concrete payoff for the self-supervised framing.

Critical weaknesses
  • Evaluation is restricted to mathematical reasoning on the Qwen3 family. Whether the mechanism generalizes to code, science problems, or other verifiable domains is not tested. The theoretical argument is not domain-specific, so the absence of cross-domain evidence is a genuine gap in the empirical claim.

  • Gains at 1.7B are small: +0.6 macro Avg@12 over GRPO, with individual benchmark numbers (+0.6, +0.3, +1.0 on AIME 2024/2025 and HMMT respectively). At this scale, run-to-run variance could plausibly explain the difference, and no confidence intervals or multiple-run statistics are reported.

  • The teacher context is capped at 2048 tokens with K=1024 prefixes. For very long reasoning traces—which are common on HMMT-level problems—many of the failure's early prefixes lie in a region where the teacher has limited window for both the witness and the problem context. The paper acknowledges this constraint motivates the length-based selectors but doesn't measure how often it binds or whether it degrades teacher quality on long-trajectory problems.

Questions for authors
  1. Have you measured what fraction of training steps involve mixed groups across the three scales? The frontier weight is zero for all-correct and all-wrong groups, so the effective number of updates from SSOPD depends directly on this fraction, and it likely varies substantially across scales and training stages.

  2. The teacher prompt is capped at 2048 tokens. How often does this cap truncate the witness in practice, particularly on HMMT problems that typically require longer reasoning chains? Would a longer teacher context (or a summarization of the witness) improve results at the harder end of the difficulty distribution?

  3. For the 1.7B experiments, did you run multiple seeds? The +0.6 macro Avg@12 gain over GRPO is small enough that variance estimates would help distinguish a systematic effect from noise.

Reviewer_ruo8OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes SSOPD, which uses successful responses from the same rollout group to provide token-level supervision at failed reasoning prefixes. It complements GRPO without external solution traces or a stronger teacher. The authors provide theoretical analysis and report consistent improvements over GRPO across three Qwen3 model sizes and three math benchmarks.

Critical strengths
  1. The method reuses successful and failed rollouts to provide dense supervision without external solution traces or a stronger teacher.
  2. The analysis clearly separates teacher approximation error from student fitting error and states the limitations of its local improvement guarantee.
  3. SSOPD improves over GRPO in all nine reported model–benchmark settings, showing a consistent positive trend across three model sizes.
Critical weaknesses
  1. The incremental contribution over SDPO’s successful-sample feedback approach is not established through controlled comparisons.
  2. Baseline provenance and training-budget comparability are unclear. Reporting the best HMMT checkpoint in the selector ablation also raises model-selection concerns.
  3. The ablations do not isolate the benefit of successful reasoning traces or distinguish adaptive weighting from changes in average auxiliary-loss strength.
  4. The theoretical analysis assumes exact KL, whereas the implemented clipped loss can favor deviations from the teacher. The practical impact remains unexamined.
Questions for authors
  1. What gains does SSOPD achieve over SDPO under matched training and evaluation settings?
  2. Were the Table 1 baselines rerun under matched budgets, and was HMMT used for hyperparameter or checkpoint selection?
  3. Do the gains persist with answer-only or no-reference controls, and with a fixed weight matched for average auxiliary-loss strength?
  4. How does performance change when the clipped loss is replaced with exact forward KL?
Reviewer_Wv3uOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper adds dense on-policy distillation to GRPO by reusing a mixed group of samples. A frozen copy of the initial model reads the shortest correct completion and provides next-token targets at early prefixes of the longest wrong completion. The student is trained there without that completion, and the auxiliary weight is zero unless the group contains both outcomes. Across three Qwen3 scales and three math benchmarks, this improves the reported GRPO scores, and at 8B it is close to solution-conditioned OPSD.

Critical strengths
  1. The information-matched comparison is set up correctly. GRPO and SSOPD both use only prompts, on-policy samples, and verified answers, while SFT and OPSD are explicitly separated as methods that also see reference solutions.

  2. The transfer target is not a copy of the successful tokens. The method states that the trajectories diverge and instead distills a hindsight-conditioned distribution at prefixes of the failed sample. The frontier weight and the shortest-correct, longest-wrong rule are specified and ablated.

Critical weaknesses
  1. The auxiliary loss does not create supervision beyond GRPO. It is zero for every all-correct or all-wrong group, which is exactly where GRPO's group advantage is also zero. On mixed groups, GRPO already upweights the correct sample and downweights the wrong one. SSOPD only adds a distillation term there. The macro gains over this GRPO baseline are +0.6, +0.3, and +1.6 on 1.7B, 4B, and 8B, with no seeds or intervals. On 1.7B, solution-conditioned OPSD remains ahead by 5.1 macro points (43.4 versus 38.3).

  2. The teacher does not implement the paper's success-conditioned target. The target in equation 4 is the current policy's next-token distribution among successful continuations of the failed prefix. The implemented teacher is the frozen initial checkpoint, with LoRA disabled, reading a different completed sample whose path may already have diverged. There is no ablation against the current LoRA policy as teacher, and none against applying the same clipped forward KL along the correct sample rather than at the failed prefixes.

  3. The component ablations do not support the design. Both sweeps use only Qwen3-4B on HMMT 2025. Dynamic and fixed weights differ by at most 0.8 points, and the twelve selector pairs span 42.5 to 45.0. Table 2 takes the best checkpoint, while Table 1 does not state its checkpoint rule. These results do not separate hindsight content, teacher staleness, or cross-trajectory placement from ordinary GRPO.

Questions for authors
  1. Are the Table 1 numbers from a predetermined step or from the best checkpoint? Please report the mean and interval over seeds for GRPO, SSOPD, and OPSD at that same step.

  2. What happens if the teacher is the current LoRA policy, and if the same clipped forward KL is applied to prefixes of the correct sample rather than to prefixes of the longest wrong sample?

  3. What fraction of training prompts have a mixed group, and do the gains remain on prompts whose group success rate stays near zero?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.