- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes U-OPSD, an on-policy self-distillation approach that uses no external answers or feedback during post-training. It constructs pseudo-solutions through majority voting over the model’s own responses and uses them to guide distillation on disagreeing responses. Experiments on five mathematical reasoning benchmarks show improvements across several Qwen3 configurations, with larger gains in non-thinking mode and performance comparable to supervised OPSD in thinking mode.
Critical strengths
-
The paper addresses a meaningful question: whether language models can improve through on-policy self-distillation without externally provided answers or feedback during post-training.
-
The approach is conceptually simple, using agreement among self-generated responses to construct a teacher reference and disagreement to identify learning opportunities.
-
The results in Tables 1–3 show improvements over the starting models across several Qwen3 configurations and five mathematical reasoning benchmarks, with particularly encouraging gains in non-thinking mode.
-
Section 5 candidly discusses the method’s limitations, including its dependence on base-model competence, incorrect consensus answers, and the narrower gains in thinking mode.
Critical weaknesses
1.Agreement among multiple responses does not necessarily imply correctness, particularly when they come from the same model. Although the paper acknowledges this limitation, the pseudo-label quality analysis in Section 4.3 is based on only 64 training prompts. A broader analysis would strengthen confidence that the approach improves reasoning rather than reinforcing existing errors. 2.The gains over supervised OPSD are substantial in non-thinking mode but much smaller in thinking mode. Since independent training runs are not reported, it remains unclear how stable the smaller differences are. Reporting variability across runs would help distinguish consistent improvements from comparable performance. 3. The motivation is clearly presented, but the central contribution could be distinguished more explicitly from prior self-training and self-distillation work. A concise explanation of the new conceptual insight would make the paper’s significance easier to assess. 4. The title “Without Any Supervision” may invite a broader interpretation than intended. Qualifying this wording to emphasize the absence of external supervision during post-training would better align the title with the paper’s scope. 5. The presentation assumes considerable familiarity with on-policy distillation and related terminology. A simple illustrative example explaining how the model learns from its own agreement and disagreement would make the main idea more accessible to a broader audience. 6. The conclusion could better distinguish the demonstrated findings from the broader prospect of autonomous self-improvement. A more measured discussion of the practical implications and remaining challenges would strengthen the paper’s overall message.
Questions for authors
See Weaknesses. As I am not an expert in this specific area, I will carefully consider the authors’ rebuttal and the other reviewers’ assessments during the discussion phase before deciding whether to raise my score.
3 comments
>>> W3
The central contribution should be distinguished more explicitly from prior self-training and self-distillation work, with a concise statement of the new conceptual insight.
>>> A3 We thank the reviewer and will state the conceptual contribution explicitly. Prior on-policy distillation constructs its teacher from privileged information external to the model — gold solutions or human annotation. Our central insight is that this privileged context can be constructed from the model itself: a consensus answer distilled from its own samples suffices to condition an effective teacher.
To show that U-OPSD is not simply self-training or self-distillation with pseudo labels, we added SFT controls using the same prompts, sampling budget (G = 8), and consensus threshold (τ = 0.5), retaining 4,274 of 4,800 prompts. We trained on consensus-selected trajectories with hyperparameters matched to our method, considering both the longest and shortest agreeing trajectories. Together with the TTRL comparison in Table 1, these controls examine three uses of the same consensus signal. The results below report Qwen3-4B accuracy averaged over five benchmarks.
| Method | Role of consensus | Trajectory selection | Avg. (%) |
|---|---|---|---|
| Base model | — | — | 40.96 |
| TTRL | Reward | — | 41.77 |
| SFT | Training target | Longest agreeing trajectory | 41.31 |
| SFT | Training target | Shortest agreeing trajectory | 41.38 |
| U-OPSD (ours) | Teacher conditioning context | Longest agreeing trajectory | 49.49 |
Using consensus as a reward or to select SFT targets improves over the base model by at most 0.81%. Using it to construct the teacher’s conditioning context yields an 8.53% improvement. These results show that directly optimizing consensus rewards or imitating consensus-selected responses is insufficient to reproduce the gains of U-OPSD. They support using a consensus-selected response as privileged context to guide the teacher’s token-level predictions along the student’s own rollouts. This interpretation is also consistent with the label-only ablation in Figure 4, where conditioning on the consensus answer alone yields performance below the base model.
Our conceptual contribution is therefore to show that the informational advantage used in on-policy distillation can be constructed from the model’s own rollout consensus, reducing reliance on external privileged supervision. We will add these controls and the accompanying discussion to the revised paper.
>>> W4
The title "Without Any Supervision" invites a broader reading than intended; qualify it to mean no external supervision during post-training.
>>> A4 Thanks for the suggestion. The intended claim is the absence of external supervision during post-training. We will revise the title to “… Without External Supervision” and align the abstract and introduction accordingly.
>>> W5
The presentation assumes familiarity with on-policy distillation. A simple illustrative example of learning from agreement and disagreement would help.
>>> A5 Thanks for the advice. We will add an illustrative walkthrough early in the paper: a single prompt with G=8 sampled rollouts, contrasting the case where samples agree — the consensus answer conditions the teacher, which then supervises the student at the token level — with the case where no consensus forms and the prompt is filtered out. Key terms (on-policy distillation, teacher conditioning, self-consistency threshold) are defined on first use.
>>> W6
The conclusion should separate demonstrated findings from the broader prospect of autonomous self-improvement and discuss practical implications and remaining challenges more carefully.
>>> A6 Thank you for this suggestion. We will distinguish our demonstrated findings from the broader prospect of autonomous self-improvement. Our experiments show that label-free post-training improves mathematical reasoning in 4B/8B models over their base models and reward-based alternatives in settings with verifiable answers and well-defined consensus. These findings suggest a practical way to reduce reliance on external supervision for such tasks. We will revise the conclusion to clarify this scope and discuss the challenges that remain, including residual pseudo-label errors (Section 4.3 and Limitations) and whether the improvements can be sustained over repeated self-distillation iterations. Establishing reliable autonomous self-improvement will require further investigation of these issues.
Thank you for the response. Most of my concerns have been addressed, and I have raised my score to 3.
We thank the reviewer for the constructive review and helpful suggestions. We also thank the reviewer for recognizing our work as conceptually simple and meaningful.
>>> W1Agreement among responses from the same model does not imply correctness. The pseudo-label analysis in Sec. 4.3 uses only 64 training prompts; a broader analysis is needed to show the method improves reasoning rather than reinforcing errors.>>> A1We thank the reviewer for this important comment and agree with its premise: agreement among samples from the same model does not imply correctness. Our method, however, does not require that implication. It requires only that consensus filtering yield labels cleaner than the policy’s own samples — a weaker and directly measurable condition. On 4,615 training prompts with known gold answers, under the full training configuration (G=8, 4,096-token budget, τ=0.5), individual rollouts of the base model are correct in 73.0% of cases (26,944/36,920), whereas the derived pseudo-labels are correct in 86.6% (3,593/4,148); this 13.6% margin constitutes the training signal. The consensus threshold also rejects the appropriate prompts: where no consensus forms, singleton answers are correct in only 9.2% of cases (168/1,834).To address the concern about sample size, we recomputed the Section 4.3 measurement over the full training set (~72× larger) using per-step vote statistics logged during training. For the run reported in Table 1, pseudo-label accuracy over all 4,321 labelled prompts is 88.5% (3,825/4,321), consistent with the 86.7% measured on 64 prompts. Unanimous votes (8/8) are correct in 95.0% of cases (2,790/2,937) versus 74.8% (1,035/1,384) for non-unanimous majorities, indicating that vote margin itself carries signal and that most training supervision derives from the high-margin stratum. We will replace the 64-prompt analysis in Section 4.3 with these full-scale statistics.
Finally, we examined directly what training does to prompts the base model answers incorrectly. We re-sampled G=8 rollouts from the base model and the trained checkpoint on identical training prompts, selecting prompts by the base model’s first four rollouts and measuring on its remaining four to avoid regression-to-the-mean artifacts. On the 693 prompts where the base majority vote is wrong, the frequency with which rollouts repeat that wrong answer falls from 47.0% (1,304/2,772) to 37.2% (1,031/2,772) after training, while rollout accuracy rises from 18.9% (525/2,772) to 32.1% (891/2,772); unparsable-output rates change negligibly (8.4% (232/2,772) → 10.1% (279/2,772)), ruling out abstention as the explanation. Accuracy on prompts the base model already solves is preserved (87.4% (12,605/14,424) → 91.1% (13,143/14,424)), and the largest improvement occurs on the hardest stratum — prompts with no base consensus — where label accuracy rises from 32.6% (103/316) to 53.8% (170/316). Label coverage increases from 89.9% (4,148/4,615) to 96.2% (4,441/4,615) at stable label accuracy (87.3%, 3,875/4,441). We will include these analyses as an appendix with complete tables.
>>> W2Gains over supervised OPSD are large in non-thinking mode and small in thinking mode. No independent training runs are reported, so the stability of the small differences is unclear.>>> A2Thank you for raising this concern. We repeated the thinking-mode experiments three times with different training seeds for both OPSD and U-OPSD at the 4B and 8B scales. The results below report the mean ± standard deviation across the three runs.U-OPSD achieves a mean accuracy 0.74% higher than OPSD at 4B and 0.06% lower at 8B. These results support our claim of comparable performance in thinking mode. Note that we do not claim a consistent advantage over supervised OPSD in this setting. The contribution is that U-OPSD achieves this performance without any external labels or gold solutions, using supervision constructed from its own rollouts. We will add these seed statistics and clarify this interpretation in the revised paper.