acceptodds
Under review as a conference paper at ICLR 2027

On-Policy Self-Distillation without Any Supervision

Abstract

Visual on-policy distillation relies heavily on an informative teacher-student asymmetry, through either a larger, stronger teacher or privileged supervision, such as reference answers or ground-truth regions of interest. This raises a fundamental question: where can informative asymmetry come from when nothing privileged is available? We answer this by inverting where the asymmetry comes from. Rather than adding privileged information to the teacher, we subtract information from the student. This asymmetry creates the same effective learning signal for free as a teacher with access to information unavailable to the student, without ground-truth annotations, rewards, or a separate stronger teacher model. Building on this principle, we introduce Self-Supervised Visual On-Policy Distillation (S2VOPD), a simple yet effective method that constructs on-policy learning signals from asymmetric augmented views. S2VOPD distills the teacher's distribution conditioned on the original image on-policy into the student distribution conditioned on a strongly augmented view of the same image. We systematically explore a broad design space of visual augmentations and uncover that (1) asymmetry matters: all four augmentation families improve performance, while symmetric self-distillation degrades it; (2) strength matters: performance peaks at a moderate strength; and (3) the gap must remain task-consistent: augmentations that completely remove the question-relevant evidence can induce large but uninformative discrepancies. Across six fine-grained perception benchmarks, S2VOPD improves Qwen3.5-4B from 70.7% to 77.4%, above all open-source models compared, up to Qwen3-VL at 235B, and surpasses GPT-5.4. While holding training data the same, it recovers 96% of the improvement achieved by methods with privileged information.

Primary area:
foundation or frontier models, including LLMs
Keywords:
on-policy distillation, large language models, on-policy self-distillation, self-supervised learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_a3NmOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes U-OPSD, an on-policy self-distillation approach that uses no external answers or feedback during post-training. It constructs pseudo-solutions through majority voting over the model’s own responses and uses them to guide distillation on disagreeing responses. Experiments on five mathematical reasoning benchmarks show improvements across several Qwen3 configurations, with larger gains in non-thinking mode and performance comparable to supervised OPSD in thinking mode.

Critical strengths
  1. The paper addresses a meaningful question: whether language models can improve through on-policy self-distillation without externally provided answers or feedback during post-training.

  2. The approach is conceptually simple, using agreement among self-generated responses to construct a teacher reference and disagreement to identify learning opportunities.

  3. The results in Tables 1–3 show improvements over the starting models across several Qwen3 configurations and five mathematical reasoning benchmarks, with particularly encouraging gains in non-thinking mode.

  4. Section 5 candidly discusses the method’s limitations, including its dependence on base-model competence, incorrect consensus answers, and the narrower gains in thinking mode.

Critical weaknesses

1.Agreement among multiple responses does not necessarily imply correctness, particularly when they come from the same model. Although the paper acknowledges this limitation, the pseudo-label quality analysis in Section 4.3 is based on only 64 training prompts. A broader analysis would strengthen confidence that the approach improves reasoning rather than reinforcing existing errors. 2.The gains over supervised OPSD are substantial in non-thinking mode but much smaller in thinking mode. Since independent training runs are not reported, it remains unclear how stable the smaller differences are. Reporting variability across runs would help distinguish consistent improvements from comparable performance. 3. The motivation is clearly presented, but the central contribution could be distinguished more explicitly from prior self-training and self-distillation work. A concise explanation of the new conceptual insight would make the paper’s significance easier to assess. 4. The title “Without Any Supervision” may invite a broader interpretation than intended. Qualifying this wording to emphasize the absence of external supervision during post-training would better align the title with the paper’s scope. 5. The presentation assumes considerable familiarity with on-policy distillation and related terminology. A simple illustrative example explaining how the model learns from its own agreement and disagreement would make the main idea more accessible to a broader audience. 6. The conclusion could better distinguish the demonstrated findings from the broader prospect of autonomous self-improvement. A more measured discussion of the practical implications and remaining challenges would strengthen the paper’s overall message.

Questions for authors

See Weaknesses. As I am not an expert in this specific area, I will carefully consider the authors’ rebuttal and the other reviewers’ assessments during the discussion phase before deciding whether to raise my score.

3 comments

We thank the reviewer for the constructive review and helpful suggestions. We also thank the reviewer for recognizing our work as conceptually simple and meaningful.

>>> W1 Agreement among responses from the same model does not imply correctness. The pseudo-label analysis in Sec. 4.3 uses only 64 training prompts; a broader analysis is needed to show the method improves reasoning rather than reinforcing errors.

>>> A1 We thank the reviewer for this important comment and agree with its premise: agreement among samples from the same model does not imply correctness. Our method, however, does not require that implication. It requires only that consensus filtering yield labels cleaner than the policy’s own samples — a weaker and directly measurable condition. On 4,615 training prompts with known gold answers, under the full training configuration (G=8, 4,096-token budget, τ=0.5), individual rollouts of the base model are correct in 73.0% of cases (26,944/36,920), whereas the derived pseudo-labels are correct in 86.6% (3,593/4,148); this 13.6% margin constitutes the training signal. The consensus threshold also rejects the appropriate prompts: where no consensus forms, singleton answers are correct in only 9.2% of cases (168/1,834).

To address the concern about sample size, we recomputed the Section 4.3 measurement over the full training set (~72× larger) using per-step vote statistics logged during training. For the run reported in Table 1, pseudo-label accuracy over all 4,321 labelled prompts is 88.5% (3,825/4,321), consistent with the 86.7% measured on 64 prompts. Unanimous votes (8/8) are correct in 95.0% of cases (2,790/2,937) versus 74.8% (1,035/1,384) for non-unanimous majorities, indicating that vote margin itself carries signal and that most training supervision derives from the high-margin stratum. We will replace the 64-prompt analysis in Section 4.3 with these full-scale statistics.

Finally, we examined directly what training does to prompts the base model answers incorrectly. We re-sampled G=8 rollouts from the base model and the trained checkpoint on identical training prompts, selecting prompts by the base model’s first four rollouts and measuring on its remaining four to avoid regression-to-the-mean artifacts. On the 693 prompts where the base majority vote is wrong, the frequency with which rollouts repeat that wrong answer falls from 47.0% (1,304/2,772) to 37.2% (1,031/2,772) after training, while rollout accuracy rises from 18.9% (525/2,772) to 32.1% (891/2,772); unparsable-output rates change negligibly (8.4% (232/2,772) → 10.1% (279/2,772)), ruling out abstention as the explanation. Accuracy on prompts the base model already solves is preserved (87.4% (12,605/14,424) → 91.1% (13,143/14,424)), and the largest improvement occurs on the hardest stratum — prompts with no base consensus — where label accuracy rises from 32.6% (103/316) to 53.8% (170/316). Label coverage increases from 89.9% (4,148/4,615) to 96.2% (4,441/4,615) at stable label accuracy (87.3%, 3,875/4,441). We will include these analyses as an appendix with complete tables.

>>> W2 Gains over supervised OPSD are large in non-thinking mode and small in thinking mode. No independent training runs are reported, so the stability of the small differences is unclear.

>>> A2 Thank you for raising this concern. We repeated the thinking-mode experiments three times with different training seeds for both OPSD and U-OPSD at the 4B and 8B scales. The results below report the mean ± standard deviation across the three runs.

SettingMethodAvg. ± std (%)
4B thinkingOPSD76.11 ± 0.49
4B thinkingU-OPSD76.85 ± 0.76
8B thinkingOPSD77.53 ± 0.36
8B thinkingU-OPSD77.47 ± 0.51

U-OPSD achieves a mean accuracy 0.74% higher than OPSD at 4B and 0.06% lower at 8B. These results support our claim of comparable performance in thinking mode. Note that we do not claim a consistent advantage over supervised OPSD in this setting. The contribution is that U-OPSD achieves this performance without any external labels or gold solutions, using supervision constructed from its own rollouts. We will add these seed statistics and clarify this interpretation in the revised paper.

>>> W3 The central contribution should be distinguished more explicitly from prior self-training and self-distillation work, with a concise statement of the new conceptual insight.

>>> A3 We thank the reviewer and will state the conceptual contribution explicitly. Prior on-policy distillation constructs its teacher from privileged information external to the model — gold solutions or human annotation. Our central insight is that this privileged context can be constructed from the model itself: a consensus answer distilled from its own samples suffices to condition an effective teacher.

To show that U-OPSD is not simply self-training or self-distillation with pseudo labels, we added SFT controls using the same prompts, sampling budget (G = 8), and consensus threshold (τ = 0.5), retaining 4,274 of 4,800 prompts. We trained on consensus-selected trajectories with hyperparameters matched to our method, considering both the longest and shortest agreeing trajectories. Together with the TTRL comparison in Table 1, these controls examine three uses of the same consensus signal. The results below report Qwen3-4B accuracy averaged over five benchmarks.

MethodRole of consensusTrajectory selectionAvg. (%)
Base model——40.96
TTRLReward—41.77
SFTTraining targetLongest agreeing trajectory41.31
SFTTraining targetShortest agreeing trajectory41.38
U-OPSD (ours)Teacher conditioning contextLongest agreeing trajectory49.49

Using consensus as a reward or to select SFT targets improves over the base model by at most 0.81%. Using it to construct the teacher’s conditioning context yields an 8.53% improvement. These results show that directly optimizing consensus rewards or imitating consensus-selected responses is insufficient to reproduce the gains of U-OPSD. They support using a consensus-selected response as privileged context to guide the teacher’s token-level predictions along the student’s own rollouts. This interpretation is also consistent with the label-only ablation in Figure 4, where conditioning on the consensus answer alone yields performance below the base model.

Our conceptual contribution is therefore to show that the informational advantage used in on-policy distillation can be constructed from the model’s own rollout consensus, reducing reliance on external privileged supervision. We will add these controls and the accompanying discussion to the revised paper.

>>> W4 The title "Without Any Supervision" invites a broader reading than intended; qualify it to mean no external supervision during post-training.

>>> A4 Thanks for the suggestion. The intended claim is the absence of external supervision during post-training. We will revise the title to “… Without External Supervision” and align the abstract and introduction accordingly.

>>> W5 The presentation assumes familiarity with on-policy distillation. A simple illustrative example of learning from agreement and disagreement would help.

>>> A5 Thanks for the advice. We will add an illustrative walkthrough early in the paper: a single prompt with G=8 sampled rollouts, contrasting the case where samples agree — the consensus answer conditions the teacher, which then supervises the student at the token level — with the case where no consensus forms and the prompt is filtered out. Key terms (on-policy distillation, teacher conditioning, self-consistency threshold) are defined on first use.

>>> W6 The conclusion should separate demonstrated findings from the broader prospect of autonomous self-improvement and discuss practical implications and remaining challenges more carefully.

>>> A6 Thank you for this suggestion. We will distinguish our demonstrated findings from the broader prospect of autonomous self-improvement. Our experiments show that label-free post-training improves mathematical reasoning in 4B/8B models over their base models and reward-based alternatives in settings with verifiable answers and well-defined consensus. These findings suggest a practical way to reduce reliance on external supervision for such tasks. We will revise the conclusion to clarify this scope and discuss the challenges that remain, including residual pseudo-label errors (Section 4.3 and Limitations) and whether the improvements can be sustained over repeated self-distillation iterations. Establishing reliable autonomous self-improvement will require further investigation of these issues.

Thank you for the response. Most of my concerns have been addressed, and I have raised my score to 3.

Reviewer_L5PSOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1(−1)
Read review
Paper summary

This paper proposes U-OPSD, an unsupervised variant of on-policy self-distillation that constructs privileged teacher context from a model's own rollouts. For each problem, it selects a vote-based pseudo-answer, uses an agreeing reasoning trace as the teacher reference, and distills the reference-conditioned next-token distribution along disagreeing student trajectories.

Critical strengths
  1. Replacing gold solutions with model-generated consensus removes the need for labeled reasoning traces during this post-training stage. Using the resulting trace as context for token-level distillation is a meaningful distinction from treating agreement only as a scalar reward.

  2. Tables 1–3 include multiple model sizes and generation modes, as well as supervised and label-free baselines. the non-thinking improvements are substantial, and the authors report rerunning OPSD rather than relying solely on published numbers.

Critical weaknesses
  1. The methodological novelty is limited, and the contribution is not sufficiently isolated. The main construction combines established self-consistency voting with the solution-conditioned distillation objective of OPSD (Zhao et al., 2026). The contribution here is therefore the combination of pseudo-reference construction and distillation on disagreement-selected prefixes. The comparisons against TTRL, RENT, and Intuitor are useful, but do not isolate this combination from simpler alternatives: teacher-forced self-training on the same selected pseudo-traces, or OPSD using the same pseudo-reference while supervising all valid rollouts. Such controls are needed to establish which component produces the gains.

  2. Consensus correctness and the resulting on-policy feedback loop are insufficiently validated. Even genuinely independent samples from the same policy can concentrate on a systematically wrong answer: vote frequency estimates the model's answer distribution, not correctness. Because subsequent rollouts and pseudo-labels depend on the updated student, this creates a plausible error-amplification loop even with a frozen teacher. Conversely, confidently unanimous wrong answers receive no direct update because there is no disagreement. The paper needs longitudinal pseudo-label accuracy and correct-minority retention measurements to determine their practical importance.

  3. Limited evidence for complex response structures and long reasoning trajectories. Although the five benchmarks contain challenging mathematics problems, they all admit extractable final answers. Extending exact-match voting to proofs, programs with multiple valid implementations, or open-ended responses would require additional mechanisms for determining semantic equivalence and correctness. Even within mathematics, agreement on the final answer does not establish the validity of intermediate reasoning.

The 4,096-token training cap also interacts with the answer-based filtering mechanism. Rollouts truncated before producing a parsable final answer are excluded from both teacher-reference selection and distillation, but remain in the denominator of the vote-confidence score. Consequently, incomplete generations can cause an entire prompt to fall below the confidence threshold and receive no training update. Problems requiring longer reasoning may therefore receive less supervision, although evaluation permits up to approximately 38k tokens (§4.1). the paper should report truncation rates and the proportion of prompts receiving updates across problem difficulty levels and generation budgets.

Furthermore, the thinking-mode experiments generate student training rollouts in non-thinking mode, leaving the effectiveness of distillation on long thinking-mode trajectories untested. The teacher also receives a complete pseudo-solution that is unavailable to the student. When evaluating a different or partially incorrect student reasoning path, this solution-conditioned teacher may assign lower probabilities to verification or backtracking steps that the student needs to recover from mistakes. Distillation could then discourage useful self-correction.

  1. The longest agreeing rollout remains an insufficiently justified teacher-selection heuristic. Figure 4 does compare longest, random, and shortest references, however, it reports one setting without uncertainty estimates. The degradation from using only the answer supports providing a reasoning trace, but does not establish that the longest trace is the most correct or informative. Longer agreeing traces may contain redundant steps or a correct final answer reached through invalid reasoning. Selecting them also increases teacher-context cost. Length-controlled comparisons and checks of intermediate-step quality are needed to distinguish useful reasoning content from verbosity and additional computation.

  2. The formal specification and exposition need consolidation.

Questions for authors
  1. How do pseudo-label accuracy, confidently wrong consensus, and retention of correct minority answers evolve during training? Could the 64-prompt probe be expanded and stratified by vote confidence and problem difficulty? To preserve the unsupervised training setting, gold answers should not influence teacher-trajectory selection, training objective, as these uses would introduce external supervision.

  2. Under the same sampled data and token budget, how does U-OPSD compare with teacher-forced training on the selected pseudo-traces and with pseudo-reference OPSD applied to all valid rollouts? These controls would clarify the value of disagreement-specific on-policy distillation.

  3. Does the longest-reference advantage persist across training seeds, model configurations, and length-controlled comparisons? Can you report intermediate-step correctness and teacher-computation cost, rather than only final benchmark accuracy?

  4. What fraction of prompts and rollouts is discarded by difficulty and response length? Can you test thinking-mode student rollouts and report performance across generation budgets?

  5. Could you clarify the exact training objective and teacher policy used in the implementation?

9 comments

>>> W1 & Q2 The methodological novelty is limited, and the contribution is not sufficiently isolated. The main construction combines established self-consistency voting with the solution-conditioned distillation objective of OPSD (Zhao et al., 2026). The contribution here is therefore the combination of pseudo-reference construction and distillation on disagreement-selected prefixes. The comparisons against TTRL, RENT, and Intuitor are useful, but do not isolate this combination from simpler alternatives: teacher-forced self-training on the same selected pseudo-traces, or OPSD using the same pseudo-reference while supervising all valid rollouts. Such controls are needed to establish which component produces the gains.

>>> A1 We thank the reviewer for suggestion. We ran both in the Table 1 setting (Qwen3-4B; same prompts, G = 8, τ = 0.5 and optimisation recipe; average over the five benchmarks).

Use of the same consensusAvg.
Base model40.96
Sparse reward (TTRL, Table 1)41.77
Control (i): teacher-forced training on consensus-selected traces (longest / shortest)41.31 / 41.38
Control (ii): pseudo-reference teacher, all disagreeing rollouts supervised47.58
U-OPSD: pseudo-reference teacher, one selected rollout (Table 1)49.49

Control (i): for each prompt whose consensus passes τ (4,274 of 4,800), we take one of the model's own rollouts that reaches the consensus answer (the longest or the shortest) and train the model on it with the standard SFT finetuning. Control (ii) retains the pseudo-reference teacher but distils every rollout that disagrees with the consensus, rather than a single selected one; rollouts that already agree with the pseudo-reference are not additionally distilled. We report its best checkpoint (step 50); its final-step score is 45.48.

These controls suggest two observations. First, reusing the pseudo-labels as training targets appears insufficient to explain the improvement: control (i) improves on the base model by at most 0.4%, whereas conditioning an on-policy teacher on the same consensus improves on it by 6.6% (control (ii)) and 8.5% (U-OPSD). We therefore believe that the main contribution lies in the mechanism, in which a self-constructed consensus takes the place of the external privileged reference of OPSD. Second, control (ii) recovers most of this improvement, which indicates that selective distillation accounts for a smaller part of it. Its main benefit is efficiency: control (ii) distils 33% more rollouts over training (6,399 versus 4,800; 2.05 versus 1.00 per prompt on prompts with disagreement) and does not fit in 80 GB of GPU memory under the U-OPSD configuration unless the loss is computed in chunks, yet it does not improve accuracy (47.58 versus 49.49). We will add both controls and this table to the paper and clarify the respective roles of the two components.

Together, these results indicate that U-OPSD is not a straightforward combination of its components. The same consensus yields 41.31 as an SFT target, 41.77 as a sparse reward and 49.49 as privileged context for an on-policy teacher; the improvement therefore depends on how the consensus is used, not on the consensus itself. Moreover, OPSD presupposes a verified reference, and whether an unverified, self-generated consensus could replace it was not evident a priori. The controls show that it can, and that distilling a single selected rollout makes this replacement both more accurate and more efficient. We will add both controls and this table to the paper and clarify the role of each component.

>>> W2 & Q1 Consensus correctness and the on-policy feedback loop are insufficiently validated. A wrong majority turns a correct minority rollout into a correction target, and later rollouts depend on the updated student, so errors may amplify; unanimous wrong answers get no update. Requested: pseudo-label accuracy, confidently wrong consensus, and correct-minority retention over training, on an expanded probe stratified by vote confidence and difficulty, with gold answers used only post hoc.

>>> A2 We agree that consensus among samples does not guarantee correctness. We therefore measured pseudo-label accuracy against gold answers on all the training samples for which one is available (Qwen3-4B, eight rollouts per prompt), before training and every 25 training steps. Gold answers were used only for this analysis.

Pseudo-labels are accurate, and accuracy tracks agreement. Before training, 89.9% of prompts receive a label, of which 86.6% are correct. Accuracy also rises with agreement: it is 66.0% when half to three quarters of the rollouts agree, 85.3% when at least three quarters agree, and 95.6% when all eight agree.

Correct minority answers are strengthened rather than suppressed. On prompts whose base majority is wrong but at least one rollout is correct, training (step 75) raises the fraction of correct rollouts from 23.5% to 46.5%. The correct answer becomes the majority on 54.2% of these prompts and disappears on 20.9%. Conversely, among prompts with a correct base majority and at least one wrong rollout, the majority turns wrong on only 6.5%.

Label accuracy does not degrade during training. The label accuracy of U-OPSD remains within 1.5% of the base at every checkpoint (table below), while the labelled fraction rises to 95.6%. At step 75, correct labels have become wrong on only 2.6% of prompts, whereas wrong labels have become correct on 30.6%.

StepU-OPSD: label accuracyU-OPSD: unanimous errors+ Unlikelihood term: label accuracy+ Unlikelihood term: unanimous errors
0 (base)86.6%6.4%86.9%4.4%
2586.7%5.6%87.4%5.1%
5087.1%6.2%90.2%4.9%
7587.3%6.2%90.9%4.5%
10085.6%6.9%90.8%4.0%
12585.1%6.6%90.5%4.2%
15085.3%6.9%90.3%3.6%

Unanimous errors: the fraction of prompts with unanimous rollouts whose answer is incorrect.

Unanimous errors. We show that unanimous errors, i.e., all rollouts agree but are wrong, are rare: before training they account for 6.4% of training prompts for both Qwen3-4B and Qwen3-8B. Under U-OPSD, their share of unanimous prompts remain within 4.4% and 6.9%.

>>> A2 (continued) Prompts without a gradient update. After submission, we also noticed that a fraction of training prompts receive no gradient update from U-OPSD: no answer reaches the consensus threshold τ, leaving no pseudo-reference. Formally, when c(x) < τ (9.7% of prompts for Qwen3-4B), no answer is reliable enough to serve as the pseudo-reference. While we cannot confidently single out the correct pseudo-trajectory, incorrect rollouts can be identified with confidence, and this provides a signal with which to update the model: a rollout whose answer no other rollout shares is incorrect in 89.9% of cases, compared with 57.8% for rollouts whose answer is shared by at least one other (Qwen3-8B: 87.7% versus 53.1%). We therefore lower the probability of such rollouts with a bounded unlikelihood term. Let $\mathcal{Y}^{\circ}_{x}=\lbrace y^{(g)}: a^{(g)}\neq\varnothing,\ a^{(g)}\neq a^{(h)}\ \forall h\neq g\rbrace $ denote the singleton rollouts and y∘y^{\circ} the longest of them (incorrect in 90.8% of cases). For each position t∈T(y∘)t\in\mathcal{T}(y^{\circ}), we attenuate all the sampled tokens except for format tokens such as \boxed{} and the end-of-sequence token, by αwt\alpha w_t nats:

\qquad \mathcal{L}_{\mathrm{UL}}(\theta) = \sum_{t\in\mathcal{T}(y^{\circ})} D_{\beta}\big(\mathrm{sg}[q_t] \big\Vert \pi_\theta(\cdot \mid x, y^{\circ}_{<t})\big),$$ where $w_t\in[\tfrac{1}{2},1]$ is the normalised token entropy, $\mathrm{sg}$ denotes stop-gradient, $D_\beta$ is the divergence used in U-OPSD, and $\alpha = 1$. Because $q_t$ departs from the current policy by at most $\alpha$ nats, each update is bounded. | Model | Base | U-OPSD | + Unlikelihood term | |---|---|---|---| | Qwen3-4B (non-thinking) | 40.96 | 49.49 | 70.23 | | Qwen3-8B (non-thinking) | 43.57 | 54.31 | 74.45 | Five-benchmark averages at the best checkpoint; Base and U-OPSD are taken from Table 1. Adding the bounded unlikelihood term increases significantly averaged performance from 49.49% to 70.23% on Qwen3-4B and from 54.31% to 74.45% on Qwen3-8B. These gains indicate that disagreement can provide effective negative supervision even when consensus is insufficient to construct a reliable pseudo-reference. The term also improves the pseudo-labels themselves: label accuracy rises to 90.9% and remains above 90% through step 150, the rate of unanimous errors continue to reduce from 4.4% to 3.6%, and mislabelled prompts decrease from 11.7% to 9.3%. These experiments were conducted after submission; we will add them to the appendix.
Reviewer_HBBwOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes Unsupervised On-Policy Self-Distillation, a post-training method that removes the need for gold solutions or external feedback during the OPSD framework. For each unlabeled problem, the model generates multiple solutions, selects a pseudo-answer by majority vote, and uses an agreeing reasoning trace as privileged context for a frozen self-teacher. The teacher’s token distribution is then distilled into the student along disagreeing trajectories. Across several Qwen3 variants and five mathematical-reasoning benchmarks, U-OPSD improves over the base models.

Critical strengths
  • The method does not require gold solutions, external verifiers, reward models, or stronger teacher models during training.

  • Replacing the ground-truth context in OPSD with a consensus-generated trace is conceptually clean and easy to implement.

Critical weaknesses
  • A major concern is that the experiments do not establish that U-OPSD expands reasoning capability rather than just sharpening the distribution. The method explicitly trains disagreement trajectories toward a consensus-conditioned teacher, but evaluation is limited to avg@4/12 and omits pass@k.

  • The conceptual contribution appears largely to combine majority-voting largely used as a test-time scaling method with the existing OPSD objective, which doesn't seem like a large contribution.

Questions for authors

See weaknesses above.

5 comments

>>> W1 The experiments do not show that U-OPSD expands reasoning capability rather than sharpening the output distribution. The method trains disagreeing trajectories toward a consensus-conditioned teacher, yet evaluation reports only avg@4/12 and omits pass@k.

>>> A1 Thank you for the suggestion. We evaluated pass@k for k = 1 to 128 on AIME24, AIME25, and HMMT25 (30 problems each) for Qwen3-4B and Qwen3-8B in non-thinking mode, drawing 128 samples per problem and using the standard unbiased pass@k estimator (https://imgur.com/WmeYhzV). u-OPSD is above its base model at every k on all three benchmarks and for both model sizes. For Qwen3-8B, pass@16 rises from 55.5% to 81.6% on AIME24, from 44.4% to 58.1% on AIME25, and from 26.8% to 42.4% on HMMT25. The gap remains at the largest budget: at k = 128, the number of problems solved at least once rises from 23 to 27 of 30 on AIME24 (76.7% → 90.0%), from 18 to 21 on AIME25 (60.0% → 70.0%), and from 12 to 19 on HMMT25 (40.0% → 63.3%). Qwen3-4B shows the same pattern (80.0% → 86.7%, 66.7% → 73.3%, 43.3% → 53.3%). The gain is therefore not limited to making already-reachable answers more likely: the trained models also solve problems that the base models do not solve in 128 attempts. We will add these results to the paper and state that our conclusions concern accuracy and coverage under the evaluated sampling budgets.

>>> W2 The conceptual contribution appears to be a combination of majority voting, widely used for test-time scaling, with the existing OPSD objective.

>>> A2 While we agree that self-consistency voting and the OPD objective are established techniques, existing OPD methods rely on costly privileged information, such as gold solutions or environmental feedback, or on a stronger teacher model. To our knowledge, we are the first to investigate whether such external supervision is necessary. Our novelty and contribution lie in conceptually showing that this informational advantage can be derived entirely from rollout consensus.

We also want to emphasize that this is not a trivial combination, which yields findings neither implies. First, the vote alone does not help: used as an RL reward under the same rollout budget and prompt set, TTRL stays at the base level in non-thinking mode (41.8 vs. base 41.0 on Qwen3-4B; 43.5 vs. base 43.6 on Qwen3-8B), whereas U-OPSD reaches 49.5 and 54.3 (Table 1). Conditioning the teacher only on the voted answer also falls below the base model on Qwen3-8B (41.9–43.4 vs. 43.6), while conditioning it on a full agreeing rollout reaches 59.0 in the same ablation (Fig. 4(a)). Second, the self-constructed context is not a weaker substitute for gold solutions: trained on the same prompts, U-OPSD is at or comparable OPSD in every configuration (non-thinking: 49.5 vs. 46.3 on 4B, 54.3 vs. 52.0 on 8B, Table 1; thinking: 77.05 vs. 76.20 and 77.99 vs. 77.97, Table 2; Instruct: 77.5 vs. 76.3 and 68.8 vs. 67.1, Table 3) without any external previlaged information. Third, consensus also decides where to supervise: distillation runs only along the rollouts that disagree with the vote, fewer than 10% of valid rollouts (Sec. 4.3). We will state these contributions explicitly in the introduction.

We have addressed all the questions and concerns in our responses above. As the discussion period closes soon, we would be grateful to hear whether these responses resolve your concerns or whether any point still needs clarification. If the new results change your assessment, we would sincerely appreciate it if you could reflect this in your final score. We appreciate your time and the constructive discussion.

Thanks for the detailed response. The response clarifies the mechanism behind the observed performance boost. However, methodological novelty still appears modest, especially given the close overlap with prior work (CANON; https://arxiv.org/abs/2607.13643) that uses majority voting as the privileged context for distillation. Thus, I maintain my original score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of rebuttal and revision of the paper

Dear Area Chair,

We thank the reviewers for their constructive feedback. Below we summarise our responses and revisions.

1. Novelty and prior work

Existing OPD methods rely on costly privileged information, such as gold solutions or environmental feedback, or on a stronger teacher model. Our contribution is to show that this informational advantage can be derived entirely from rollout consensus. U-OPSD is not a straightforward combination of components: the same consensus yields 41.31% as an SFT target, 41.77% as a sparse reward, and 49.49% as privileged context for an on-policy teacher. The improvement depends on how consensus is used, not consensus alone.

Reviewer HBBw questions novelty relative to CANON (https://arxiv.org/abs/2607.13643). We respectfully disagree that CANON limits our novelty. CANON is an unpublished arXiv preprint concurrently submitted to ICLR 2027, rather than established prior work. According to the ICLR 2027 Reviewer Guide, arXiv is not considered a peer-reviewed venue and authors are not required to compare to papers solely on arXiv and the lack of such comparisons cannot be a basis for rejection. We also identified substantial differences in methodology, implementation, experimental settings, and results. We therefore ask the Area Chair to assess U-OPSD on its own methodological and empirical merits, consistent with the ICLR 2027 reviewing guidelines.

2. Consensus correctness

Pseudo-labels are accurate, and accuracy tracks agreement. Before training, 89.9% of prompts receive pseudo-labels, 86.6% of which are correct. Accuracy increases with agreement: 66.0% at 50–75% agreement, 85.3% at ≥75%, and 95.6% with unanimous agreement. On prompts with an incorrect majority but a correct minority, the fraction of correct rollouts rises from 23.5% to 46.5% by step 75; only 2.6% of initially correct labels become incorrect.

We also noticed that a rollout whose answer no other rollout shares is incorrect in 89.9% of cases. We therefore lower the probability of such rollouts with a bounded unlikelihood term. Adding this term substantially improves average accuracy from 49.49% to 70.23% on Qwen3-4B and from 54.31% to 74.45% on Qwen3-8B. We have included this version in the uploaded paper revision.

3. Reasoning capability rather than sharpening

We evaluated pass@k for k = 1 to 128 on AIME24, AIME25, and HMMT25. U-OPSD is above its base model at every k on all three benchmarks and for both model sizes. The gain is therefore not limited to making already-reachable answers more likely: the trained models also solve problems that the base models do not solve in 128 attempts.

U-OPSD also recovers from erroneous prefixes more often at every truncation point. We truncated incorrect base-model rollouts from five benchmarks at 25%, 50%, and 75%, then sampled four continuations per prefix from each model. U-OPSD repeats the original incorrect answer less often (26.0% versus 31.6%).

4. Generality and stability

We further evaluate two open-ended domains, replacing exact-match with equivalence-based voting. On LiveCodeBench, U-OPSD outperforms both the base model and ground-truth-supervised OPSD (+4.5% over OPSD). On science, it improves in-domain accuracy by 5.9% over the base and nearly matches OPSD out of domain (−0.9%).

Across three thinking-mode seeds, U-OPSD averages +0.74% versus OPSD at 4B and −0.06% at 8B, supporting comparable performance. The longest reference beats the shortest in all four comparisons (+1.90–4.63%).

5. Changes to the paper

  • Add the pass@k results.
  • Add SFT and "distil everything that disagrees" controls and their table; clarify the role of each component.
  • Replace the 64-prompt analysis in Section 4.3 with the full-scale statistics.
  • Add the seed statistics and Tables R1–R3.
  • Correct Algorithm 1: c(x) is normalised by the number of rollouts with a parsable answer, rather than by G.
  • Revise the title to "… Without External Supervision".
  • Add an illustrative walkthrough early in the paper.
  • Revise the conclusion to clarify the scope and discuss the challenges that remain.
  • Add the full-training-set pseudo-label analyses to the appendix with complete tables.
  • Add the bounded unlikelihood term experiments to the appendix.

We thank all reviewers for their time and constructive discussion. With only a few hours left in the discussion period, we would appreciate the Area Chair encouraging further discussion of our responses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.