acceptodds
Under review as a conference paper at ICLR 2027

View-Decomposed On-Policy Distillation for Robust Reasoning in Large Language Models

Abstract

On-policy distillation reduces training–inference mismatch by supervising large language models (LLMs) on their own generated trajectories. For robust reasoning, this supervision should preserve useful teacher uncertainty while limiting the transfer of prompt-sensitive behavior. We investigate how consistency across task-preserving prompt variations can guide distillation toward more accurate reasoning with less sensitivity to prompt wording. We introduce View-Decomposed On-Policy Distillation (VD-OPD), which compares a frozen teacher's predictions for the same student-generated prefix under two prompt views. Their shared probability mass defines both the learning target and its weight, retaining alternatives supported by both views while attenuating disagreement. VD-OPD uses no answer labels during distillation and adds no inference cost. On six mathematics benchmarks, it raises Qwen3-1.7B-Base's macro-average Avg@8 from 34.8% to 36.7% over EOPD (full), with better unseen-prompt robustness. Gains remain 1.4/1.2 points under two-view/total-FLOPs controls. Cross-view agreement determines VD-OPD's target and weight. Our analysis explains when pooling improves the target and why shared teacher errors can persist.

Primary area:
foundation or frontier models, including LLMs
Keywords:
on-policy distillation, large language models, reasoning, prompt robustness, multi-view consistency

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_MiMpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates on-policy knowledge distillation for improving the reasoning capabilities of large language models and proposes VD-OPD. Instead of directly imitating a single teacher view on student-generated rollout trajectories, the method constructs a consensus supervision target based on the consistency of teacher predictions across multiple task-preserving views. It further employs state-dependent weighting to adjust the supervision strength at different tokens and states, aiming to mitigate unstable teacher signals and the teacher–student gap. The main contributions include a distillation objective centered on cross-view consensus and an adaptive weighting mechanism. The paper also analyzes the advantages and limitations of the normalized consensus target compared with a single-view teacher target. For experimental validation, the authors conduct matched-budget comparisons with GKD, sampled-token OPD, and adaptive supervision baselines under settings such as Qwen3-8B to Qwen3-1.7B-Base. The evaluation covers MATH500, AMC23, AIME24/25, Minerva, and subsets of OlympiadBench, examining improvements in mathematical reasoning accuracy and robustness to prompt wrappers.

Critical strengths

First, the paper reframes the problem of unstable teacher supervision in on-policy distillation as a more interpretable problem of cross-view consistency supervision. VD-OPD introduces a consensus target and state-dependent weighting mechanism, while also analyzing the conditions under which the normalized consensus target outperforms a single-view teacher target and explaining why shared errors may still persist. This gives the method a certain degree of principled justification rather than making it merely an empirical weighting strategy. Second, the experimental evidence is relatively convincing. Compared with EOPD(full), two-view EOPD, and FLOPs-matched baselines, VD-OPD improves the six-benchmark Avg@8 of Qwen3-1.7B-Base from 34.8% to 36.7%, while increasing the worst-view accuracy on unseen prompt wrappers from 28.9% to 30.2%. Furthermore, the method also maintains performance improvements on cross-domain benchmarks such as GPQA Diamond and MMLU-Pro.

Critical weaknesses

The core contribution of the proposed method is not sufficiently distinguished from existing work on adaptive supervision, prompt-level teacher gating, and EOPD. The main innovation appears to lie in using multi-view teacher consistency for target construction and selective weighting. However, the reported performance improvements are relatively modest. For example, on Qwen3-1.7B, the six-benchmark Avg@8 increases only from 34.8% to 36.7%, while the worst-view accuracy on unseen prompt wrappers improves from 28.9% to 30.2%. The gains on GPQA Diamond and MMLU-Pro are also relatively small. These results make the method appear more like an empirical enhancement than a fundamentally new paradigm with a sufficiently strong contribution to independently justify acceptance.

Questions for authors

The paper acknowledges that cross-view agreement does not guarantee target correctness, and Figure 6 shows a negative-utility tail in low-mass regions. Could the authors provide an analysis stratified by target correctness, such as the proportion of shared teacher errors, update utility, and error propagation under high and low values of Zt? How sensitive are the benefits of VD-OPD to the choice of prompt views? If random paraphrases, simple templates, or lower-quality task-preserving views are used, does VD-OPD still maintain its advantages over EOPD(full) and two-view EOPD?

2 comments
AuthorsOpenReview ↗

Response to Reviewer MiMp (1/2): W1: Contribution and controlled comparisons

Thank you for the time you devoted to our paper and for your constructive suggestions. Below, we explain the contribution and present the additional analyses of teacher correctness and view selection.

Accuracy is in %, and SD is across three training seeds. The 95% CIs use paired question bootstrap conditional on the evaluated training runs.

W1. What distinguishes VD-OPD from adaptive supervision, teacher gating, and EOPD?

Response.{\color{green}\mathbf{Response.}} VD-OPD asks which teacher preferences remain supported when the same task is expressed differently. We evaluate this at the prefixes the student actually generates. The probability mass supported by both teacher views becomes the supervision signal. A single unnormalized target determines what the student learns and how strongly each state contributes.

EOPD [1] uses single-view teacher entropy. ToDi [2] uses teacher–student probability ratios, and TGOPD [3] uses verifier-based reliability at the prompt level. Veto [4] forms a geometric target from teacher–student agreement. VD measures agreement between task-preserving views of the same teacher at each student prefix.

For two tokens:

Teacher viewsNormalized consensus rrConsensus mass ZZ
p1=p2=(0.5,0.5)p_1=p_2=(0.5,0.5)(0.5,0.5)(0.5,0.5)1
p1=(0.99,0.01), p2=(0.01,0.99)p_1=(0.99,0.01),\ p_2=(0.01,0.99)(0.5,0.5)(0.5,0.5)≈0.199

The same normalized target can arise from shared uncertainty or contradictory confidence. Its mass distinguishes these cases and sets their supervision strength. Building on the geometric pooling formulation in logarithmic opinion pools [5], VD retains this mass and uses it to guide learning along student-generated trajectories.

The benefit holds under controls for compute, teacher observations, and baseline tuning:

ComparisonEOPD Macro Avg@8 ± SDVD Macro Avg@8 ± SDVD gain, 95% CI
Equal FLOPs: full vs. VD-Budget34.82 ± 0.6236.00 ± 0.62+1.2 [+0.2, +2.2]
Equal views/updates: 2V vs. VD35.30 ± 0.6336.70 ± 0.62+1.4 [+0.4, +2.4]
Additional dev tuning: full vs. VD35.2 ± 0.536.70 ± 0.62+1.5 (point estimate)
Additional dev tuning: 2V vs. VD35.5 ± 0.536.70 ± 0.62+1.2 (point estimate)

VD also improves on the strongest student-augmentation baseline under structural changes and paraphrases:

MethodFLOPs (101810^{18})Structural Mean/WorstParaphrase Mean/Worst
EOPD(2V)+StudentAug5.931.9 / 28.331.1 / 27.3
VD-OPD5.832.6 / 29.131.8 / 28.2

Table 3 also reports transfer to GPQA Diamond and MMLU-Pro.

[1] Entropy-Aware On-Policy Distillation of Language Models.

[2] ToDi: Token-wise Distillation via Fine-Grained Divergence Control.

[3] Verify Before You Distill: Prompt-Level Teacher Gating for On-Policy Distillation.

[4] Stable On-Policy Distillation through Adaptive Target Reformulation.

[5] Selecting Weighting Factors in Logarithmic Opinion Pools.

AuthorsOpenReview ↗

Response to Reviewer MiMp (2/2): Q1: Correctness and view selection

Q1(a). How do target correctness, shared errors, and update utility vary with ZZ?

Response.{\color{green}\mathbf{Response.}} We evaluate teacher correctness through final answers and track how updates affect the student's answer distribution. Figure 6 provides a complementary view of how well an update fits the consensus target, measured by the change in cross-entropy.

At the middle checkpoint, 250 measurement questions provide four prefixes each. The low/high strata contain 243/251 states, each with four paired continuations. Proportions are approximate pair-level micro-averages. “Both correct” includes different valid reasoning paths. “Same wrong” requires equivalent incorrect final answers.

StratumMean ZZBoth correctOne correctSame wrongDifferent wrongSame top-1 token
Q10.46≈12%≈29%≈19%≈40%30%
Q40.97≈47%≈15%≈33%≈5%93%

For update utility and error propagation, branches share the checkpoint, optimizer state, student rollouts, teacher distributions, and number of effective update positions. The 250 update questions and 250 measurement questions are source-disjoint. The geometric variants share the same normalized target. These local results concern one middle checkpoint.

Stratum / updateOracle KL decrease, 95% CIShared-wrong probability change, 95% CICorrectness change (pp), 95% CI
Low / EOPD(full)−0.0021 [−0.0061, +0.0019]+0.009 [−0.003, +0.021]−0.28 [−1.38, +0.82]
Low / Geometric-1−0.0008 [−0.0048, +0.0032]+0.006 [−0.006, +0.018]−0.12 [−1.22, +0.98]
Low / Batch-mean-ZZ+0.0001 [−0.0039, +0.0041]+0.004 [−0.008, +0.016]+0.04 [−1.06, +1.14]
Low / Shuffled-ZZ−0.0004 [−0.0044, +0.0036]+0.005 [−0.007, +0.017]−0.03 [−1.13, +1.07]
Low / VD+0.0026 [−0.0014, +0.0066]−0.003 [−0.015, +0.009]+0.39 [−0.71, +1.49]
High / EOPD(full)+0.0060 [+0.0020, +0.0100]+0.008 [−0.004, +0.020]+0.65 [−0.45, +1.75]
High / Geometric-1+0.0071 [+0.0031, +0.0111]+0.009 [−0.003, +0.021]+0.73 [−0.37, +1.83]
High / Batch-mean-ZZ+0.0074 [+0.0034, +0.0114]+0.009 [−0.003, +0.021]+0.77 [−0.33, +1.87]
High / Shuffled-ZZ+0.0073 [+0.0033, +0.0113]+0.009 [−0.003, +0.021]+0.75 [−0.35, +1.85]
High / VD+0.0079 [+0.0039, +0.0119]+0.010 [−0.002, +0.022]+0.84 [−0.26, +1.94]

Oracle KL is measured in nats against a fixed four-candidate answer target (0.99,0.01/3,0.01/3,0.01/3)(0.99,0.01/3,0.01/3,0.01/3). Wrong-answer probabilities use the same 67/119 eligible low/high states across methods. Positive changes mean reinforcement.

Among the five methods, VD has the largest low-ZZ point estimates for oracle KL improvement and continuation accuracy. The corresponding CIs span zero. The correctness categories also identify both cases you asked about: successful continuations under low agreement and shared wrong answers under high agreement. The latter can be reinforced by VD, consistent with the shared-bias analysis in Appendices D.9–D.10.


Q1(b). How sensitive are the gains to random paraphrases, simple templates, and lower-quality valid views?

Response.{\color{green}\mathbf{Response.}} Your proposed view-pool comparison changes the teacher views while keeping EOPD(full) on the original prompt. Two-view EOPD and VD each use 5.8×10185.8\times10^{18} training FLOPs. External paraphrase generation and human validation costs are outside that training budget.

Training view poolMean ZZEOPD(full)EOPD(2V) ± SDVD ± SDVD−2V, 95% CIWorstView, 2V / VD
Simple templates0.74034.835.3 ± 0.6336.7 ± 0.62+1.4 [+0.4, +2.4]29.3 / 30.2
Random valid paraphrases0.67534.835.0 ± 0.4636.1 ± 0.49+1.1 [+0.1, +2.1]29.5 / 30.5
Lower-quality, task-preserving0.55834.834.5 ± 0.5134.8 ± 0.57+0.3 [−0.8, +1.4]27.5 / 27.9

With valid random paraphrases, VD improves over two-view EOPD by 1.1 points, with a positive 95% CI. Under weaker views, VD matches EOPD(full) and gains 0.3 points over two-view EOPD, with a CI spanning zero. Simple templates and verified paraphrases provide the most useful supervision among the tested pools.

Reviewer_CgotOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes VD-OPD, a method designed to address the issue where teacher supervision might inadvertently transfer sensitivity to specific prompts. Given the same student-generated prefix, the method compares the teacher's predictions across two views of the prompt—views that remain invariant regarding the task—and simultaneously constructs distillation targets and supervision weights based on geometric consensus; this approach preserves mutually supported continuations while attenuating divergent signals. The authors analyze the conditions and limitations under which this consensus objective improves predictions. Through experiments involving multiple mathematical reasoning benchmarks, model families, and computational cost controls, they demonstrate the method's gains in reasoning accuracy and robustness to unseen prompts.

Critical strengths
  1. This paper offers a valuable, complementary perspective on distillation methods that rely solely on entropy. High entropy in a teacher model does not necessarily imply unreliable supervision, while low entropy does not necessarily indicate stable predictions.

  2. The method is concise; geometric consensus simultaneously determines both what predictions a student should learn and the strength of supervision it should receive, eliminating the need for additional, separate filters or complex gating mechanisms.

Critical weaknesses
  1. The theoretical analysis focuses on the target distribution error given a fixed prefix; it does not directly guarantee the final reasoning accuracy of the student model, nor does it provide a theoretical explanation for the training benefits of consistency weighting.

  2. The prompt format is overly simplistic. Both training and testing primarily involve prepending a brief problem-solving instruction or title to the original question, resulting in a test format that is structurally very similar to the training format.

  3. Cross-view consistency is not a sufficient proxy for the quality of supervision. Since both views originate from the same teacher, they may consistently repeat the same errors; conversely, different—yet equally valid—reasoning paths can lead to divergent predictions. The paper should provide a more detailed explanation of cross-view consistency.

  4. The experimental performance gains are modest. Compared to shuffling weights or averaging weights across batches, the full method yields improvements of approximately 0.5 and 0.3 percentage points, respectively.

Questions for authors

See Weakness

3 comments
AuthorsOpenReview ↗

Response to Reviewer Cgot (1/2): W1–W2: Theory and unseen transformations

Thank you for your time and thoughtful comments on the theory and experimental design. Below, we clarify the role of consistency weighting and present the additional evaluations you requested.

Accuracy is in %, and SD is across three training seeds. Reported 95% CIs use paired question bootstrap conditional on the evaluated training runs.

W1. Does the theory explain final student accuracy and the benefit of consistency weighting?

Response.{\color{green}\mathbf{Response.}} Theorem 1 gives an exact improvement criterion under a shared ideal target p⋆p^\star at a fixed prefix:

=\mathcal C-\mathcal G,$$ Here $p_0$ is the original-view teacher and $\delta$ is the centered cross-view logit shift. The curvature cost is $\mathcal C=\mathbb{E}D_{\mathrm{KL}}(p_0\Vert r)$, and the mean correction is $\mathcal G=\tfrac12\langle p^\star-p_0,\mathbb{E}\delta\rangle$. Pooling improves the target exactly when $\mathcal G>\mathcal C$. Eq. (6) characterizes $c=\sqrt{p_1p_2}$ as an unnormalized generalized-KL barycenter. Its mass $Z$ gives the exact logit gradient $Z(q-r)$ and bound $\Vert \nabla_z\ell\Vert _1\leq2Z$ (Eq. (8), Appendix D.3). This explains how disagreement reduces supervision strength. The complete training experiments test the resulting effect on reasoning accuracy: | Budget control | EOPD Macro Avg@8 ± SD | VD Macro Avg@8 ± SD | VD gain, 95% CI | | --- | ---: | ---: | ---: | | Equal total FLOPs: full vs. VD-Budget | 34.82 ± 0.62 | 36.00 ± 0.62 | +1.2 [+0.2, +2.2] | | Equal two-view/update budget: 2V vs. VD | 35.30 ± 0.63 | 36.70 ± 0.62 | +1.4 [+0.4, +2.4] | For a direct local task diagnostic, matched branches use source-disjoint update and measurement questions: | Stratum | Update | Four-candidate oracle KL decrease, 95% CI | Correctness change (pp), 95% CI | | --- | --- | ---: | ---: | | Low $Z$ | EOPD(full) | −0.0021 [−0.0061, +0.0019] | −0.28 [−1.38, +0.82] | | Low $Z$ | VD-OPD | +0.0026 [−0.0014, +0.0066] | +0.39 [−0.71, +1.49] | | High $Z$ | EOPD(full) | +0.0060 [+0.0020, +0.0100] | +0.65 [−0.45, +1.75] | | High $Z$ | VD-OPD | +0.0079 [+0.0039, +0.0119] | +0.84 [−0.26, +1.94] | The oracle assigns 0.99 to the correct candidate and $0.01/3$ to each alternative. This measures the student's distribution over four answer candidates. The local estimates favor VD, with low-$Z$ CIs spanning zero. The full training comparisons provide stronger evidence: VD improves reasoning accuracy under both equal-compute and equal-view/update budgets, with positive 95% CIs. --- > #### **W2. Are the test transformations too similar to the training wrappers?** ${\color{green}\mathbf{Response.}}$ We evaluated two additional families of held-out transformations. Structural changes include instruction relocation and layout/delimiter changes. Verified paraphrases preserve the question's conditions, variables, answer, and roles. Each family has four views fixed before evaluation. Training uses the original simple wrappers. | Held-out family | Method | MeanView | WorstView | WorstTemplate | Answer flips | Parse failures | | --- | --- | ---: | ---: | ---: | ---: | ---: | | Structural | EOPD(full) | 30.5 | 26.5 | 27.8 | 33% | 1.3% | | Structural | EOPD(2V) | 31.0 | 27.0 | 28.0 | 32% | 1.2% | | Structural | EOPD(full)+StudentAug | 31.3 | 27.5 | 28.3 | 28% | 1.2% | | Structural | EOPD(2V)+StudentAug | 31.9 | 28.3 | 29.1 | 26% | 1.1% | | Structural | VD-OPD | 32.6 | 29.1 | 30.0 | 25% | 1.1% | | Paraphrase | EOPD(full) | 29.7 | 25.5 | 26.5 | 36% | 1.6% | | Paraphrase | EOPD(2V) | 30.2 | 26.2 | 27.0 | 34% | 1.5% | | Paraphrase | EOPD(full)+StudentAug | 30.4 | 26.5 | 27.4 | 32% | 1.3% | | Paraphrase | EOPD(2V)+StudentAug | 31.1 | 27.3 | 28.0 | 29% | 1.3% | | Paraphrase | VD-OPD | 31.8 | 28.2 | 29.0 | 28% | 1.2% | WorstView minimizes within each question. WorstTemplate minimizes template-level means. VD achieves the highest mean, worst-view, and worst-template accuracy in both families, including comparisons with student-side augmentation.
AuthorsOpenReview ↗

Response to Reviewer Cgot (2/2): W3–W4: Agreement and ablations

W3. Can agreement distinguish shared errors from different valid reasoning paths?

Response.{\color{green}\mathbf{Response.}} Cross-view affinity measures distributional compatibility at the same prefix. The construction preserves shared uncertainty: when both views have the same high-entropy distribution pp, we have c=pc=p and Z=1Z=1. When the views confidently prefer different continuations, ZZ decreases, even if both continuations eventually succeed.

At the middle checkpoint, the low/high groups contain 243/251 states, with four paired continuations per state. Percentages are approximate pair-level micro-averages.

ZZ stratumBoth correctExactly one correctSame wrong answerDifferent wrong answers
Low (Q1, mean 0.46)≈12%≈29%≈19%≈40%
High (Q4, mean 0.97)≈47%≈15%≈33%≈5%

Shared errors are also tracked after matched local updates. The same 67 low-ZZ and 119 high-ZZ states with an eligible shared-wrong candidate are used for both methods:

StratumEOPD: wrong-candidate probability change, 95% CIVD: wrong-candidate probability change, 95% CI
Low ZZ+0.009 [−0.003, +0.021]−0.003 [−0.015, +0.009]
High ZZ+0.008 [−0.004, +0.020]+0.010 [−0.002, +0.022]

The low-ZZ point estimate favors weaker error reinforcement under VD. High-ZZ shared errors can persist, as characterized in Appendices D.9–D.10. The CIs span zero for both strata. ZZ controls supervision strength, while answer outcomes assess correctness.


W4. Are the 0.5- and 0.3-point gains over weight controls substantial enough?

Response.{\color{green}\mathbf{Response.}} The 0.5- and 0.3-point comparisons isolate how weights are assigned while keeping VD's geometric target. The full factorial comparison also measures the contribution of target shape:

Target / weightMacro Avg@8 ± SDVD gain, 95% CI
Arithmetic / 135.0 ± 0.47+1.7 [+0.9, +2.5]
Arithmetic / ZZ35.7 ± 0.47+1.0 [+0.2, +1.8]
Geometric / 136.1 ± 0.47+0.6 [−0.2, +1.4]
Geometric / shuffled ZZ36.2 ± 0.47+0.5 [−0.1, +1.1]
Geometric / batch-mean ZZ36.4 ± 0.47+0.3 [−0.3, +0.9]
VD-OPD36.7 ± 0.62—

Changing target shape alone yields a 1.1-point increase. State-specific weighting provides a further estimated gain of 0.6 points. The CIs for this increment and the shuffle/batch-mean comparisons include zero. Results against dev-tuned baselines are:

MethodMacro Avg@8 ± SDAdditional coefficient-search configurations
Retuned EOPD(full)35.2 ± 0.59
Retuned EOPD(2V)35.5 ± 0.59
VD-OPD36.7 ± 0.620

VD retains gains of 1.5 and 1.2 points over the retuned baselines. The ablations attribute much of the improvement to the consensus target, with a smaller estimated contribution from assigning ZZ to each state.

Thank you for the careful and detailed response. I have thoroughly read the authors’ replies and the other reviewers’ comments. However, the additional results reinforce my concern in W4: they do not clearly establish a benefit of state-specific consistency weighting over simpler weighting schemes, as the reported confidence intervals for comparisons with unit, shuffled, and batch-mean weights all include zero. Its independent contribution remains uncertain. Therefore, I have decided to maintain my original score.

Reviewer_KNu8OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

In this paper, the authors studied On-policy distillation (OPD), which trains a student on its own rollouts, using the teacher's token distributions as targets. The paper argues that teacher entropy does not reflect whether the teacher's distribution is stable, especially under the case when the prompt is reworded. Hence, a recent work EOPD (Jin et al., 2026) that adds forward KL at high-entropy teacher states does not fully resolve the teacher’s distribution instability issue. They first ran a pilot study to show that high entropy doesn't imply stable views. Then they proposed VD-OPD, where at each student prefix, VD-OPD forms the unnormalized geometric mean and trained with a loss built based on this mean. They then compare the performance of VD-OPD with EOPD math reasoning benchmarks for two different model families.

Critical strengths

The paper points out an interesting problem with EOPD that teacher entropy may not reflect whether the teacher's distribution is stable regrading prompt rephrasing. The authors are honest about the limitation and weaknesses of VD-OPD, for example, the costs of VD-OPD.

Critical weaknesses

I list the main concerns here and minor issues in the Questions for Author box.

  1. The main novelty of this paper comes from utilizing two views to evaluate the teacher’s distribution stability. However, I think the analysis of this problem is too weak. The authors did not provide enough evidence to show that variations in the teacher's output under rewording (two views studied in the paper) are harmful for OPD. If the key tokens are stable and only lower-probability tokens or phrasing preferences shift, the variation is legitimate. They never show low-Z states have worse teacher predictions, and thereby the OPD performance will regress. The wrappers used in this paper, like "Problem:" "Solve the following problem.", could legitimately shift the teacher toward restating the problem, or towards answering directly, which is not wrong at all.
  2. There seems to be a disconnection between Theorem 1 and VD-OPD, only part of the design is supported by the analysis in Theorem 1. It is not clear how Z plays the roll theoretically. There are only empirical results regarding weight Z, see Table 4. Considering the variances, it is hard to tell if using weight Z is the best choice.
  3. In practice, why two views are enough to measure the stability of teacher’s output? The authors did not discuss why only two samples are enough in practice to perform this evaluation.
  4. Main results in Table 1 2 and 4, do not have variances, therefore, with the small improvements VD-OPD achieved, it is hard to tell if the proposed method indeed achieve better performance over EOPD for those benchmarks on new models, especially small benchmarks like AIME24 and AIME25.
  5. In Table 3, it seems like the gaps between mean view and worst view are very close for VD-OPD and EOPD. Similarly, in Table 10 from the Appendix, VD-OPD loses 3.05 points on average from canonical to unseen wrappers (36.7 -> 33.7), on every benchmark, but there is no result for other baselines, so it is unclear if VD-OPD helps in stabilize student’s outputs. Moreover, a naïve baseline, that is, student-side prompt augmentation is never compared against.
Questions for authors
  1. How does the Qwen3-4B student reach 76.1 on MATH500 against a teacher at 72.8 (Table 7)? Are teacher and student decoded and prompted identically?
  2. Can you add a baseline in which the student is also trained on randomly wrapped prompts?
  3. Which baseline hyperparameters beyond the learning rate were tuned? Do your EOPD numbers reproduce those in the EOPD paper?
  4. Why there are many baseline cells blank (Table 6, 7 and 8), and will complete tables be provided?
7 comments
AuthorsOpenReview ↗

Response to Reviewer KNu8 (1/5): W1–W2: Supervision and theory

Thank you for taking the time to review our paper carefully and for your detailed, valuable feedback. We address your questions below with additional experiments on supervision quality, statistical reliability, and prompt robustness.

Unless stated otherwise, results use Qwen3-1.7B. Accuracy is in %, and gains are in percentage points. SD is across three training seeds. Paired question-bootstrap 95% confidence intervals (CIs) keep all eight samples and paired seeds together for each question. They measure question-sampling uncertainty conditional on the evaluated training runs.

W1. Does disagreement reflect harmful supervision, or merely valid phrasing and low-probability variation?

Response.{\color{green}\mathbf{Response.}} You raise an important distinction: task-preserving views can support different valid solutions. VD-OPD uses their agreement to set supervision strength. At the middle checkpoint, 250 held-out questions provide four prefixes each. Each prefix has four paired teacher continuations.

Teacher diagnosticLow ZZ (Q1)High ZZ (Q4)
States / continuation pairs243 / 972251 / 1,004
Mean ZZ0.460.97
Same top-1 next token30%93%
Disagreement outside the union of top-10 tokens26%69%
Both continuations correct≈12%≈47%
Same incorrect final answer≈19%≈33%

Continuation proportions are approximate pair-level micro-averages. The tail statistic measures the share of total Hellinger disagreement outside the top-10 union. Low-ZZ states show substantial disagreement at dominant next-token predictions. Their double-correct cases also confirm that some disagreement reflects valid reasoning diversity. These are descriptive associations across the sampled states.

To measure task utility, all update branches start from the same checkpoint and optimizer state and use matched update positions. Evaluation uses questions disjoint from the update set. The low-ZZ results are:

UpdateCorrectness change, 95% CIChange in shared-wrong-answer probability, 95% CI
EOPD(full)−0.28 [−1.38, +0.82]+0.009 [−0.003, +0.021]
Geometric / 1−0.12 [−1.22, +0.98]+0.006 [−0.006, +0.018]
VD-OPD+0.39 [−0.71, +1.49]−0.003 [−0.015, +0.009]

The local estimates favor VD-OPD, with CIs spanning zero. The complete training comparison tests the objective at fixed teacher access and update budget:

Two-view methodFLOPs (101810^{18})Macro Avg@8 ± SDVD gain, 95% CI
EOPD(2V)5.835.30 ± 0.63+1.4 [+0.4, +2.4]
VD-OPD5.836.70 ± 0.62—

At the training level, VD gains 1.4 points under the same two-view budget, with a positive 95% CI.


W2. What theoretically connects ZZ to the objective, and is its weighting optimal?

Response.{\color{green}\mathbf{Response.}} Theorem 1 analyzes the normalized target. The mass ZZ follows directly from the generalized-KL barycenter in Eq. (6) and Appendix D.2:

\qquad c(a)=\sqrt{p_1(a)p_2(a)},\qquad Z=\sum_a c(a),$$ where $I(v\Vert p)=\sum_a[v(a)\log(v(a)/p(a))-v(a)+p(a)]$. Consequently, the single loss in Eq. (7) yields both target $r=c/Z$ and weight $Z$. For student logits $z$, Eq. (8) gives $$\ell=-c^\top\log q, \qquad \nabla_z\ell=Z(q-r), \qquad \Vert \nabla_z\ell\Vert _1\leq 2Z.$$ The loss is averaged over the unweighted number of valid tokens, preserving this attenuation. The bound applies to the per-state logit gradient. | Weight with the same geometric target | Macro Avg@8 ± SD | VD gain, 95% CI | | --- | ---: | ---: | | 1 | 36.1 ± 0.47 | +0.6 [−0.2, +1.4] | | Shuffled $Z$ | 36.2 ± 0.47 | +0.5 [−0.1, +1.1] | | Batch-mean $Z$ | 36.4 ± 0.47 | +0.3 [−0.3, +0.9] | | State-specific $Z$ | 36.7 ± 0.62 | — | All three weight comparisons favor state-specific $Z$ in their point estimates, with CIs that include zero.
AuthorsOpenReview ↗

Response to Reviewer KNu8 (2/5): W3–W4: Views and statistical reliability

W3. Why are two views sufficient?

Response.{\color{green}\mathbf{Response.}} Two views provide a low-cost stochastic update for the pairwise objective. Each response uses the original prompt and one alternative sampled uniformly from three wrappers. At fixed student prefixes, this estimates the average update over the three possible pairs. The four-view geometric mean defines a separate joint objective, which we compare directly:

Views / budgetMacro Avg@8 ± SDWorstViewFLOPs (101810^{18})GPU-hGain over K=2K=2, 95% CI
K=2K=236.7 ± 0.6230.25.837.2—
K=4K=4, same updates37.1 ± 0.6530.68.051.3+0.4 [−0.3, +1.1]
K=4K=4, same FLOPs36.3 ± 0.7030.05.837.5−0.4 [−1.2, +0.4]

The four-view target is c4(a)=(∏m=14pm(a))1/4c_4(a)=(\prod_{m=1}^4p_m(a))^{1/4}. With the same updates, it gains 0.4 points at approximately 38% more FLOPs. Under equal FLOPs, two views give the higher score. Both difference CIs span zero. This cost–accuracy comparison supports two views as an efficient default.

On the same fixed states, the three possible pairs also produce similar gradient directions:

Original prompt paired withCosine to the three-pair mean gradient
Instruction 10.91
Instruction 20.93
Layout wrapper0.90

W4. Are the improvements reliable across seeds, models, and small benchmarks?

Response.{\color{green}\mathbf{Response.}} For AIME, we use the 30 questions as bootstrap sampling units and keep each question's eight continuations together. The three-seed results corresponding to Table 1 and Tables 6–8 are:

StudentBenchmarkEOPD(full), mean ± SDVD-OPD, mean ± SDVD gain, 95% CI
Qwen3-1.7BMATH50066.0 ± 0.4067.8 ± 0.40+1.8 [+0.6, +3.0]
Qwen3-1.7BAMC2343.6 ± 0.6545.9 ± 0.63+2.3 [−1.2, +5.8]
Qwen3-1.7BAIME2414.4 ± 0.8715.8 ± 0.83+1.4 [−3.8, +6.6]
Qwen3-1.7BAIME2510.4 ± 0.8311.7 ± 0.83+1.3 [−3.9, +6.5]
Qwen3-1.7BMinerva34.5 ± 0.5136.9 ± 0.51+2.4 [+1.2, +3.6]
Qwen3-1.7BOlympiadBench40.0 ± 0.5142.1 ± 0.51+2.1 [+0.9, +3.3]
Qwen3-4BMATH50074.3 ± 0.4076.1 ± 0.40+1.8 [+0.6, +3.0]
Qwen3-4BAMC2350.8 ± 0.6553.0 ± 0.65+2.2 [−1.3, +5.7]
Qwen3-4BAIME2418.7 ± 0.8320.1 ± 0.87+1.4 [−3.8, +6.6]
Qwen3-4BAIME2514.6 ± 0.8316.0 ± 0.87+1.4 [−3.8, +6.6]
Qwen3-4BMinerva41.9 ± 0.5144.0 ± 0.51+2.1 [+0.9, +3.3]
Qwen3-4BOlympiadBench45.8 ± 0.5148.0 ± 0.51+2.2 [+1.0, +3.4]
Llama-3.2-3BMATH50046.3 ± 0.4047.9 ± 0.40+1.6 [+0.4, +2.8]
Llama-3.2-3BAMC2330.0 ± 0.6232.1 ± 0.65+2.1 [−1.4, +5.6]
Llama-3.2-3BAIME249.0 ± 0.8710.4 ± 0.83+1.4 [−3.8, +6.6]
Llama-3.2-3BAIME256.5 ± 0.877.8 ± 0.87+1.3 [−3.9, +6.5]
Llama-3.2-3BMinerva25.8 ± 0.5127.8 ± 0.51+2.0 [+0.8, +3.2]
Llama-3.2-3BOlympiadBench30.2 ± 0.5133.5 ± 0.51+3.3 [+2.1, +4.5]

For the compute controls in Table 2:

ComparisonFLOPs (101810^{18}), baseline / VDMacro Avg@8 ± SD, baseline / VDVD gain, 95% CI
EOPD(full) vs. VD-Budget4.7 / 4.734.82 ± 0.62 / 36.00 ± 0.62+1.2 [+0.2, +2.2]
EOPD(2V) vs. VD5.8 / 5.835.30 ± 0.63 / 36.70 ± 0.62+1.4 [+0.4, +2.4]
EOPD(full) vs. VD4.7 / 5.834.82 ± 0.62 / 36.70 ± 0.62+1.9 [+0.9, +2.9]

For Table 4:

Target / weightMacro Avg@8 ± SDVD gain, 95% CI
Arithmetic / 135.0 ± 0.47+1.7 [+0.9, +2.5]
Arithmetic / ZZ35.7 ± 0.47+1.0 [+0.2, +1.8]
Geometric / 136.1 ± 0.47+0.6 [−0.2, +1.4]
Geometric / shuffled ZZ36.2 ± 0.47+0.5 [−0.1, +1.1]
Geometric / batch-mean ZZ36.4 ± 0.47+0.3 [−0.3, +0.9]
VD-OPD36.7 ± 0.62—

VD improves the mean score in all 18 model–benchmark comparisons. Gains on MATH500, Minerva, and OlympiadBench have positive 95% CIs for all three students. The smaller AMC/AIME benchmarks and the weight-only comparisons have CIs spanning zero. Both matched-budget comparisons also have positive CIs.

AuthorsOpenReview ↗

Response to Reviewer KNu8 (3/5): W5, Q1–Q2: Robustness and evaluation

W5. Do the unseen-view results demonstrate stability, and how do other baselines compare?

Response.{\color{green}\mathbf{Response.}} Your observation about the canonical-to-unseen drop is correct. VD achieves higher unseen-view accuracy and also has a larger drop from its canonical score. WorstView takes the minimum across views for each question before averaging. WorstTemplate takes the minimum of template-level mean accuracies.

MethodCanonicalMeanViewWorstViewWorstTemplateCanonical−MeanMean−WorstGreedy answer flips
EOPD(full)34.832.828.930.12.03.927%
EOPD(2V)35.333.029.330.52.33.726%
EOPD(full)+StudentAug35.033.029.630.42.03.424%
EOPD(2V)+StudentAug35.733.530.030.92.23.522%
VD-OPD36.733.730.231.13.03.523%

The VD drop is 3.05 points when calculated from the per-benchmark entries in Table 10, versus 3.0 from the rounded macro means. Per-benchmark MeanView / WorstView results are:

BenchmarkEOPD(full)EOPD(2V)Full+StudentAug2V+StudentAugVD-OPD
MATH50063.6 / 59.463.7 / 60.063.7 / 60.363.9 / 60.464.0 / 60.4
AMC2341.4 / 37.041.5 / 37.841.5 / 38.141.6 / 38.141.7 / 38.5
AIME2412.9 / 10.613.1 / 9.413.1 / 9.713.7 / 10.213.9 / 11.5
AIME259.1 / 6.39.4 / 5.79.4 / 6.010.0 / 6.510.2 / 7.6
Minerva32.6 / 27.532.7 / 29.032.7 / 29.333.0 / 29.533.1 / 29.1
OlympiadBench37.2 / 32.637.6 / 33.937.6 / 34.238.7 / 35.239.0 / 34.1

StudentAug is competitive on sensitivity and some individual worst-view scores. VD achieves the highest macro MeanView, WorstView, and WorstTemplate scores in this comparison.


Q1. How can the Qwen3-4B student outperform its teacher, and were they evaluated identically?

Response.{\color{green}\mathbf{Response.}} Yes. Teacher and students use the same 500 MATH500 questions and the same non-thinking serialization, including the empty thinking block and role/assistant prefixes. Decoding uses temperature 1.0, top-p 0.8, and an 8,192-token response limit. Stopping rules and the mathematical-equivalence grader are also shared. We use eight samples per question in each of three evaluations.

ModelMATH500 Avg@8Pass@8
Qwen3-8B teacher72.891.4
Qwen3-4B, EOPD(full)74.392.6
Qwen3-4B, VD-OPD76.193.4

These evaluations reproduce Table 7 under a common protocol. A distilled student can outperform its teacher, as also demonstrated by Born Again Neural Networks [1]. Here, EOPD exceeds the teacher by 1.5 points. VD contributes a further 1.8-point gain over the EOPD-trained student.

[1] Born Again Neural Networks.


Q2. What happens when the student trains on randomly wrapped prompts?

Response.{\color{green}\mathbf{Response.}} We evaluated your proposed baseline with StudentAug. It samples original and wrapped training prompts with equal probability for both student rollout and loss evaluation. The one-view baseline uses one teacher-scoring view. The two-view baseline scores both teacher views. All held-out views are fixed before model evaluation.

MethodFLOPs (101810^{18})CanonicalPrefix Mean/WorstStructural Mean/WorstParaphrase Mean/Worst
EOPD(full)4.734.832.8 / 28.930.5 / 26.529.7 / 25.5
EOPD(2V)5.835.333.0 / 29.331.0 / 27.030.2 / 26.2
EOPD(full)+StudentAug4.835.033.0 / 29.631.3 / 27.530.4 / 26.5
EOPD(2V)+StudentAug5.935.733.5 / 30.031.9 / 28.331.1 / 27.3
VD-OPD5.836.733.7 / 30.232.6 / 29.131.8 / 28.2

Augmentation improves the baseline scores. VD achieves a further 0.7-point gain in MeanView on both structural changes and paraphrases, using slightly fewer training FLOPs than two-view StudentAug. Its WorstView scores are also higher in all three families.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.