- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates on-policy knowledge distillation for improving the reasoning capabilities of large language models and proposes VD-OPD. Instead of directly imitating a single teacher view on student-generated rollout trajectories, the method constructs a consensus supervision target based on the consistency of teacher predictions across multiple task-preserving views. It further employs state-dependent weighting to adjust the supervision strength at different tokens and states, aiming to mitigate unstable teacher signals and the teacher–student gap. The main contributions include a distillation objective centered on cross-view consensus and an adaptive weighting mechanism. The paper also analyzes the advantages and limitations of the normalized consensus target compared with a single-view teacher target. For experimental validation, the authors conduct matched-budget comparisons with GKD, sampled-token OPD, and adaptive supervision baselines under settings such as Qwen3-8B to Qwen3-1.7B-Base. The evaluation covers MATH500, AMC23, AIME24/25, Minerva, and subsets of OlympiadBench, examining improvements in mathematical reasoning accuracy and robustness to prompt wrappers.
Critical strengths
First, the paper reframes the problem of unstable teacher supervision in on-policy distillation as a more interpretable problem of cross-view consistency supervision. VD-OPD introduces a consensus target and state-dependent weighting mechanism, while also analyzing the conditions under which the normalized consensus target outperforms a single-view teacher target and explaining why shared errors may still persist. This gives the method a certain degree of principled justification rather than making it merely an empirical weighting strategy. Second, the experimental evidence is relatively convincing. Compared with EOPD(full), two-view EOPD, and FLOPs-matched baselines, VD-OPD improves the six-benchmark Avg@8 of Qwen3-1.7B-Base from 34.8% to 36.7%, while increasing the worst-view accuracy on unseen prompt wrappers from 28.9% to 30.2%. Furthermore, the method also maintains performance improvements on cross-domain benchmarks such as GPQA Diamond and MMLU-Pro.
Critical weaknesses
The core contribution of the proposed method is not sufficiently distinguished from existing work on adaptive supervision, prompt-level teacher gating, and EOPD. The main innovation appears to lie in using multi-view teacher consistency for target construction and selective weighting. However, the reported performance improvements are relatively modest. For example, on Qwen3-1.7B, the six-benchmark Avg@8 increases only from 34.8% to 36.7%, while the worst-view accuracy on unseen prompt wrappers improves from 28.9% to 30.2%. The gains on GPQA Diamond and MMLU-Pro are also relatively small. These results make the method appear more like an empirical enhancement than a fundamentally new paradigm with a sufficiently strong contribution to independently justify acceptance.
Questions for authors
The paper acknowledges that cross-view agreement does not guarantee target correctness, and Figure 6 shows a negative-utility tail in low-mass regions. Could the authors provide an analysis stratified by target correctness, such as the proportion of shared teacher errors, update utility, and error propagation under high and low values of Zt? How sensitive are the benefits of VD-OPD to the choice of prompt views? If random paraphrases, simple templates, or lower-quality task-preserving views are used, does VD-OPD still maintain its advantages over EOPD(full) and two-view EOPD?
2 comments
Response to Reviewer MiMp (2/2): Q1: Correctness and view selection
Q1(a). How do target correctness, shared errors, and update utility vary with ?
We evaluate teacher correctness through final answers and track how updates affect the student's answer distribution. Figure 6 provides a complementary view of how well an update fits the consensus target, measured by the change in cross-entropy.
At the middle checkpoint, 250 measurement questions provide four prefixes each. The low/high strata contain 243/251 states, each with four paired continuations. Proportions are approximate pair-level micro-averages. “Both correct” includes different valid reasoning paths. “Same wrong” requires equivalent incorrect final answers.
| Stratum | Mean | Both correct | One correct | Same wrong | Different wrong | Same top-1 token |
|---|---|---|---|---|---|---|
| Q1 | 0.46 | ≈12% | ≈29% | ≈19% | ≈40% | 30% |
| Q4 | 0.97 | ≈47% | ≈15% | ≈33% | ≈5% | 93% |
For update utility and error propagation, branches share the checkpoint, optimizer state, student rollouts, teacher distributions, and number of effective update positions. The 250 update questions and 250 measurement questions are source-disjoint. The geometric variants share the same normalized target. These local results concern one middle checkpoint.
| Stratum / update | Oracle KL decrease, 95% CI | Shared-wrong probability change, 95% CI | Correctness change (pp), 95% CI |
|---|---|---|---|
| Low / EOPD(full) | −0.0021 [−0.0061, +0.0019] | +0.009 [−0.003, +0.021] | −0.28 [−1.38, +0.82] |
| Low / Geometric-1 | −0.0008 [−0.0048, +0.0032] | +0.006 [−0.006, +0.018] | −0.12 [−1.22, +0.98] |
| Low / Batch-mean- | +0.0001 [−0.0039, +0.0041] | +0.004 [−0.008, +0.016] | +0.04 [−1.06, +1.14] |
| Low / Shuffled- | −0.0004 [−0.0044, +0.0036] | +0.005 [−0.007, +0.017] | −0.03 [−1.13, +1.07] |
| Low / VD | +0.0026 [−0.0014, +0.0066] | −0.003 [−0.015, +0.009] | +0.39 [−0.71, +1.49] |
| High / EOPD(full) | +0.0060 [+0.0020, +0.0100] | +0.008 [−0.004, +0.020] | +0.65 [−0.45, +1.75] |
| High / Geometric-1 | +0.0071 [+0.0031, +0.0111] | +0.009 [−0.003, +0.021] | +0.73 [−0.37, +1.83] |
| High / Batch-mean- | +0.0074 [+0.0034, +0.0114] | +0.009 [−0.003, +0.021] | +0.77 [−0.33, +1.87] |
| High / Shuffled- | +0.0073 [+0.0033, +0.0113] | +0.009 [−0.003, +0.021] | +0.75 [−0.35, +1.85] |
| High / VD | +0.0079 [+0.0039, +0.0119] | +0.010 [−0.002, +0.022] | +0.84 [−0.26, +1.94] |
Oracle KL is measured in nats against a fixed four-candidate answer target . Wrong-answer probabilities use the same 67/119 eligible low/high states across methods. Positive changes mean reinforcement.
Among the five methods, VD has the largest low- point estimates for oracle KL improvement and continuation accuracy. The corresponding CIs span zero. The correctness categories also identify both cases you asked about: successful continuations under low agreement and shared wrong answers under high agreement. The latter can be reinforced by VD, consistent with the shared-bias analysis in Appendices D.9–D.10.
Q1(b). How sensitive are the gains to random paraphrases, simple templates, and lower-quality valid views?
Your proposed view-pool comparison changes the teacher views while keeping EOPD(full) on the original prompt. Two-view EOPD and VD each use training FLOPs. External paraphrase generation and human validation costs are outside that training budget.
| Training view pool | Mean | EOPD(full) | EOPD(2V) ± SD | VD ± SD | VD−2V, 95% CI | WorstView, 2V / VD |
|---|---|---|---|---|---|---|
| Simple templates | 0.740 | 34.8 | 35.3 ± 0.63 | 36.7 ± 0.62 | +1.4 [+0.4, +2.4] | 29.3 / 30.2 |
| Random valid paraphrases | 0.675 | 34.8 | 35.0 ± 0.46 | 36.1 ± 0.49 | +1.1 [+0.1, +2.1] | 29.5 / 30.5 |
| Lower-quality, task-preserving | 0.558 | 34.8 | 34.5 ± 0.51 | 34.8 ± 0.57 | +0.3 [−0.8, +1.4] | 27.5 / 27.9 |
With valid random paraphrases, VD improves over two-view EOPD by 1.1 points, with a positive 95% CI. Under weaker views, VD matches EOPD(full) and gains 0.3 points over two-view EOPD, with a CI spanning zero. Simple templates and verified paraphrases provide the most useful supervision among the tested pools.
Response to Reviewer MiMp (1/2): W1: Contribution and controlled comparisons
Thank you for the time you devoted to our paper and for your constructive suggestions. Below, we explain the contribution and present the additional analyses of teacher correctness and view selection.
Accuracy is in %, and SD is across three training seeds. The 95% CIs use paired question bootstrap conditional on the evaluated training runs.
Response. VD-OPD asks which teacher preferences remain supported when the same task is expressed differently. We evaluate this at the prefixes the student actually generates. The probability mass supported by both teacher views becomes the supervision signal. A single unnormalized target determines what the student learns and how strongly each state contributes.
EOPD [1] uses single-view teacher entropy. ToDi [2] uses teacher–student probability ratios, and TGOPD [3] uses verifier-based reliability at the prompt level. Veto [4] forms a geometric target from teacher–student agreement. VD measures agreement between task-preserving views of the same teacher at each student prefix.
For two tokens:
The same normalized target can arise from shared uncertainty or contradictory confidence. Its mass distinguishes these cases and sets their supervision strength. Building on the geometric pooling formulation in logarithmic opinion pools [5], VD retains this mass and uses it to guide learning along student-generated trajectories.
The benefit holds under controls for compute, teacher observations, and baseline tuning:
VD also improves on the strongest student-augmentation baseline under structural changes and paraphrases:
Table 3 also reports transfer to GPQA Diamond and MMLU-Pro.
[1] Entropy-Aware On-Policy Distillation of Language Models.
[2] ToDi: Token-wise Distillation via Fine-Grained Divergence Control.
[3] Verify Before You Distill: Prompt-Level Teacher Gating for On-Policy Distillation.
[4] Stable On-Policy Distillation through Adaptive Target Reformulation.
[5] Selecting Weighting Factors in Logarithmic Opinion Pools.