acceptodds
Under review as a conference paper at ICLR 2027

Verifiable On-Policy Distillation

Abstract

Reinforcement learning with verifiable rewards (RLVR) has become a powerful approach for improving language-model reasoning, but its supervision is inherently sparse: a sequence-level verifier reward is propagated only through sampled trajectories and provides no direct signal for alternative actions. On-policy distillation (OPD) offers a complementary form of supervision by providing dense teacher feedback over the full next-token distribution, but this signal is not grounded in verified task outcomes. Simply co-optimizing the two objectives does not couple these signals: verifier feedback remains sparse and does not inform the dense teacher supervision. We introduce Verifiable On-Policy Distillation (VOPD), which transfers sparse verifier feedback into dense, verifier-grounded supervision. Our key observation is that the implicit-reward view of OPD places teacher supervision and verifier feedback in a common reward space while retaining a reward signal available for every next-token action. Under this view, VOPD projects verifier preference onto the teacher's implicit-reward direction and Rao–Blackwellizes the resulting signal over the full vocabulary, propagating verifier-grounded supervision to actions that were never sampled. The resulting objective can also be interpreted as adaptively extrapolating the distillation target according to teacher–verifier agreement. Across two teacher–student configurations and eight mathematical reasoning benchmarks, VOPD consistently improves both final performance and learning efficiency over vanilla OPD and other on-policy distillation baselines.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Knowledge Distillation, Reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_DHNdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes VOPD to combine dense teacher supervision from OPD with correctness feedback from verifiable rewards. For multiple student-generated responses to the same prompt, VOPD computes the correlation between cumulative teacher implicit rewards and verifiable rewards, then uses this coefficient to scale the teacher-reward component of the update at each token position. The authors evaluate VOPD under same-size distillation and strong-to-weak distillation across eight benchmarks.

Critical strengths
  1. The proposed method retains both supervision signals and is straightforward to implement.
  2. The method design and update rule have clear mathematical interpretations, providing interpretability.
  3. The paper is clearly written, and the experiments complement and support the analysis of the method.
Critical weaknesses
  1. The motivation and distinction from prior work are insufficiently established. The authors characterize existing methods as using “predefined reweighting rules to modulate the dense teacher signal,” but do not explain the specific limitations of this design, while VOPD also adopts such a rule. Besides, its novelty and rationale relative to OPDVR[1] remain unclear.

  2. The direct comparisons with the most relevant recent methods (e.g. OPDVR[1], Uni-OPD[2]) are missing.

  3. The reliability of the correlation estimates requires further investigation. Only four responses are generated per prompt, which may yield noisy correlation estimates. Experiments with larger group sizes would help assess their effects on coefficient stability and final performance.

  4. A sensitivity analysis of the OPD top-k parameter is missing. As a key implementation parameter, the choice of top-k may affect the supervision signal and training outcomes. Evaluating different values of k would help establish whether the method is sensitive to the current setting.

[1]Lin W, Zhao J, Jiang X, et al. On-policy distillation with verifiable reward[J]. arXiv preprint arXiv:2608.24696, 2026.

[2] Hou W, Peng S, Wang W, et al. Uni-opd: Unifying on-policy distillation with a dual-perspective recipe[J]. arXiv preprint arXiv:2605.03677, 2026.

Questions for authors

Please refer to weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer DHNd (1/2)

We appreciate the helpful feedback and address the points below.

[W1, W2] Motivation and distinction from relevant recent works

We agree that the motivation and distinctions from OPDVR and Uni-OPD deserve further clarification. Similar to VOPD, both methods leverage verifier feedback to improve teacher supervision. However, their mechanisms may suppress or alter useful fine-grained teacher feedback.

OPDVR employs correctness-dependent gating that masks teacher signals inconsistent with verifier outcomes. While intuitive, this hard gating may discard useful feedback. For example, even when a trajectory produces the correct answer, the teacher's negative preference for certain tokens can still provide valuable guidance for improving the reasoning process.

Uni-OPD instead introduces outcome-guided margin calibration, adding uniform offsets to token-level teacher rewards to enforce a prescribed separation between correct and incorrect trajectories. However, these offsets may weaken or even reverse the negative rewards assigned to teacher-disfavored tokens, potentially reinforcing suboptimal reasoning steps despite a correct final outcome.

In contrast, VOPD derives its adaptation rule by projecting verifier advantages onto teacher-reward advantages. Rather than masking individual token rewards or introducing additive biases, VOPD uses the correlation coefficient cxc_x to adjust the entire teacher-reward signal, retaining its fine-grained structure while adapting the direction and strength of supervision to verifier agreement. Furthermore, VOPD admits an explicit interpretation as distillation toward an adaptively extrapolated teacher distribution.

To further substantiate these distinctions, we implemented both OPDVR and Uni-OPD and evaluated them under our strong-to-weak distillation setting. For Uni-OPD, we implement its margin calibration mechanism but exclude its separate data-balancing strategy, which is orthogonal to our focus on verifier-guided teacher supervision. This allows a more controlled comparison of the distillation objectives.

MethodAIME24AIME25AMC23HMMT25-FebHMMT25-NovMATH-500MinervaOlympiad-BenchAverage
Teacher72.5062.2995.0037.7143.7594.2541.6466.1764.16
OPD50.6240.0086.8821.2526.2588.6034.8358.6450.88
OPDVR50.0039.5887.9722.7126.0490.4033.1859.4651.17
Uni-OPD51.6740.4287.3420.8324.3789.8534.2858.0550.85
ExOPD52.7141.0489.2222.5028.5488.9535.5760.2052.34
VOPD57.2942.7188.2822.2929.3890.2036.0360.3953.32

According to the results above, both OPDVR and Uni-OPD yield only marginal improvements or comparable performance over standard OPD, and both fall behind ExOPD and VOPD, suggesting that their verifier-based gating and margin calibration provide limited benefits over standard teacher distillation.

We have expanded the discussion of methodological differences and added these empirical comparisons to the revised draft. .

[W3] Group-size Sensitivity

We agree that estimating verifier–teacher correlation from only four responses may introduce noise. To investigate sensitivity to group size, we additionally evaluate VOPD with n=8n=8 under the strong-to-weak distillation setting, compared with the default n=4n=4.

Group sizeAIME24AIME25
n=4n=457.2942.71
n=8n=857.0843.54

Increasing the group size from n=4n=4 to n=8n=8 slightly reduces the fraction of groups with zero verifier-reward variance, allowing verifier-conditioned adaptation to be applied more frequently. Meanwhile, downstream performance remains comparable, with a slight improvement on AIME25. These results suggest that VOPD is relatively robust to the group size within the evaluated range.

We have added the group-size analysis to the revised draft.

AuthorsOpenReview ↗

Response to Reviewer DHNd (2/2)

[W4] Top-kk distillation implementation and sensitivity

We agree that the implementation details and sensitivity of top-kk distillation are important for interpreting the results. As mentioned in Appendix A, our implementation selects the student's top-kk tokens at each visited prefix. Specifically, we select the kk highest-probability tokens under the current student policy, forming a candidate set Sk(s)\mathcal S_k(s). The full-vocabulary VOPD objective is then approximated by restricting the summation to this selected set:

jVOPD(k)(θ;s)=cx∑a∈Sk(s)pθ(a∣s)log⁡pT(a∣s)pref(a∣s).j_{\mathrm{VOPD}}^{(k)}(\theta;s) = c_x\sum_{a\in\mathcal S_k(s)} p_\theta(a\mid s) \log\frac{p_T(a\mid s)}{p_{\mathrm{ref}}(a\mid s)}.

Importantly, the student, teacher, and reference probabilities retain their full-vocabulary normalization and are not renormalized over the top-kk set. Thus, this approximation can be viewed as simply discarding signals from tokens outside the selected set.

To evaluate sensitivity to this approximation, we additionally train both ExOPD and VOPD using k=64k=64, compared with the default k=16k=16, under the strong-to-weak distillation setting.

MethodTop-kkAIME24AIME25
ExOPD1652.7141.04
ExOPD6452.5041.25
VOPD1657.2942.71
VOPD6456.4942.92

Increasing kk from 16 to 64 produces only minor changes in performance for both methods, suggesting that our results are not particularly sensitive to the selected support size within this range. This observation is also consistent with prior OPD studies [1, 2, 3], which report limited sensitivity to the choice of kk. In sum, the primary benefit of top-kk distillation is reducing the memory footprint and data transfer associated with storing and processing full-vocabulary logits, rather than improving performance.

We have clarified the token selection, normalization, and treatment of excluded tokens in the revised draft, along with the additional sensitivity results.

[1] Hübotter et al. “Reinforcement Learning via Self-Distillation.” arXiv, 2026.

[2] Li et al. “Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe.” TMLR, 2026.

[3] Jin et al. “Entropy-Aware On-Policy Distillation of Language Models.” ICML, 2026.

AuthorsOpenReview ↗

Looking forward to your feedback

Dear Reviewer DHNd,

We sincerely thank you for the detailed review and feedback.

As the author-reviewer discussion period is coming to a close on Nov 18th, we would appreciate it if you could clarify any remaining concerns you may have. This will ensure that we can adequately address them during this period.

Thank you again!

Best,

Authors

Reviewer_oW4bOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

VOPD adapts distillation strength using a correlation between teacher scores and verifier outcomes within each rollout group. On eight math benchmarks, its average gains over ExOPD are ~1.13 and ~0.98 points for 4B and 1.7B students.

Critical strengths
  • The nondegenerate least-squares coefficient and fixed-coefficient target identity are interpretable and valid.
  • VOPD leads trained methods on six of eight benchmarks in both settings; the 1.7B AIME24 gain over ExOPD is 4.58 points.
  • Final checkpoints and substantial verifier/training details are reported.
Critical weaknesses
  1. Expectation claim: Equation 15 takes a coefficient outside an action expectation although it was fitted from those actions and future verifier outcomes. Detaching does not create independence; the discarded baseline has related dependence. This leaves a legitimate local surrogate and fixed-coefficient identity, but not the claimed general equivalence.
  2. Attribution: The strong-to-weak diagnostic's mean correlation of 0.36 reduces the extra coefficient from 0.25 to approximately 0.09 in those groups. A smaller constant could explain the gain without prompt-specific adaptation.
  3. Reliability: Roughly one-point margins lack accuracy uncertainty or independent runs. AIME's development/test roles need clarification. Zero-variance handling and top-16 normalization/tail treatment are unspecified. These gaps do not establish insignificant gains or contamination.
Questions for authors
  1. Can you state the conditioning behind Equation 15 and narrow the equivalence claim, or justify independent coefficient estimation?
  2. Please compare tuned constant-strength ExOPD, mean-matched constants, and shuffled coefficients under the same protocol.
  3. What are the zero-variance fallback, top-16 loss, and development/test selection rules? Please add held-out uncertainty and document the OPD+GRPO tuning budget.
  4. Can you provide a better general figure and more details for the experiments?
4 comments
AuthorsOpenReview ↗

Response to Reviewer oW4b (1/3)

We appreciate the helpful feedback and address the points below.

[W1,Q1] Conditioning in the Rao–Blackwellization derivation

We thank the reviewer for pointing out the ambiguity in Eq. (15). The key distinction is that Rao–Blackwellization marginalizes over a hypothetical next-token action, rather than the actions involved in the sampled rollouts used to estimate cxc_x.

Specifically, we treat the rollout group Gx\mathcal G_x and visited prefix ss as already observed, and consider only randomness from a new hypothetical next-token action a∼pθ(⋅∣s)a\sim p_\theta(\cdot\mid s). Conditional on Gx\mathcal G_x and ss, both cxc_x and the baseline Dˉ\bar D are fixed. Consequently, cxc_x can be taken outside this action expectation, while the baseline term vanishes by the score-function identity: $ \mathbb E_{a\sim p_\theta(\cdot\mid s)} \left[\nabla_\theta\log p_\theta(a\mid s)\right]=0.$ The resulting full-vocabulary gradient of VOPD therefore has the same expectation as the sampled-action gradient under this conditional construction.

For further details, we refer the reviewer to the revised Eq. (15) and its surrounding discussion, where we explicitly clarify the conditioning assumptions and the scope of this gradient equivalence.

[W2,Q2] Isolation of the benefit of verifier-conditioned adaptation

We agree that independently tuning extrapolation strength is important to isolate the contribution of verifier-conditioned adaptation. We therefore conducted an additional sweep on the fixed-extrapolation baseline ExOPD over coefficients λ∈{0,0.1,0.2,0.25,0.5}\lambda\in\{0,0.1,0.2,0.25,0.5\}, using the same evaluation setting as the original extrapolation ablation in Section 5.3. We clarify that VOPD's average extrapolation strength is approximately 0.20.2, rather than 0.090.09, due to the zero-variance fallback described in our response to Zero-variance Handling. Specifically, groups with undefined correlation (approximately 65% of rollout groups) use the default coefficient 0.250.25, while the remaining groups use 0.25cx0.25c_x. Given the observed mean correlation of approximately 0.360.36 among groups with defined correlation, the overall average extrapolation coefficient is approximately 0.1940.194. We therefore include ExOPD with λ=0.2\lambda=0.2 as a mean-matched baseline.

Beyond the coefficient sweep, we additionally evaluate two VOPD variants: shuffled cxc_x, which breaks the correspondence between verifier alignment and extrapolation strength, and sign-reversed cxc_x, which replaces cxc_x with −cx-c_x to intentionally extrapolate in the direction opposite to that suggested by the verifier.

Methodλ\lambdaAIME24AIME25
ExOPD0.0050.6240.00
ExOPD0.1052.2940.83
ExOPD (mean-matched)0.2052.2941.25
ExOPD0.2552.7141.04
ExOPD0.5050.8339.79
VOPD (shuffled cxc_x)0.2551.8840.63
VOPD (sign-reversed cxc_x)0.2549.7939.38
VOPD0.2557.2942.71

ExOPD achieves its best performance at λ=0.25\lambda=0.25, while λ=0.2\lambda=0.2 yields comparable results. VOPD still outperforms this mean-matched baseline, indicating that its gains cannot be explained simply by weaker average extrapolation.

Furthermore, both shuffled-cxc_x and sign-reversed VOPD perform worse than ExOPD. In particular, the shuffled control indicates that the benefit depends on the correspondence between verifier feedback and extrapolation strength, rather than merely the distribution of coefficients. Together, these results provide evidence that VOPD benefits from verifier-aligned extrapolation, rather than simply using a different average extrapolation strength or coefficient distribution.

We have clarified the coefficient parameterization and added these controlled comparisons to the revised draft.

AuthorsOpenReview ↗

Response to Reviewer oW4b (2/3)

[W3,Q3] Zero-variance Handling and Group-size Sensitivity

When either the verifier rewards have zero within-group variance, the correlation is undefined. In our implementation, these groups fall back to the default extrapolation coefficient of 0.250.25, while groups with nonzero variance use the verifier-conditioned coefficient 0.25cx0.25c_x. This fallback is motivated by the empirical benefit of fixed extrapolation over standard OPD, as demonstrated by ExOPD in our experiments. We have clarified this implementation detail in the revised draft.

To investigate how frequently this fallback occurs and its sensitivity to group size, we additionally evaluate VOPD with n=8n=8 under the strong-to-weak setting and compare it with the default n=4n=4. Figure 4 in the revised draft reports the fractions of all-correct, all-incorrect, and total zero-variance groups throughout training. The table below compares downstream performance.

Group sizeAIME24AIME25
n=4n=457.2942.71
n=8n=857.0843.54

With n=4n=4, approximately 65% of rollout groups have zero verifier-reward variance on average. As training progresses, the fraction of all-incorrect groups decreases while the fraction of all-correct groups increases, leading to an overall increase in zero-variance groups. Increasing the group size to n=8n=8 slightly reduces the fraction of zero-variance groups, allowing more groups to provide informative verifier–teacher correlation estimates. Despite this difference, VOPD achieves similar or slightly better performance with n=8n=8, suggesting that the method is relatively robust to group size. The modest improvement is consistent with more groups receiving the beneficial verifier-conditioned extrapolation, although other effects of increasing group size cannot be ruled out.

We have clarified the zero-variance fallback in the method section and added the group-size analysis to the revised draft.

[W3,Q3] Top-kk distillation implementation and sensitivity

We agree that the implementation details and sensitivity of top-kk distillation are important for interpreting the results. As mentioned in Appendix A, our implementation selects the student's top-kk tokens at each visited prefix. Specifically, we select the kk highest-probability tokens under the current student policy, forming a candidate set Sk(s)\mathcal S_k(s). The full-vocabulary VOPD objective is then approximated by restricting the summation to this selected set:

jVOPD(k)(θ;s)=cx∑a∈Sk(s)pθ(a∣s)log⁡pT(a∣s)pref(a∣s).j_{\mathrm{VOPD}}^{(k)}(\theta;s) = c_x\sum_{a\in\mathcal S_k(s)} p_\theta(a\mid s) \log\frac{p_T(a\mid s)}{p_{\mathrm{ref}}(a\mid s)}.

Importantly, the student, teacher, and reference probabilities retain their full-vocabulary normalization and are not renormalized over the top-kk set. Thus, this approximation can be viewed as simply discarding signals from tokens outside the selected set.

To evaluate sensitivity to this approximation, we additionally train both ExOPD and VOPD using k=64k=64, compared with the default k=16k=16, under the strong-to-weak distillation setting.

MethodTop-kkAIME24AIME25
ExOPD1652.7141.04
ExOPD6452.5041.25
VOPD1657.2942.71
VOPD6456.4942.92

Increasing kk from 16 to 64 produces only minor changes in performance for both methods, suggesting that our results are not particularly sensitive to the selected support size within this range. This observation is also consistent with prior OPD studies [1, 2, 3], which report limited sensitivity to the choice of kk. In sum, the primary benefit of top-kk distillation is reducing the memory footprint and data transfer associated with storing and processing full-vocabulary logits, rather than improving performance.

We have clarified the token selection, normalization, and treatment of excluded tokens in the revised draft, along with the additional sensitivity results.

[1] Hübotter et al. “Reinforcement Learning via Self-Distillation.” arXiv, 2026.

[2] Li et al. “Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe.” TMLR, 2026.

[3] Jin et al. “Entropy-Aware On-Policy Distillation of Language Models.” ICML, 2026.

AuthorsOpenReview ↗

Response to Reviewer oW4b (3/3)

[W3,Q3,Q4] Experimental Reliability and Evaluation Protocol

Training-seed variability. We agree that reporting variability across independent training runs is important for assessing the reliability of the performance improvements. We therefore conducted two additional independent training runs for OPD, ExOPD, and VOPD under the strong-to-weak distillation setting (Qwen3-4B teacher and Qwen3-1.7B student). The results show the mean and standard deviation across the three independent training runs. Moreover, as specified in Table 4, we control for evaluation randomness by evaluating each checkpoint across multiple generation seeds and averaging the results.

MethodAIME24AIME25AMC23HMMT25-FebHMMT25-NovMATH-500MinervaOlympiad-BenchAverage
OPD50.10±0.7450.10\pm0.7441.25±1.7741.25\pm1.7786.09±1.1186.09\pm1.1121.56±0.4521.56\pm0.4526.46±0.3026.46\pm0.3089.12±0.7489.12\pm0.7434.00±1.1734.00\pm1.1758.83±0.2658.83\pm0.2650.92±0.0650.92\pm0.06
ExOPD52.19±0.7452.19\pm0.7442.50±2.0642.50\pm2.0688.98±0.3388.98\pm0.3323.23±1.0323.23\pm1.0328.64±0.1528.64\pm0.1589.28±0.4689.28\pm0.4635.34±0.3335.34\pm0.3359.73±0.6659.73\pm0.6652.52±0.0852.52\pm0.08
VOPD56.04±1.7756.04\pm1.7743.34±0.8843.34\pm0.8887.97±0.4487.97\pm0.4422.91±0.8822.91\pm0.8829.48±0.1429.48\pm0.1489.53±0.9589.53\pm0.9535.94±0.1335.94\pm0.1360.13±0.3760.13\pm0.3753.13±0.0353.13\pm0.03

The overall performance trends remain consistent across the two training runs, with relatively small variation in average accuracy. VOPD achieves the highest mean score of 53.13, compared with 52.52 for ExOPD and 50.92 for OPD. These additional results provide further evidence that the improvement over ExOPD is reproducible across independent training runs.

Evaluation protocol and checkpoint selection. We use AIME24 and AIME25 to monitor downstream performance during training because they are relatively inexpensive to evaluate and empirically exhibit larger performance differences across checkpoints than other benchmarks. This is why we primarily use them for training curves in Figure 1 and hyperparameter ablations. However, we do not select checkpoints based on their AIME performance. As specified in Appendix A.1, we report the checkpoint shortly after training convergence for all methods, corresponding to step 140 in the 4B same-size setting and step 160 in the strong-to-weak setting.

Hyperparameter tuning. The primary hyperparameter tuned for VOPD is the extrapolation coefficient λ\lambda, whose sensitivity is studied in Section 5.3. Other training hyperparameters follow the standard OPD configuration to ensure a controlled comparison and are not tuned. For the OPD+GRPO co-optimization baseline, we use a shared learning rate of 2×10−62\times10^{-6}, matching the default learning rate used by both components. We also experimented with doubling the GRPO learning rate but observed training instability, so we retained the default configuration.

Method visualization. We agree that a more comprehensive illustration would improve the presentation. We have added an overview figure (Figure 1) to the revised draft to better visualize how VOPD combines the complementary signals in RLVR and OPD to achieve dense verifier-agreed supervision.

AuthorsOpenReview ↗

Looking forward to your feedback

Dear Reviewer oW4b,

We sincerely thank you for the detailed review and feedback.

As the author-reviewer discussion period is coming to a close on Nov 18th, we would appreciate it if you could clarify any remaining concerns you may have. This will ensure that we can adequately address them during this period.

Thank you again!

Best,

Authors

Reviewer_quXBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Verifiable On-Policy Distillation (VOPD) to combine dense teacher supervision with sparse sequence-level verifier feedback in mathematical reasoning. Starting from the implicit-reward interpretation of reverse-KL on-policy distillation, it defines a cumulative teacher reward for each sampled response and estimates its within-prompt correlation (cxc_x), with binary verifier outcomes. It then uses cxc_x to scale the teacher-reference log-probability ratio, yielding a dense local distillation term. A Rao-Blackwellization argument connects the sampled-token estimator to a vocabulary expectation, and the resulting objective can be interpreted as distillation toward an adaptively extrapolated teacher distribution. The experiments use one RL-trained Qwen3-4B teacher with Qwen3-4B and Qwen3-1.7B students, evaluated on eight mathematical benchmarks. VOPD attains mean scores of 66.38 and 53.32, respectively, compared with 65.25 and 52.34 for fixed-extrapolation ExOPD, and shows favorable step-wise validation curves.

Critical strengths
  1. A coherent and practical integration of two supervision signals. Eqs. (6)-(18) provide a compact construction: teacher implicit rewards are aggregated across responses, aligned with verifier outcomes via a prompt-level correlation, and converted into a tractable distillation objective. The adaptive-target interpretation in Eq. (18) is useful for understanding both reinforcement and opposition of the teacher direction.

  2. Promising results across two student sizes and multiple evaluations. VOPD improves the reported eight-benchmark average over OPD, OPD+GRPO, and ExOPD in both the 4B and 1.7B student settings (Tables 1-2), and Figure 1 suggests faster improvement in training-step terms. The presence of an additive OPD+GRPO control and analyses of the distribution of (cxc_x)(Figure 2) makes the empirical story more informative than a simple final-score comparison.

Critical weaknesses
  1. The evaluation does not sufficiently isolate the benefit of verifier-conditioned adaptation. The main advantage over the closest fixed-extrapolation baseline, ExOPD, is relatively modest: +1.13 average points for the 4B student and +0.98 for the 1.7B student (Tables 1–2). Both ExOPD and VOPD use $\lambda=0.25$, but it appears VOPD's actual extrapolation factor is λcx\lambda c_x rather than λ\lambda; Figure 2(c) reports mean $c_x\approx0.36 $ in the analyzed strong-to-weak setting. Thus the comparison does not control for average extrapolation magnitude. There is no reported tuning sweep for ExOPD, no repeated-seed variability or confidence intervals for the final results, and no empirical comparison with verifier-aware alternatives discussed in the related work (e.g., OPDVR or Uni-OPD). These omissions limit confidence that the observed small gains are reliably attributable to the proposed projection.

  2. Important gaps remain between the claimed dense verifier transfer, its approximation, and its implementation. The correlation in Eq. (9) is estimated from only four responses per prompt (Appendix Table 5) with binary verifier rewards (Appendix A.1), and is undefined when the verifier rewards are all identical (or when the cumulative teacher rewards have zero variance). Algorithm 1 does not specify what occurs in these cases, nor how frequently they occur. Moreover, the transition from the sequence-level projected signal in Eq. (13) to Eq. (14) uses an immediate-token approximation; the subsequent Rao–Blackwellization preserves the expectation of that local surrogate, not necessarily the gradient of the original verifier-based sequence objective. Finally, the actual experiments use the student's top 16 tokens rather than the full vocabulary (Section 5.1 and Appendix A.1). These details matter to the paper's central interpretation and to reproducibility.

Questions for authors
  1. Can you tune ExOPD's extrapolation coefficient independently, including values matched to the empirical mean of λcx\lambda c_x, and compare against constant-mean, sign-only, or shuffled-cxc_x controls? Please also report multiple seeds and variability for the 4B and 1.7B results. This would help determine whether the contribution is adaptive verifier alignment rather than simply weaker regularized extrapolation.

  2. With group size four and binary verifier rewards, how are groups with σV=0\sigma_V=0 or σD=0\sigma_D=0 handled? What fraction of all prompt groups receive a nonzero verifier-conditioned update, and how sensitive is performance to group size or the noise in estimated cxc_x?

  3. Can you clarify precisely which gradient is preserved by the Rao–Blackwellization after the immediate-token approximation, and provide an ablation of the top-kk approximation (e.g., k=16k=16 versus larger kk)? This would establish how closely the implemented method realizes the proposed dense verifier-grounded objective.

5 comments
AuthorsOpenReview ↗

Response to Reviewer quXB (1/4)

We appreciate the helpful feedback and address the points below.

[W1,Q1] Isolation of the benefit of verifier-conditioned adaptation

We agree that independently tuning extrapolation strength is important to isolate the contribution of verifier-conditioned adaptation. We therefore conducted an additional sweep on the fixed-extrapolation baseline ExOPD over coefficients λ∈{0,0.1,0.2,0.25,0.5}\lambda\in\{0,0.1,0.2,0.25,0.5\}, using the same evaluation setting as the original extrapolation ablation in Section 5.3. We clarify that VOPD's average extrapolation strength is approximately 0.20.2, rather than 0.090.09, due to the zero-variance fallback described in our response to Zero-variance Handling. Specifically, groups with undefined correlation (approximately 65% of rollout groups) use the default coefficient 0.250.25, while the remaining groups use 0.25cx0.25c_x. Given the observed mean correlation of approximately 0.360.36 among groups with defined correlation, the overall average extrapolation coefficient is approximately 0.1940.194. We therefore include ExOPD with λ=0.2\lambda=0.2 as a mean-matched baseline.

Beyond the coefficient sweep, we additionally evaluate two VOPD variants: shuffled cxc_x, which breaks the correspondence between verifier alignment and extrapolation strength, and sign-reversed cxc_x, which replaces cxc_x with −cx-c_x to intentionally extrapolate in the direction opposite to that suggested by the verifier.

Methodλ\lambdaAIME24AIME25
ExOPD0.0050.6240.00
ExOPD0.1052.2940.83
ExOPD (mean-matched)0.2052.2941.25
ExOPD0.2552.7141.04
ExOPD0.5050.8339.79
VOPD (shuffled cxc_x)0.2551.8840.63
VOPD (sign-reversed cxc_x)0.2549.7939.38
VOPD0.2557.2942.71

ExOPD achieves its best performance at λ=0.25\lambda=0.25, while λ=0.2\lambda=0.2 yields comparable results. VOPD still outperforms this mean-matched baseline, indicating that its gains cannot be explained simply by weaker average extrapolation.

Furthermore, both shuffled-cxc_x and sign-reversed VOPD perform worse than ExOPD. In particular, the shuffled control indicates that the benefit depends on the correspondence between verifier feedback and extrapolation strength, rather than merely the distribution of coefficients. Together, these results provide evidence that VOPD benefits from verifier-aligned extrapolation, rather than simply using a different average extrapolation strength or coefficient distribution.

We have clarified the coefficient parameterization and added these controlled comparisons to the revised draft.

AuthorsOpenReview ↗

Response to Reviewer quXB (2/4)

[W1] Motivation and distinction from relevant recent works

We agree that the motivation and distinctions from OPDVR and Uni-OPD deserve further clarification. Similar to VOPD, both methods leverage verifier feedback to improve teacher supervision. However, their mechanisms may suppress or alter useful fine-grained teacher feedback.

OPDVR employs correctness-dependent gating that masks teacher signals inconsistent with verifier outcomes. While intuitive, this hard gating may discard useful feedback. For example, even when a trajectory produces the correct answer, the teacher's negative preference for certain tokens can still provide valuable guidance for improving the reasoning process.

Uni-OPD instead introduces outcome-guided margin calibration, adding uniform offsets to token-level teacher rewards to enforce a prescribed separation between correct and incorrect trajectories. However, these offsets may weaken or even reverse the negative rewards assigned to teacher-disfavored tokens, potentially reinforcing suboptimal reasoning steps despite a correct final outcome.

In contrast, VOPD derives its adaptation rule by projecting verifier advantages onto teacher-reward advantages. Rather than masking individual token rewards or introducing additive biases, VOPD uses the correlation coefficient cxc_x to adjust the entire teacher-reward signal, retaining its fine-grained structure while adapting the direction and strength of supervision to verifier agreement. Furthermore, VOPD admits an explicit interpretation as distillation toward an adaptively extrapolated teacher distribution.

To further substantiate these distinctions, we implemented both OPDVR and Uni-OPD and evaluated them under our strong-to-weak distillation setting. For Uni-OPD, we implement its margin calibration mechanism but exclude its separate data-balancing strategy, which is orthogonal to our focus on verifier-guided teacher supervision. This allows a more controlled comparison of the distillation objectives.

MethodAIME24AIME25AMC23HMMT25-FebHMMT25-NovMATH-500MinervaOlympiad-BenchAverage
Teacher72.5062.2995.0037.7143.7594.2541.6466.1764.16
OPD50.6240.0086.8821.2526.2588.6034.8358.6450.88
OPDVR50.0039.5887.9722.7126.0490.4033.1859.4651.17
Uni-OPD51.6740.4287.3420.8324.3789.8534.2858.0550.85
ExOPD52.7141.0489.2222.5028.5488.9535.5760.2052.34
VOPD57.2942.7188.2822.2929.3890.2036.0360.3953.32

According to the results above, both OPDVR and Uni-OPD yield only marginal improvements or comparable performance over standard OPD, and both fall behind ExOPD and VOPD, suggesting that their verifier-based gating and margin calibration provide limited benefits over standard teacher distillation.

We have expanded the discussion of methodological differences and added these empirical comparisons to the revised draft.

[W1] Experimental Reliability

We agree that reporting variability across independent training runs is important for assessing the reliability of the performance improvements. We therefore conducted two additional independent training runs for OPD, ExOPD, and VOPD under the strong-to-weak distillation setting (Qwen3-4B teacher and Qwen3-1.7B student). The results show the mean and standard deviation across the three independent training runs. Moreover, as specified in Table 4, we control for evaluation randomness by evaluating each checkpoint across multiple generation seeds and averaging the results.

MethodAIME24AIME25AMC23HMMT25-FebHMMT25-NovMATH-500MinervaOlympiad-BenchAverage
OPD50.10±0.7450.10\pm0.7441.25±1.7741.25\pm1.7786.09±1.1186.09\pm1.1121.56±0.4521.56\pm0.4526.46±0.3026.46\pm0.3089.12±0.7489.12\pm0.7434.00±1.1734.00\pm1.1758.83±0.2658.83\pm0.2650.92±0.0650.92\pm0.06
ExOPD52.19±0.7452.19\pm0.7442.50±2.0642.50\pm2.0688.98±0.3388.98\pm0.3323.23±1.0323.23\pm1.0328.64±0.1528.64\pm0.1589.28±0.4689.28\pm0.4635.34±0.3335.34\pm0.3359.73±0.6659.73\pm0.6652.52±0.0852.52\pm0.08
VOPD56.04±1.7756.04\pm1.7743.34±0.8843.34\pm0.8887.97±0.4487.97\pm0.4422.91±0.8822.91\pm0.8829.48±0.1429.48\pm0.1489.53±0.9589.53\pm0.9535.94±0.1335.94\pm0.1360.13±0.3760.13\pm0.3753.13±0.0353.13\pm0.03

The overall performance trends remain consistent across the two training runs, with relatively small variation in average accuracy. VOPD achieves the highest mean score of 53.13, compared with 52.52 for ExOPD and 50.92 for OPD. These additional results provide further evidence that the improvement over ExOPD is reproducible across independent training runs.

AuthorsOpenReview ↗

Response to Reviewer quXB (3/4)

[W2, Q2] Zero-variance Handling and Group-size Sensitivity

When either the verifier rewards have zero within-group variance, the correlation is undefined. In our implementation, these groups fall back to the default extrapolation coefficient of 0.250.25, while groups with nonzero variance use the verifier-conditioned coefficient 0.25cx0.25c_x. This fallback is motivated by the empirical benefit of fixed extrapolation over standard OPD, as demonstrated by ExOPD in our experiments. We have clarified this implementation detail in the revised draft.

To investigate how frequently this fallback occurs and its sensitivity to group size, we additionally evaluate VOPD with n=8n=8 under the strong-to-weak setting and compare it with the default n=4n=4. Figure 4 in the revised draft reports the fractions of all-correct, all-incorrect, and total zero-variance groups throughout training. The table below compares downstream performance.

Group sizeAIME24AIME25
n=4n=457.2942.71
n=8n=857.0843.54

With n=4n=4, approximately 65% of rollout groups have zero verifier-reward variance on average. As training progresses, the fraction of all-incorrect groups decreases while the fraction of all-correct groups increases, leading to an overall increase in zero-variance groups. Increasing the group size to n=8n=8 slightly reduces the fraction of zero-variance groups, allowing more groups to provide informative verifier–teacher correlation estimates. Despite this difference, VOPD achieves similar or slightly better performance with n=8n=8, suggesting that the method is relatively robust to group size. The modest improvement is consistent with more groups receiving the beneficial verifier-conditioned extrapolation, although other effects of increasing group size cannot be ruled out.

We have clarified the zero-variance fallback in the method section and added the group-size analysis to the revised draft.

[W2, Q3] Immediate-token approximation and Rao–Blackwellization

We agree that the immediate-token approximation does not preserve the original sequence-level policy gradient. However, this approximation is adopted in the standard OPD formulation and brings advantages over the full sequence-level gradient. As analyzed in [1], although the approximation introduces bias relative to the sequence-level objective, it substantially reduces gradient variance by removing future-reward coupling. VOPD adopts the same approximation, with an additional verifier-derived coefficient cxc_x scaling the teacher-reward signal.

Regarding which gradient is preserved after Rao–Blackwellization, we clarify that it is applied after the immediate-token approximation. Therefore, it preserves the expected local gradient after adopting the immediate-token approximation, rather than recovering the original sequence-level gradient.

We have clarified this distinction in the revised derivation.

[1] Fu et al. “Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes.” COLM, 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.