- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes VOPD to combine dense teacher supervision from OPD with correctness feedback from verifiable rewards. For multiple student-generated responses to the same prompt, VOPD computes the correlation between cumulative teacher implicit rewards and verifiable rewards, then uses this coefficient to scale the teacher-reward component of the update at each token position. The authors evaluate VOPD under same-size distillation and strong-to-weak distillation across eight benchmarks.
Critical strengths
- The proposed method retains both supervision signals and is straightforward to implement.
- The method design and update rule have clear mathematical interpretations, providing interpretability.
- The paper is clearly written, and the experiments complement and support the analysis of the method.
Critical weaknesses
-
The motivation and distinction from prior work are insufficiently established. The authors characterize existing methods as using “predefined reweighting rules to modulate the dense teacher signal,” but do not explain the specific limitations of this design, while VOPD also adopts such a rule. Besides, its novelty and rationale relative to OPDVR[1] remain unclear.
-
The direct comparisons with the most relevant recent methods (e.g. OPDVR[1], Uni-OPD[2]) are missing.
-
The reliability of the correlation estimates requires further investigation. Only four responses are generated per prompt, which may yield noisy correlation estimates. Experiments with larger group sizes would help assess their effects on coefficient stability and final performance.
-
A sensitivity analysis of the OPD top-k parameter is missing. As a key implementation parameter, the choice of top-k may affect the supervision signal and training outcomes. Evaluating different values of k would help establish whether the method is sensitive to the current setting.
[1]Lin W, Zhao J, Jiang X, et al. On-policy distillation with verifiable reward[J]. arXiv preprint arXiv:2608.24696, 2026.
[2] Hou W, Peng S, Wang W, et al. Uni-opd: Unifying on-policy distillation with a dual-perspective recipe[J]. arXiv preprint arXiv:2605.03677, 2026.
Questions for authors
Please refer to weaknesses.
3 comments
Response to Reviewer DHNd (2/2)
[W4] Top- distillation implementation and sensitivity
We agree that the implementation details and sensitivity of top- distillation are important for interpreting the results. As mentioned in Appendix A, our implementation selects the student's top- tokens at each visited prefix. Specifically, we select the highest-probability tokens under the current student policy, forming a candidate set . The full-vocabulary VOPD objective is then approximated by restricting the summation to this selected set:
Importantly, the student, teacher, and reference probabilities retain their full-vocabulary normalization and are not renormalized over the top- set. Thus, this approximation can be viewed as simply discarding signals from tokens outside the selected set.
To evaluate sensitivity to this approximation, we additionally train both ExOPD and VOPD using , compared with the default , under the strong-to-weak distillation setting.
| Method | Top- | AIME24 | AIME25 |
|---|---|---|---|
| ExOPD | 16 | 52.71 | 41.04 |
| ExOPD | 64 | 52.50 | 41.25 |
| VOPD | 16 | 57.29 | 42.71 |
| VOPD | 64 | 56.49 | 42.92 |
Increasing from 16 to 64 produces only minor changes in performance for both methods, suggesting that our results are not particularly sensitive to the selected support size within this range. This observation is also consistent with prior OPD studies [1, 2, 3], which report limited sensitivity to the choice of . In sum, the primary benefit of top- distillation is reducing the memory footprint and data transfer associated with storing and processing full-vocabulary logits, rather than improving performance.
We have clarified the token selection, normalization, and treatment of excluded tokens in the revised draft, along with the additional sensitivity results.
[1] Hübotter et al. “Reinforcement Learning via Self-Distillation.” arXiv, 2026.
[2] Li et al. “Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe.” TMLR, 2026.
[3] Jin et al. “Entropy-Aware On-Policy Distillation of Language Models.” ICML, 2026.
Looking forward to your feedback
Dear Reviewer DHNd,
We sincerely thank you for the detailed review and feedback.
As the author-reviewer discussion period is coming to a close on Nov 18th, we would appreciate it if you could clarify any remaining concerns you may have. This will ensure that we can adequately address them during this period.
Thank you again!
Best,
Authors
Response to Reviewer DHNd (1/2)
We appreciate the helpful feedback and address the points below.
We agree that the motivation and distinctions from OPDVR and Uni-OPD deserve further clarification. Similar to VOPD, both methods leverage verifier feedback to improve teacher supervision. However, their mechanisms may suppress or alter useful fine-grained teacher feedback.
OPDVR employs correctness-dependent gating that masks teacher signals inconsistent with verifier outcomes. While intuitive, this hard gating may discard useful feedback. For example, even when a trajectory produces the correct answer, the teacher's negative preference for certain tokens can still provide valuable guidance for improving the reasoning process.
Uni-OPD instead introduces outcome-guided margin calibration, adding uniform offsets to token-level teacher rewards to enforce a prescribed separation between correct and incorrect trajectories. However, these offsets may weaken or even reverse the negative rewards assigned to teacher-disfavored tokens, potentially reinforcing suboptimal reasoning steps despite a correct final outcome.
In contrast, VOPD derives its adaptation rule by projecting verifier advantages onto teacher-reward advantages. Rather than masking individual token rewards or introducing additive biases, VOPD uses the correlation coefficient cx to adjust the entire teacher-reward signal, retaining its fine-grained structure while adapting the direction and strength of supervision to verifier agreement. Furthermore, VOPD admits an explicit interpretation as distillation toward an adaptively extrapolated teacher distribution.
To further substantiate these distinctions, we implemented both OPDVR and Uni-OPD and evaluated them under our strong-to-weak distillation setting. For Uni-OPD, we implement its margin calibration mechanism but exclude its separate data-balancing strategy, which is orthogonal to our focus on verifier-guided teacher supervision. This allows a more controlled comparison of the distillation objectives.
According to the results above, both OPDVR and Uni-OPD yield only marginal improvements or comparable performance over standard OPD, and both fall behind ExOPD and VOPD, suggesting that their verifier-based gating and margin calibration provide limited benefits over standard teacher distillation.
We have expanded the discussion of methodological differences and added these empirical comparisons to the revised draft. .
We agree that estimating verifier–teacher correlation from only four responses may introduce noise. To investigate sensitivity to group size, we additionally evaluate VOPD with n=8 under the strong-to-weak distillation setting, compared with the default n=4.
Increasing the group size from n=4 to n=8 slightly reduces the fraction of groups with zero verifier-reward variance, allowing verifier-conditioned adaptation to be applied more frequently. Meanwhile, downstream performance remains comparable, with a slight improvement on AIME25. These results suggest that VOPD is relatively robust to the group size within the evaluated range.
We have added the group-size analysis to the revised draft.