- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper investigates the training instability of on-policy distillation (OPD) caused by substantial distribution mismatch between teacher and student models. The authors propose Trust Region On-Policy Distillation (TrOPD), which consists of three components: (1) trust-region learning that applies reverse-KL supervision to student-generated tokens selected by a teacher–student probability ratio; (2) outlier estimation that uses top-k forward KL to preserve informative supervision outside the trust region; and (3) off-policy guidance that combines teacher-generated prefixes with student-generated continuations to improve exploration. The authors also establish a unified evaluation setting for existing OPD methods. Experiments across multiple teacher–student configurations and benchmarks covering mathematical reasoning, code generation, instruction following, and STEM tasks show that TrOPD consistently improves over existing OPD baselines. Additional ablation studies examine the contributions of individual components and the robustness of key design choices.
Critical strengths
1. Meaningful problem and effective method design. The paper addresses an important practical challenge in on-policy distillation: unreliable supervision caused by teacher–student distribution mismatch. TrOPD provides a reasonable solution by combining trust-region learning, outlier estimation, and off-policy guidance to improve training stability while preserving informative supervision.
2. Strong empirical performance and comprehensive evaluation. The experiments cover multiple teacher–student configurations and diverse benchmarks, including mathematical reasoning, code generation, instruction following, and STEM tasks. TrOPD consistently outperforms competitive OPD baselines, including EOPD and REOPOLD. The ablation studies further demonstrate the contributions of different components, providing convincing evidence of the method's practical effectiveness.
Critical weaknesses
1. Limited methodological novelty compared with existing OPD approaches. The proposed framework combines several established techniques, including token-level KL optimization, outlier handling, and teacher-prefix guidance. Recent methods such as EOPD, REOPOLD, and AOPD have already explored related strategies for improving supervision reliability. Although TrOPD introduces a probability-ratio-based trust-region criterion, its fundamental novelty over these approaches remains somewhat limited.
2. Insufficient justification for the trust-region criterion. The proposed trust-region selection rule is inspired by the acceptance probability in speculative decoding, but its connection to gradient reliability in OPD is not rigorously established. A high teacher–student probability ratio does not necessarily imply that the corresponding policy gradient is reliable or beneficial for reasoning performance. Moreover, the ablation against a fixed-threshold alternative shows only a modest average improvement (0.49 points), leaving uncertainty about whether the specific trust-region criterion is essential to the overall performance gains.
Questions for authors
1. Novelty and contribution of the proposed components. Several components of TrOPD are closely related to existing techniques, including adaptive KL objectives, outlier handling, and teacher-prefix guidance. While the overall framework achieves promising empirical results, could the authors clarify the key methodological novelty of TrOPD compared with existing approaches such as EOPD and REOPOLD? In particular, which design choice is primarily responsible for the improvement beyond combining established techniques?
2. Justification and effectiveness of the trust-region criterion. The proposed trust-region criterion is motivated by the acceptance probability in speculative decoding, but its relationship to policy-gradient reliability is not sufficiently established. Could the authors explain why this probability ratio provides a reliable measure of supervision quality in OPD? Additionally, Appendix B.1 reports only a 0.49-point average improvement over a fixed-threshold alternative. Is there further evidence that the proposed adaptive criterion is essential to the overall performance gains, rather than the improvements mainly arising from outlier forward-KL supervision and off-policy guidance?
2 comments
W1 Limited methodological novelty compared with existing OPD approaches. The proposed framework combines several established techniques, including token-level KL optimization, outlier handling, and teacher-prefix guidance. Recent methods such as EOPD, REOPOLD, and AOPD have already explored related strategies for improving supervision reliability. Although TrOPD introduces a probability-ratio-based trust-region criterion, its fundamental novelty over these approaches remains somewhat limited.
Our main contribution is to identify and empirically validate the reliability problem of teacher supervision in OPD, and develop a stand-alone, unified policy-gradient method based on a trust region defined through TV distance. The novelty lies in unified supervision allocation and outlier regularization within this framework.
1. TrOPD defines the supervision region through teacher–student distributional overlap. REOPOLD clips extreme RKL rewards without explicitly defining teacher-supervision confidence through distributional overlap; clipping can also attenuate informative supervision. In contrast, TrOPD uses TV to characterize the shared probability mass: the expected trust-region acceptance rate equals . It then selectively regularizes rejected tokens using a discrepancy-based reward. Full FKL bounds squared TV through Pinsker's inequality, motivating the practical Top- FKL reward. This gives the selection-and-regularization mechanism an explicit distributional meaning. In Table 4, REOPOLD improves over OPD by only 0.27 points on average, whereas TrOPD improves by 3.44 points.
2. TrOPD uses unified policy-gradient optimization rather than adding a direct imitation-learning objective. EOPD and AOPD directly optimize FKL on selected tokens to imitate the teacher, combining policy gradients with imitation learning. TrOPD instead uses negative FKL as a detached reward to regularize outlier sampled tokens, motivated by the TV bound; gradients do not propagate through the FKL expression as an imitation-learning loss. Crucially, the sampled token determines whether the regularization applies through the trust-region gate, while the teacher's Top- distribution determines its strength. The gate and regularizer therefore play interdependent roles. This makes TrOPD a stand-alone policy-gradient approach, complementary to the imitation-learning components of EOPD and AOPD. With suitable data, imitation-learning methods such as EOPD and AOPD can still be used to further improve performance.
Q1 In particular, which design choice is primarily responsible for the improvement beyond combining established techniques?
Based on our response to W2, we attribute the main improvement to defining trust through TV distance and regularizing tokens outside the trust region using direct TV estimates or an FKL-based surrogate as rewards.
Q2 Additionally, Appendix B.1 reports only a 0.49-point average improvement over a fixed-threshold alternative. Is there further evidence that the proposed adaptive criterion is essential to the overall performance gains, rather than the improvements mainly arising from outlier forward-KL supervision and off-policy guidance?
Appendix B.1 primarily examines robustness to different trust-region definitions. All variants include our proposed FKL reward regularization and therefore already achieve strong performance. By contrast, REOPOLD does not use this FKL reward regularizer, and TrOPD outperforms it by 3.17 points in Table 4.
Additionally, following checks prompted by other reviewers' comments, we identified a transcription error in Appendix B.1: the correct AMC 23 score for adaptive acceptance is 78.51, giving an average of 49.85. The actual improvement is 0.98 points, rather than 0.49. All other components are held unchanged in this comparison, isolating the contribution of the trust-region selection rule.
We will incorporate these additional explanations into the final version of the paper. Thank you again for your review.
Thank you for the constructive feedback; this is exactly the point we wish to clarify. The original presentation may lead to some misunderstanding: TrOPD's components are connected operations within a stand-alone policy-gradient framework, without direct imitation learning, rather than independent techniques. Below, we clarify their roles and dependencies, particularly the distinction between trusted supervision and outlier reward regularization. We first explain this formulation and its supporting evidence, then discuss its core differences from existing OPD methods.
The trust-region criterion has a unified interpretation:
- The acceptance rule has a distributional-overlap interpretation beyond its speculative-decoding motivation. Because OPD uses student-generated trajectories, we argue that reliable teacher supervision is largely confined to regions of teacher–student distributional overlap. We quantify this overlap using total variation (TV) distance, TV(πT,πS)=21∑v∈W∣πT(v)−πS(v)∣=1−∑v∈Wmin{πT(v),πS(v)}.
The acceptance rule has a concrete probability-mass interpretation: the expected fraction of student-sampled tokens assigned to the trusted branch equals the teacher–student overlap mass, while the expected rejected fraction equals their TV distance:
=\mathbb E_{x\sim\pi_S}\left[\min\left(\frac{\pi_T(x)}{\pi_S(x)},1\right)\right] =1-\operatorname{TV}(\pi_T,\pi_S).$$ Thus, the trusted branch selects shared probability mass, while the rejected branch selects the student's excess mass relative to the teacher. Its expected size adapts to the distributional mismatch. This connects the probability-ratio rule to the allocation of supervision under teacher–student mismatch, rather than relying solely on its analogy to speculative decoding. 2) **This overlap interpretation determines how supervision and regularization are combined through stand-alone policy-gradient optimization**: accepted tokens are treated as lying within the teacher's reliable supervision region and retain standard sampled-token OPD. In standard on-policy OPD, using the negative $K_1$ estimate, $\log(\pi_T(x)/\pi_S(x))$, as a fixed advantage gives the gradient for minimizing expected reverse KL. For tokens sampled outside the trust region, teacher–student distributional mismatch can make the teacher supervision unreliable. We therefore use negative FKL as the advantage to penalize sampled tokens outside the trust region, rather than as a direct imitation-learning loss: $$A(x)=-\sum_{v\in\mathcal V_T^{64}}\pi_T(v)\log\frac{\pi_T(v)}{\pi_S(v)}, \qquad \operatorname{TV}(\pi_T,\pi_S)\le\sqrt{\tfrac12D_{\mathrm{KL}}(\pi_T\|\pi_S)}.$$ By Pinsker's inequality, FKL provides an upper bound on squared TV; we use its Top-64 truncation as a practical reward-level regularization signal, analogous in role to a KL penalty in PPO. 3) **To test this interpretation beyond an FKL-specific effect, we directly use TV-based penalties as outlier rewards**, instead of the FKL surrogate motivated by the TV bound: | Method | Outlier Regularization | AIME 24 | AIME 25 | AMC 23 | Avg. | |---|---|---:|---:|---:|---:| | DeepSeek-Qwen-1.5B | -- | 28.64 | 24.16 | 71.01 | 41.27 | | OPD | -- | 35.83 | 29.16 | 75.39 | 46.79 | | $\mathrm{TrOPD}_{\text{sample-token TV}}$ | $-(1-\min\{1,\pi_T(x)/\pi_S(x)\})$ | 38.54 | 30.52 | 77.34 | 48.80 | | $\mathrm{TrOPD}_{\text{top64 TV}}$ | $-\frac12\sum_{v\in\mathcal V_T^{64}}\lvert\pi_T(v)-\pi_S(v)\rvert$ | 38.33 | 30.21 | 78.44 | 48.99 | | $\mathrm{TrOPD}_{\text{top64 FKL}}$ | $-\sum_{v\in\mathcal V_T^{64}}\pi_T(v)\log\frac{\pi_T(v)}{\pi_S(v)}$ | 39.16 | 29.89 | 77.96 | 49.00 | To separate this evidence from prefix guidance, we compare different distance-based outlier rewards under the same acceptance rule and policy-gradient framework, with off-policy prefix guidance disabled. **TrOPD (sample-token TV)** uses $D(h,x)=1-P_{\mathrm{trust}}(x)$, with $\mathbb{E}[D(h,x)]=\operatorname{TV}(\pi_T,\pi_S)$ for student samples $x\sim\pi_S$. **TrOPD (top64 TV)** uses $D(h,x)=\frac{1}{2}\sum_{v\in\mathcal V_T^{64}}\lvert\pi_T(v)-\pi_S(v)\rvert$, a truncated TV measure over the teacher's Top-64 tokens. All three variants achieve similar performance and consistently improve over OPD, supporting outlier regularization outside the trust region across different discrepancy-based rewards, rather than an effect specific to FKL. **Therefore, the trust-region criterion only partitions trusted OPD supervision and OPD regularization; our contribution is a stand-alone, unified policy-gradient method for OPD based on TV as a measure.**