acceptodds
Under review as a conference paper at ICLR 2027

Trust Region On-Policy Distillation

11 views

Abstract

On-Policy Distillation (OPD) is a fundamental technique for efficient post-training of large language models (LLMs), with broad applications in agent learning, multi-task enhancement, and model compression. However, OPD training becomes unstable when the teacher and student distributions differ substantially: teacher supervision on student-generated tokens may then yield unreliable policy gradients and even cause optimization failure. To make token-level on-policy supervision reliable, we propose Trust Region On-Policy Distillation (TrOPD), which consists of three components: 1) Trust-Region On-Policy Learning: TrOPD performs OPD only in regions where the teacher provides reliable supervision, mitigating the optimization difficulty of the K_1 reverse-KL estimator under distribution mismatch. 2) Outlier Estimation: For outliers outside the trust region, we compare clipping, masking, and forward-KL estimation, and adopt top-k forward KL to retain informative supervision without unreliable gradients. 3) The student imitates teacher-generated prefixes via forward KL and then continues generation on-policy, steering exploration toward the trust region. Experiments show that TrOPD consistently outperforms SoTA OPD baselines, including OPD, EOPD, and REOPOLD, across mathematical reasoning, code generation, instruction following, and STEM benchmarks. Our code is publicly available to further benchmark advanced OPD methods in community.

Primary area:
foundation or frontier models, including LLMs
Keywords:
on-policy distillation, small language model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_PLxzOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates the training instability of on-policy distillation (OPD) caused by substantial distribution mismatch between teacher and student models. The authors propose Trust Region On-Policy Distillation (TrOPD), which consists of three components: (1) trust-region learning that applies reverse-KL supervision to student-generated tokens selected by a teacher–student probability ratio; (2) outlier estimation that uses top-k forward KL to preserve informative supervision outside the trust region; and (3) off-policy guidance that combines teacher-generated prefixes with student-generated continuations to improve exploration. The authors also establish a unified evaluation setting for existing OPD methods. Experiments across multiple teacher–student configurations and benchmarks covering mathematical reasoning, code generation, instruction following, and STEM tasks show that TrOPD consistently improves over existing OPD baselines. Additional ablation studies examine the contributions of individual components and the robustness of key design choices.

Critical strengths

1. Meaningful problem and effective method design. The paper addresses an important practical challenge in on-policy distillation: unreliable supervision caused by teacher–student distribution mismatch. TrOPD provides a reasonable solution by combining trust-region learning, outlier estimation, and off-policy guidance to improve training stability while preserving informative supervision.

2. Strong empirical performance and comprehensive evaluation. The experiments cover multiple teacher–student configurations and diverse benchmarks, including mathematical reasoning, code generation, instruction following, and STEM tasks. TrOPD consistently outperforms competitive OPD baselines, including EOPD and REOPOLD. The ablation studies further demonstrate the contributions of different components, providing convincing evidence of the method's practical effectiveness.

Critical weaknesses

1. Limited methodological novelty compared with existing OPD approaches. The proposed framework combines several established techniques, including token-level KL optimization, outlier handling, and teacher-prefix guidance. Recent methods such as EOPD, REOPOLD, and AOPD have already explored related strategies for improving supervision reliability. Although TrOPD introduces a probability-ratio-based trust-region criterion, its fundamental novelty over these approaches remains somewhat limited.

2. Insufficient justification for the trust-region criterion. The proposed trust-region selection rule is inspired by the acceptance probability in speculative decoding, but its connection to gradient reliability in OPD is not rigorously established. A high teacher–student probability ratio does not necessarily imply that the corresponding policy gradient is reliable or beneficial for reasoning performance. Moreover, the ablation against a fixed-threshold alternative shows only a modest average improvement (0.49 points), leaving uncertainty about whether the specific trust-region criterion is essential to the overall performance gains.

Questions for authors

1. Novelty and contribution of the proposed components. Several components of TrOPD are closely related to existing techniques, including adaptive KL objectives, outlier handling, and teacher-prefix guidance. While the overall framework achieves promising empirical results, could the authors clarify the key methodological novelty of TrOPD compared with existing approaches such as EOPD and REOPOLD? In particular, which design choice is primarily responsible for the improvement beyond combining established techniques?

2. Justification and effectiveness of the trust-region criterion. The proposed trust-region criterion is motivated by the acceptance probability in speculative decoding, but its relationship to policy-gradient reliability is not sufficiently established. Could the authors explain why this probability ratio provides a reliable measure of supervision quality in OPD? Additionally, Appendix B.1 reports only a 0.49-point average improvement over a fixed-threshold alternative. Is there further evidence that the proposed adaptive criterion is essential to the overall performance gains, rather than the improvements mainly arising from outlier forward-KL supervision and off-policy guidance?

2 comments

Thank you for the constructive feedback; this is exactly the point we wish to clarify. The original presentation may lead to some misunderstanding: TrOPD's components are connected operations within a stand-alone policy-gradient framework, without direct imitation learning, rather than independent techniques. Below, we clarify their roles and dependencies, particularly the distinction between trusted supervision and outlier reward regularization. We first explain this formulation and its supporting evidence, then discuss its core differences from existing OPD methods.

W2 Insufficient justification for the trust-region criterion. The proposed trust-region selection rule is inspired by the acceptance probability in speculative decoding, but its connection to gradient reliability in OPD is not rigorously established.

The trust-region criterion has a unified interpretation:

  1. The acceptance rule has a distributional-overlap interpretation beyond its speculative-decoding motivation. Because OPD uses student-generated trajectories, we argue that reliable teacher supervision is largely confined to regions of teacher–student distributional overlap. We quantify this overlap using total variation (TV) distance, TV⁡(πT,πS)=12∑v∈W∣πT(v)−πS(v)∣=1−∑v∈Wmin⁡{πT(v),πS(v)}\operatorname{TV}(\pi_T,\pi_S)=\frac12\sum_{v\in\mathcal W}|\pi_T(v)-\pi_S(v)|=1-\sum_{v\in\mathcal W}\min\{\pi_T(v),\pi_S(v)\}. The acceptance rule has a concrete probability-mass interpretation: the expected fraction of student-sampled tokens assigned to the trusted branch equals the teacher–student overlap mass, while the expected rejected fraction equals their TV distance:
=\mathbb E_{x\sim\pi_S}\left[\min\left(\frac{\pi_T(x)}{\pi_S(x)},1\right)\right] =1-\operatorname{TV}(\pi_T,\pi_S).$$ Thus, the trusted branch selects shared probability mass, while the rejected branch selects the student's excess mass relative to the teacher. Its expected size adapts to the distributional mismatch. This connects the probability-ratio rule to the allocation of supervision under teacher–student mismatch, rather than relying solely on its analogy to speculative decoding. 2) **This overlap interpretation determines how supervision and regularization are combined through stand-alone policy-gradient optimization**: accepted tokens are treated as lying within the teacher's reliable supervision region and retain standard sampled-token OPD. In standard on-policy OPD, using the negative $K_1$ estimate, $\log(\pi_T(x)/\pi_S(x))$, as a fixed advantage gives the gradient for minimizing expected reverse KL. For tokens sampled outside the trust region, teacher–student distributional mismatch can make the teacher supervision unreliable. We therefore use negative FKL as the advantage to penalize sampled tokens outside the trust region, rather than as a direct imitation-learning loss: $$A(x)=-\sum_{v\in\mathcal V_T^{64}}\pi_T(v)\log\frac{\pi_T(v)}{\pi_S(v)}, \qquad \operatorname{TV}(\pi_T,\pi_S)\le\sqrt{\tfrac12D_{\mathrm{KL}}(\pi_T\|\pi_S)}.$$ By Pinsker's inequality, FKL provides an upper bound on squared TV; we use its Top-64 truncation as a practical reward-level regularization signal, analogous in role to a KL penalty in PPO. 3) **To test this interpretation beyond an FKL-specific effect, we directly use TV-based penalties as outlier rewards**, instead of the FKL surrogate motivated by the TV bound: | Method | Outlier Regularization | AIME 24 | AIME 25 | AMC 23 | Avg. | |---|---|---:|---:|---:|---:| | DeepSeek-Qwen-1.5B | -- | 28.64 | 24.16 | 71.01 | 41.27 | | OPD | -- | 35.83 | 29.16 | 75.39 | 46.79 | | $\mathrm{TrOPD}_{\text{sample-token TV}}$ | $-(1-\min\{1,\pi_T(x)/\pi_S(x)\})$ | 38.54 | 30.52 | 77.34 | 48.80 | | $\mathrm{TrOPD}_{\text{top64 TV}}$ | $-\frac12\sum_{v\in\mathcal V_T^{64}}\lvert\pi_T(v)-\pi_S(v)\rvert$ | 38.33 | 30.21 | 78.44 | 48.99 | | $\mathrm{TrOPD}_{\text{top64 FKL}}$ | $-\sum_{v\in\mathcal V_T^{64}}\pi_T(v)\log\frac{\pi_T(v)}{\pi_S(v)}$ | 39.16 | 29.89 | 77.96 | 49.00 | To separate this evidence from prefix guidance, we compare different distance-based outlier rewards under the same acceptance rule and policy-gradient framework, with off-policy prefix guidance disabled. **TrOPD (sample-token TV)** uses $D(h,x)=1-P_{\mathrm{trust}}(x)$, with $\mathbb{E}[D(h,x)]=\operatorname{TV}(\pi_T,\pi_S)$ for student samples $x\sim\pi_S$. **TrOPD (top64 TV)** uses $D(h,x)=\frac{1}{2}\sum_{v\in\mathcal V_T^{64}}\lvert\pi_T(v)-\pi_S(v)\rvert$, a truncated TV measure over the teacher's Top-64 tokens. All three variants achieve similar performance and consistently improve over OPD, supporting outlier regularization outside the trust region across different discrepancy-based rewards, rather than an effect specific to FKL. **Therefore, the trust-region criterion only partitions trusted OPD supervision and OPD regularization; our contribution is a stand-alone, unified policy-gradient method for OPD based on TV as a measure.**

W1 Limited methodological novelty compared with existing OPD approaches. The proposed framework combines several established techniques, including token-level KL optimization, outlier handling, and teacher-prefix guidance. Recent methods such as EOPD, REOPOLD, and AOPD have already explored related strategies for improving supervision reliability. Although TrOPD introduces a probability-ratio-based trust-region criterion, its fundamental novelty over these approaches remains somewhat limited.

Our main contribution is to identify and empirically validate the reliability problem of teacher supervision in OPD, and develop a stand-alone, unified policy-gradient method based on a trust region defined through TV distance. The novelty lies in unified supervision allocation and outlier regularization within this framework.

1. TrOPD defines the supervision region through teacher–student distributional overlap. REOPOLD clips extreme K1K_1 RKL rewards without explicitly defining teacher-supervision confidence through distributional overlap; clipping can also attenuate informative supervision. In contrast, TrOPD uses TV to characterize the shared probability mass: the expected trust-region acceptance rate equals 1−TV⁡(πT,πS)1-\operatorname{TV}(\pi_T,\pi_S). It then selectively regularizes rejected tokens using a discrepancy-based reward. Full FKL bounds squared TV through Pinsker's inequality, motivating the practical Top-kk FKL reward. This gives the selection-and-regularization mechanism an explicit distributional meaning. In Table 4, REOPOLD improves over OPD by only 0.27 points on average, whereas TrOPD improves by 3.44 points.

2. TrOPD uses unified policy-gradient optimization rather than adding a direct imitation-learning objective. EOPD and AOPD directly optimize FKL on selected tokens to imitate the teacher, combining policy gradients with imitation learning. TrOPD instead uses negative FKL as a detached reward to regularize outlier sampled tokens, motivated by the TV bound; gradients do not propagate through the FKL expression as an imitation-learning loss. Crucially, the sampled token determines whether the regularization applies through the trust-region gate, while the teacher's Top-kk distribution determines its strength. The gate and regularizer therefore play interdependent roles. This makes TrOPD a stand-alone policy-gradient approach, complementary to the imitation-learning components of EOPD and AOPD. With suitable data, imitation-learning methods such as EOPD and AOPD can still be used to further improve performance.

Q1 In particular, which design choice is primarily responsible for the improvement beyond combining established techniques?

Based on our response to W2, we attribute the main improvement to defining trust through TV distance and regularizing tokens outside the trust region using direct TV estimates or an FKL-based surrogate as rewards.

Q2 Additionally, Appendix B.1 reports only a 0.49-point average improvement over a fixed-threshold alternative. Is there further evidence that the proposed adaptive criterion is essential to the overall performance gains, rather than the improvements mainly arising from outlier forward-KL supervision and off-policy guidance?

Appendix B.1 primarily examines robustness to different trust-region definitions. All variants include our proposed FKL reward regularization and therefore already achieve strong performance. By contrast, REOPOLD does not use this FKL reward regularizer, and TrOPD outperforms it by 3.17 points in Table 4.

Additionally, following checks prompted by other reviewers' comments, we identified a transcription error in Appendix B.1: the correct AMC 23 score for adaptive acceptance is 78.51, giving an average of 49.85. The actual improvement is 0.98 points, rather than 0.49. All other components are held unchanged in this comparison, isolating the contribution of the trust-region selection rule.

We will incorporate these additional explanations into the final version of the paper. Thank you again for your review.

Reviewer_TMRhOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

To make token-level on-policy supervision reliable, the paper propose an on policy distillation variant: TrOPD. TrOPD performs OPD only in regions where the teacher provides reliable supervision, mitigating the optimization difficulty in distribution mismatch. Experiments show that TrOPD outperforms some OPD baselines, such as OPD, EOPD, and REOPOLD.

Critical strengths
  1. The motivation is clear. By treating trajectories that are in/out the trust region with different objectives, TrOPD reduces the noise during optimization.

  2. The ablations are clear. The paper compares different objectives for outliers, such as clip/mask/FKL, shows that TrOPD is a better choice.

  3. The evaluation cover mathematical reasoning, code generation, instruction following, and STEM benchmarks, which is comprehensive.

Critical weaknesses
  1. A low teacher-student probability ratio identifies there is disagreement between them, but does not directly establish that the resulting supervision is unreliable. Are there evidence connects the probability-ratio acceptance rule to supervision reliability.

  2. The evaluation seems to only conduct on models of small size (1.5B model as student and 4B model as the teacher), are there more experiments conducted on larger models?

  3. Could you reconcile the default TrOPD results in Tables 1, 3, and 5 and Appendix B? In particular, why is AMC23 reported as 78.51 in Tables 1/5 but 77.03 in Table 3 and the appendix?

Questions for authors

See weaknesses

2 comments

W1 A low teacher-student probability ratio identifies there is disagreement between them, but does not directly establish that the resulting supervision is unreliable. Are there evidence connects the probability-ratio acceptance rule to supervision reliability?

Yes. We first define the trust region using the TV distance between the teacher and student distributions, with an expected acceptance probability of 1−TV⁡(πT,πS)1-\operatorname{TV}(\pi_T,\pi_S). We regularize rejected outliers using TV distance or an FKL-based upper bound, and optimize the entire method through a unified policy-gradient framework:

  1. Because OPD uses student-generated trajectories, we argue that reliable teacher supervision is largely confined to regions of teacher–student distributional overlap. We quantify this overlap using total variation (TV) distance, TV⁡(πT,πS)=12∑v∈W∣πT(v)−πS(v)∣=1−∑v∈Wmin⁡{πT(v),πS(v)}\operatorname{TV}(\pi_T,\pi_S)=\frac12\sum_{v\in\mathcal W}|\pi_T(v)-\pi_S(v)|=1-\sum_{v\in\mathcal W}\min\{\pi_T(v),\pi_S(v)\}. The acceptance rule has a concrete probability-mass interpretation: the expected fraction of student-sampled tokens assigned to the trusted branch equals the teacher–student overlap mass, while the expected rejected fraction equals their TV distance:
=\mathbb E_{x\sim\pi_S}\left[\min\left(\frac{\pi_T(x)}{\pi_S(x)},1\right)\right] =1-\operatorname{TV}(\pi_T,\pi_S).$$ Thus, the trusted branch selects shared probability mass, while the rejected branch selects the student's excess mass relative to the teacher. Its expected size adapts to the distributional mismatch. This gives the OBRS-style rule a precise interpretation for allocating supervision. 2) **To perform OPD according to trust region quantified by TV distance, we propose the stand-alone policy-gradient optimization**: accepted tokens are treated as lying within the teacher's reliable supervision region and retain standard sampled-token OPD. In standard on-policy OPD, using the negative $K_1$ estimate, $\log(\pi_T(x)/\pi_S(x))$, as a fixed advantage gives the gradient for minimizing expected reverse KL. For tokens sampled outside the trust region, teacher–student distributional mismatch can make the teacher supervision unreliable. We therefore use negative FKL as the advantage to penalize sampled tokens outside the trust region, rather than as a direct imitation-learning loss: $$A(x)=-\sum_{v\in\mathcal V_T^{64}}\pi_T(v)\log\frac{\pi_T(v)}{\pi_S(v)}, \qquad \operatorname{TV}(\pi_T,\pi_S)\le\sqrt{\tfrac12D_{\mathrm{KL}}(\pi_T\|\pi_S)}.$$ By Pinsker's inequality, FKL provides an upper bound on squared TV; we use its Top-64 truncation as a practical reward-level regularization signal, analogous in role to a KL penalty in PPO. 3) To further validate the effectiveness of TV-distance regularization, **we directly use TV-based penalties as outlier rewards for tokens outside the trust region**, instead of the FKL surrogate motivated by the upper bound of TV distance: | Method | Outlier Regularization | AIME 24 | AIME 25 | AMC 23 | Avg. | |---|---|---:|---:|---:|---:| | DeepSeek-Qwen-1.5B | -- | 28.64 | 24.16 | 71.01 | 41.27 | | OPD | -- | 35.83 | 29.16 | 75.39 | 46.79 | | $\mathrm{TrOPD}_{\text{sample-token TV}}$ | $-(1-\min\{1,\pi_T(x)/\pi_S(x)\})$ | 38.54 | 30.52 | 77.34 | 48.80 | | $\mathrm{TrOPD}_{\text{top64 TV}}$ | $-\frac12\sum_{v\in\mathcal V_T^{64}}\lvert\pi_T(v)-\pi_S(v)\rvert$ | 38.33 | 30.21 | 78.44 | 48.99 | | $\mathrm{TrOPD}_{\text{top64 FKL}}$ | $-\sum_{v\in\mathcal V_T^{64}}\pi_T(v)\log\frac{\pi_T(v)}{\pi_S(v)}$ | 39.16 | 29.89 | 77.96 | 49.00 | We compare different distance-based rewards for outlier regularization in TrOPD, using the same acceptance rule and policy-gradient framework and disabling off-policy prefix guidance to isolate the outlier signal. **TrOPD (sample-token TV)** uses $D(h,x)=1-P_{\mathrm{trust}}(x)$, with $\mathbb{E}[D(h,x)]=\operatorname{TV}(\pi_T,\pi_S)$ for student samples $x\sim\pi_S$. **TrOPD (top64 TV)** uses $D(h,x)=\frac{1}{2}\sum_{v\in\mathcal V_T^{64}}\lvert\pi_T(v)-\pi_S(v)\rvert$, a truncated TV measure over the teacher's Top-64 tokens. All three variants achieve similar performance and consistently improve over OPD, supporting outlier regularization outside the trust region across different discrepancy-based rewards, rather than an effect specific to FKL.

W2 The evaluation seems to only conduct on models of small size (1.5B model as student and 4B model as the teacher), are there more experiments conducted on larger models?

Thank you for the suggestion. We conducted additional experiments with Skywork-OR1-32B as the teacher and DeepSeek-R1-Distill-Qwen-7B as the student, training OPD, REOPOLD, and TrOPD for 200 steps:

MethodAIME 24AIME 25AMC 23Avg.
OPD51.1540.2184.3058.55
REOPOLD52.1939.4885.9459.20
TrOPD56.1541.6788.4462.08

TrOPD outperforms both baselines, improving average accuracy over OPD by +3.53 points, demonstrating its effectiveness with larger teacher and student models.

W3 Could you reconcile the default TrOPD results in Tables 1, 3, and 5 and Appendix B? In particular, why is AMC23 reported as 78.51 in Tables 1/5 but 77.03 in Table 3 and the appendix?

Thank you for checking. We have corrected the table transcription error: 77.03 is the AMC 23 score for TrOPD Clip in Table 5, while the correct score for TrOPD is 78.51.

We will incorporate these additional explanations into the final version of the paper. We sincerely appreciate your review. Thank you again for your time.

Reviewer_df5UOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that distilling reasoning capabilities (which requires long context generation) on existing OPD algorithms may suffer from instabilities if teacher and student differs much in distribution, and yet full vocab OPD is expensive in memory overhead. To address this, they propose TrOPD that uses K1 estimator when the student and teacher matches, but top-k forward KL when they fall out of the trust region. The trust region is selected by a probabilistic method.

Critical strengths
  1. The paper targets an important topic in OPD: reasoning (long context generation) distillation, and provides a practically useful recipe.

  2. By comparing existing OPD baselines, this paper achieves improved metrics over them.

  3. Some ablation studies, including masking/clipping the tokens outside the trust region are presented.

Critical weaknesses
  1. The trust region is heuristic. The term has an OBRS acceptance form, but its interpretation as a criterion for reliable teacher supervision is insufficiently justified. The adaptive design’s contribution is also only weakly supported by the existing ablations.

  2. The source of the gains remains underexplained. Improvements may partly come from implicit outlier suppression or training–inference mismatch correction, especially with raw advantages. Switching rejected tokens to FKL does not separate these stabilization effects from the claimed mechanism.

  3. The efficiency framing is overstated. Although the paper acknowledges sampled-token K1, its emphasis on the 2.4 TB full-vocabulary estimate gives a misleading impression of existing OPD methods. Sampled-token RKL already uses O(n) probability storage, and small-support alternatives are available. My concern is with this framing, rather than the fairness of the reported baseline comparison. Even if we use FKL, we can still use top-K as a valid choice.

  4. The stated objectives and implementation need better examination. The supplied code appears to use the FKL value as a detached PPO advantage, which generally produces a different gradient from directly optimizing FKL. The prefix objective also needs clarification. This complicates interpretation of both the method and the reported standalone FKL collapse.

Questions for authors
  1. Do the gains persist against a sampled-token RKL baseline with training–inference mismatch measured and properly controlled?

  2. Why do the default results differ between Tables 1/5 and Tables 7–9?

(See weaknesses for those to be explained points. )

9 comments

W1 The trust region is heuristic. The term has an OBRS acceptance form, but its interpretation as a criterion for reliable teacher supervision is insufficiently justified. The adaptive design’s contribution is also only weakly supported by the existing ablations.

  1. Because OPD uses student-generated trajectories, we argue that reliable teacher supervision is largely confined to regions of teacher–student distributional overlap. We quantify this overlap using total variation (TV) distance, TV⁡(πT,πS)=12∑v∈W∣πT(v)−πS(v)∣=1−∑v∈Wmin⁡{πT(v),πS(v)}\operatorname{TV}(\pi_T,\pi_S)=\frac12\sum_{v\in\mathcal W}|\pi_T(v)-\pi_S(v)|=1-\sum_{v\in\mathcal W}\min\{\pi_T(v),\pi_S(v)\}. The acceptance rule has a concrete probability-mass interpretation: the expected fraction of student-sampled tokens assigned to the trusted branch equals the teacher–student overlap mass, while the expected rejected fraction equals their TV distance:
=\mathbb E_{x\sim\pi_S}\left[\min\left(\frac{\pi_T(x)}{\pi_S(x)},1\right)\right] =1-\operatorname{TV}(\pi_T,\pi_S).$$ Thus, the trusted branch selects shared probability mass, while the rejected branch selects the student's excess mass relative to the teacher. Its expected size adapts to the distributional mismatch. This gives the OBRS-style rule a precise interpretation for allocating supervision. 2) **To perform OPD according to trust region quantified by TV distance, we propose the stand-alone policy-gradient optimization**: accepted tokens are treated as lying within the teacher's reliable supervision region and retain standard sampled-token OPD. In standard on-policy OPD, using the negative $K_1$ estimate, $\log(\pi_T(x)/\pi_S(x))$, as a fixed advantage gives the gradient for minimizing expected reverse KL. For tokens sampled outside the trust region, teacher–student distributional mismatch can make the teacher supervision unreliable. We therefore use negative FKL as the advantage to penalize sampled tokens outside the trust region, rather than as a direct imitation-learning loss: $$A(x)=-\sum_{v\in\mathcal V_T^{64}}\pi_T(v)\log\frac{\pi_T(v)}{\pi_S(v)}, \qquad \operatorname{TV}(\pi_T,\pi_S)\le\sqrt{\tfrac12D_{\mathrm{KL}}(\pi_T\|\pi_S)}.$$ By Pinsker's inequality, FKL provides an upper bound on squared TV; we use its Top-64 truncation as a practical reward-level regularization signal, analogous in role to a KL penalty in PPO. 3) To further validate the effectiveness of TV-distance regularization, **we directly use TV-based penalties as outlier rewards for tokens outside the trust region**, instead of the FKL surrogate motivated by the upper bound of TV distance: | Method | Outlier Regularization | AIME 24 | AIME 25 | AMC 23 | Avg. | |---|---|---:|---:|---:|---:| | DeepSeek-Qwen-1.5B | -- | 28.64 | 24.16 | 71.01 | 41.27 | | OPD | -- | 35.83 | 29.16 | 75.39 | 46.79 | | $\mathrm{TrOPD}_{\text{sample-token TV}}$ | $-(1-\min\{1,\pi_T(x)/\pi_S(x)\})$ | 38.54 | 30.52 | 77.34 | 48.80 | | $\mathrm{TrOPD}_{\text{top64 TV}}$ | $-\frac12\sum_{v\in\mathcal V_T^{64}}\lvert\pi_T(v)-\pi_S(v)\rvert$ | 38.33 | 30.21 | 78.44 | 48.99 | | $\mathrm{TrOPD}_{\text{top64 FKL}}$ | $-\sum_{v\in\mathcal V_T^{64}}\pi_T(v)\log\frac{\pi_T(v)}{\pi_S(v)}$ | 39.16 | 29.89 | 77.96 | 49.00 | We compare different distance-based rewards for outlier regularization in TrOPD, using the same acceptance rule and policy-gradient framework and disabling off-policy prefix guidance to isolate the outlier signal. **TrOPD (sample-token TV)** uses $D(h,x)=1-P_{\mathrm{trust}}(x)$, with $\mathbb{E}[D(h,x)]=\operatorname{TV}(\pi_T,\pi_S)$ for student samples $x\sim\pi_S$. **TrOPD (top64 TV)** uses $D(h,x)=\frac{1}{2}\sum_{v\in\mathcal V_T^{64}}\lvert\pi_T(v)-\pi_S(v)\rvert$, a truncated TV measure over the teacher's Top-64 tokens. All three variants achieve similar performance and consistently improve over OPD, supporting outlier regularization outside the trust region across different discrepancy-based rewards, rather than an effect specific to FKL.

W2 The source of the gains remains underexplained. Improvements may partly come from implicit outlier suppression or training–inference mismatch correction, especially with raw advantages. Switching rejected tokens to FKL does not separate these stabilization effects from the claimed mechanism.

The W1 comparisons support the source of improvement: teacher supervision within the trust region and reward regularization outside it. Replacing FKL with TV rewards yields similar gains over OPD, supporting discrepancy-based regularization beyond FKL. We added diagnostic experiments in Appendices E and F to examine this mechanism and training–inference mismatch.

  1. Effectiveness of outlier regularization (Appendix E, Figure 4). We monitor mean teacher-Top-64 FKL and shared teacher–student probability mass at outlier positions. OPD's outlier FKL increases from approximately 0.5 to 1.5, while its probability overlap decreases from approximately 0.65 to 0.52. These results support outlier regularization in preventing teacher–student divergence and improving performance.

  2. Ruling out training–inference mismatch correction as the source of gains (Appendix F, Figure 5). We compare training- and rollout-engine log probabilities for the same tokens, over all valid positions and outlier positions. OPD and TrOPD exhibit comparable differences: the all-token means fluctuate near −5×10−4-5\times10^{-4}, and the outlier curves also substantially overlap. Thus, the gains occur without a clear reduction in training–inference mismatch, arguing against mismatch correction as their source. In contrast, Figure 4 shows a clear separation in teacher–student discrepancy. Together with W1, this supports trust-region supervision and outlier regularization as the source of improvement.

W3 The efficiency framing is overstated. Although the paper acknowledges sampled-token K1, its emphasis on the 2.4 TB full-vocabulary estimate gives a misleading impression of existing OPD methods. Sampled-token RKL already uses O(n) probability storage, and small-support alternatives are available.

Our previous discussion of full KL was intended to explain the need for memory efficient KL estimators, rather than to emphasize the efficiency of our method. Specifically, we aimed to distinguish instruction OPD, where full KL is commonly used, from reasoning OPD, where long generations make full KL impractical and sampled-token OPD has become a representative approach. Since this discussion appeared in the problem formulation, it served only as background. Thank you for this suggestion. We have accordingly removed the relevant statements from the revised manuscript and now directly explain the need for KL estimators in reasoning OPD.

W4 The stated objectives and implementation need better examination. The supplied code appears to use the FKL value as a detached PPO advantage, which generally produces a different gradient from directly optimizing FKL. The prefix objective also needs clarification. This complicates interpretation of both the method and the reported standalone FKL collapse.

  1. Outlier FKL is a reward regularizer. As clarified in W1, when a student-sampled token falls outside the trust region, its negative Top-6464 FKL value is used as a stop-gradient reward (advantage). The student is updated through policy gradients; we do not directly differentiate through FKL to perform imitation learning on these tokens. We will emphasize this distinction and clarify the corresponding objectives in the final version.

  2. Teacher-generated prefixes. As specified in the Off-Policy Guidance row of Table 2, prefix tokens are sampled from the teacher, x[:l]∼πTx[:l]\sim\pi_T. The K1K_1 estimator therefore gives an unbiased numerical estimate of forward KL: $ \mathbb E_{x\sim\pi_T} \left[\log\frac{\pi_T(x)}{\pi_S(x)}\right] =\mathrm{KL}(\pi_T\parallel\pi_S). $ We use its negative, βlog⁡(πS(x)/πT(x))\beta\log(\pi_S(x)/\pi_T(x)), as a sampled-token FKL reward regularizer to penalize teacher–student distributional mismatch.

  3. Standalone Top-kk FKL rewards provide no token-discriminative signal. At a fixed context, the detached Top-kk FKL value FKL\mathrm{FKL} is the same for every possible sampled token xx: $ \mathbb E_{x\sim\pi_S} \left[-\mathrm{FKL}\nabla_\theta\log\pi_S(x)\right] =-\mathrm{FKL}\nabla_\theta\sum_x\pi_S(x)=0. $ The absence of a token-discriminative reward helps explain the failure of this baseline. It differs from the sampled-token FKL reward in Point 2, whose log-ratio depends on the sampled token. It also differs from TrOPD, where the sampled token determines the TV-related acceptance probability, and the outlier penalty is token-dependent. We will clarify these distinctions and the corresponding derivation in the final version.

Q1 Do the gains persist against a sampled-token RKL baseline with training–inference mismatch measured and properly controlled?

Thank you for the suggestion. As detailed in our response to W2, we added a training–inference mismatch comparison, which does not support mismatch reduction as the source of the gains.

Q2 Why do the default results differ between Tables 1/5 and Tables 7–9?

Thank you for checking. We have corrected the table transcription error: 77.03 is the AMC 23 score for TrOPD Clip in Table 5, while the correct score for TrOPD is 78.51.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.