acceptodds
Under review as a conference paper at ICLR 2027

ReTaCo: Residual-Target Control for On-Policy Distillation

Abstract

On-policy distillation trains a student with teacher feedback on student-generated prefixes, but communicating or storing full-vocabulary teacher distributions at every generated token is costly. Entropy-aware on-policy distillation (EOPD) supplements reverse Kullback-Leibler (KL) divergence with forward supervision over the teacher's top- tokens to help recover plausible tokens that the student underestimates. Renormalizing the retained teacher probabilities assigns all target mass to the selected tokens and none to the omitted vocabulary. We prove that the loss continues to push the student's selected mass toward one even after the relative probabilities within the selected set match the teacher's. Consequently, matching the teacher is not a stationary point whenever the omitted vocabulary has positive teacher probability. This finding motivates ReTaCo (Residual-Target Control for On-Policy Distillation), which combines a single-sample estimator whose expectation equals the full-vocabulary reverse KL with a forward target that represents the top- tokens individually and groups all remaining tokens into one residual symbol. For teacher selected mass , the residual target is , while the relative probabilities of the selected tokens remain unchanged. Thus, preserves teacher mass, and larger values explicitly allocate more target mass to selected tokens. At a fixed prefix, we prove that the population objective has a unique optimum, whose selected mass lies between and and increases monotonically with . At , the forward term also retains non-vanishing recovery gradients for underestimated selected tokens. Numerical optimization confirms the predicted optimal mass and conditional distributions. Extensive experiments demonstrate the superiority of ReTaCo, outperforming EOPD on most mathematics and code benchmarks.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
on-policy distillation, knowledge distillation, large language models, top-k supervision, KL divergence

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_P5uzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper shows that EOPD's renormalized top-k forward loss in on-policy distillation equals KL(q‖p) − log P, so it implicitly targets unit selected mass and teacher matching is not stationary. The proposed approach, ReTaCo, instead uses a (k+1)-way target that keeps the tail as one residual symbol with mass (1−β)(1−m), plus a single-sample reverse-KL estimator. Theorem 3.4 gives a unique fixed-prefix optimum with selected mass in [m, r_β], monotone in β. Three student–teacher pairs show gains over Sampled OPD and EOPD on math and code.

Critical strengths
  • The paper has a sound theoretical investigation. The mass decomposition isolates the implicit unit-mass target. A drop-in O(k) loss with an interpretable knob: β=0 preserves teacher mass, β=1 recovers EOPD's forward target.

  • The empirical results are promising across different model family. Table 4 and Fig. 3a match the predicted optimum to ~1e-8. I appreciate the authors acknowledge the mixed OOD results and the confounded EOPD comparison.

  • The paper is easy to follow.

Critical weaknesses
  • The conditional-only variant R + mC matches or beats full ReTaCo (TAMC 54.37 vs 52.56; MATH500 83.55 vs 83.78). Since mC is the within-S-normalized loss EOPD already supports, the paper's central contribution seems somewhat to lack strong empirical support.
  • The diagnosed effect seems not what training shows. At k=16, m≈0.998, and ReTaCo and EOPD selected masses differ by only 0.0014. Both students keep more tail mass than the teacher, opposite to the inflation predicted by Prop. 3.3, and ε-projection is active at 31.65% of positions. The theory seems not fully explain the gains.
  • 47–50% of ablation responses hit the 8,192-token cap (Table 9), and β=0.5 causes length degeneration, so Fig. 3b may measure termination failure, but probably not mass control.
Questions for authors
  • Can the authors report truncation rates for every table row and evaluate at a larger length budget
  • Can the authors measure actual communication savings versus full-vocabulary OPD
  • EOPD runs in a different stack (OpenRLHF, PPO clipping, entropy gate, different GPU placement). However, the paper reports a single-run results. I think it's more meaningful to report the multi-seed or multi-stack experiments.
Reviewer_pnUmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ReTaCo studies a probability-mass distortion in entropy-aware on-policy distillation. Renormalizing the teacher’s top-k probabilities implicitly assigns zero target mass to the omitted vocabulary and continues pushing the student’s selected mass toward one after its conditional top-k distribution already matches the teacher. ReTaCo instead represents the omitted vocabulary by one residual symbol and uses β to control how much residual mass is transferred to the selected tokens. The paper derives the fixed-prefix optimum, introduces a single-sample reverse-KL estimator that is exact in expectation, and evaluates three teacher-student pairs on mathematics, code, and OOD benchmarks. ReTaCo obtains higher point estimates than EOPD throughout the reported mathematics and code results.

Critical strengths
  1. Clean diagnosis and formulation: The decomposition Lren=DKL(qS∥pS)−log⁡PL_{\mathrm{ren}} = D_{\mathrm{KL}}(q^S \Vert p^S) - \log P clearly identifies the mass inflation induced by top-k renormalization. The residual-symbol construction separates conditional top-k matching from aggregate mass allocation.

  2. Useful theoretical characterization: The paper establishes a unique population optimum whose selected mass lies between the teacher mass and the β-dependent target, and shows that β=0 preserves teacher mass while retaining non-vanishing recovery gradients for underestimated selected tokens.

  3. Empirical breadth: Three teacher-student pairs are evaluated on five mathematics, two code, and three OOD benchmarks. The component and β ablations are informative.

Critical weaknesses
  1. Gain attribution: Table 2 provides a meaningful target-level control within the ReTaCo training recipe: β=0 outperforms the renormalized β=1 target on four of five Qwen3-1.7B mathematics benchmarks and ties on the fifth. However, Table 1 compares complete pipelines that also differ in reverse objective, entropy gating, and training framework. The complementary experiment—introducing the residual target into the original EOPD recipe—is absent. The evidence therefore supports a local benefit of the β=0 target, but does not yet establish that residual-target control explains the broader cross-model gains in Table 1.

  2. Role of mass control: The ablations do not show that the mass term is responsible for most of the downstream improvement. The conditional-only objective nearly matches full ReTaCo on four tasks and improves AMC from 52.56 to 54.37, while β=0 outperforms β=0.5 and β=1 on four tasks and ties on AIME25. Moreover, intermediate or complete residual trimming does not improve any of the five reported tasks. The paper convincingly diagnoses the implicit unit-mass target in EOPD, but the experiments currently support selected-token conditional recovery more strongly than residual-mass control. A broader conditional-only comparison is needed.

Questions for authors
  1. Can the authors evaluate the residual target within the original EOPD recipe, keeping the reverse objective, entropy gate, and training framework fixed? Ideally, this comparison would cover the main teacher-student pairs and clarify whether the broader gains in Table 1 persist when only the forward target is changed.

  2. How consistently does the explicit mass term improve over conditional recovery alone? Please report, if available, the conditional-only objective R+mCR+mC alongside full ReTaCo R+B0+mCR+B_0+mC for the main teacher-student pairs, and clarify which downstream gains can be specifically attributed to B0B_0.

Reviewer_t9HROpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes ReTaCo, an on-policy distillation (OPD) method. Its starting point is a theoretical analysis showing that EOPD's renormalization of the teacher's top-k probabilities pushes the student's total probability on the top-k set above the teacher's (Props. 3.1 and 3.3), whereas ReTaCo with β = 0 attains its optimum exactly at the teacher's value (Theorem 3.4). Building on EOPD, ReTaCo makes three main changes:

  1. The reverse term uses the single-sample k3+ estimator, whose expectation equals the full-vocabulary reverse KL.
  2. In the forward term, all tokens outside the top-k set are merged into a single term whose target probability is set to (1 − β)(1 − m), where m is the teacher's total probability on the top-k set.
  3. EOPD's entropy gate is removed.

The experiments mainly use small and large models from the Qwen and Gemma families as teacher–student pairs, and compare ReTaCo with Sampled OPD and EOPD on five mathematics, two code, and three out-of-distribution benchmarks.

Critical strengths
  1. The paper is clearly conceived, the method is promising, and the theory is solid; although the improvement is incremental, both the idea and the theory are sound.
  2. The theorems are fully proved, and Appendix D provides numerical verification in float64.
  3. The authors are candid about the limitations (Section 5).
Critical weaknesses

Concerns about the main contribution

β is one of ReTaCo's core components, yet the experiments test only β = 0, 0.5, and 1 (Table 2, Qwen3-1.7B, one epoch), and β = 0 is clearly better: it outperforms β = 0.5 and β = 1 on MATH500, OlympiadBench, AMC, and AIME24, while all three settings score 21.67 on AIME25. The authors' recommended default is likewise β = 0 (§4.1: "Default ReTaCo uses k = 16, α = 1, and β = 0"). §4.2 describes the benefit of trimming as "task-dependent", yet no entry in Table 2 favors trimming.

Given this, the paper is in essence very similar to prior work: at β = 0, the forward term is a plain forward KL over 17 classes (the top-16 tokens plus one tail term). Dasgupta et al. 2026 (arXiv 2602.20816), which the paper cites, already contains the same term, albeit in offline distillation. The "ghost token" of Sparse Logit Sampling (Anshumann et al., arXiv 2503.16870), which the paper does not cite, is the same representation.

Since β = 0 is clearly better, is it really meaningful to make β a tunable variable?

The theoretical motivation seems to lack empirical support

The mass inflation predicted by the theory (Prop. 3.3) is not observed in the training curves, and the residual mass that the method targets is very small on average. First, Prop. 3.3 predicts that EOPD's student top-k mass exceeds the teacher's (P* > m; "Even ideal reverse KL therefore leaves selected-mass inflation"), yet Fig. 2 and §4.2 report last-20-step means of 0.989900 for EOPD and 0.991252 for ReTaCo, both below the teacher's ≈0.9977 and differing by only 0.0014. Second, the teacher's average top-16 mass is ≈0.9977 in Fig. 2 and ≈0.99974 in Appendix E (Table 6), so the residual mass is only ≈0.23% or less.

Reliability of the evaluation

I have concerns about the reliability of the evaluation:

  1. Generation is truncated at an 8,192-token limit: in Table 9, 47.51%, 48.33%, and 49.98% of the responses of the three component variants hit this limit, whereas the official Qwen3-1.7B and Qwen3.5-2B model cards recommend an output length of 32,768 tokens.
  2. Out-of-distribution scores are even below chance. For example, on GPQA-Diamond (four choices, chance = 25%), all three Qwen3-1.7B scores (18.94, 12.44, 17.49) and two Gemma4-E2B scores (13.26, 13.64) are below 25%, whereas Qwen3-1.7B's official non-thinking score is 28.6.

The above weaknesses are my major concerns, so my current inclination is weak rejection. I look forward to the authors' response; if these issues can be resolved, I would be very glad to raise my score.

Questions for authors
  1. EOPD gives no reason for renormalization and no ablation of it; it states that it restricts to the teacher's top-k tokens so that the student does not have to learn from the teacher's low-probability tails. Could the authors discuss this rationale and explain why it does not apply to the aggregate residual mass?
  2. For all three models, math evaluation uses temperature 0.7, top-p 0.95, no top-k truncation, and an 8,192-token limit, which differs from the official recommendations (e.g., for Qwen3 in non-thinking mode: temperature 0.7, top-p 0.8, top-k 20, and presence penalty 1.5 in the technical report). Why did the authors choose different inference settings?
Reviewer_9PwHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper identifies a theoretical mass-inflation issue in Entropy-aware On-Policy Distillation (EOPD) caused by top-k renormalization, mathematically demonstrating that it creates an implicit unit-mass target. To address this, the authors propose ReTaCo, which introduces an explicit residual target controlled by a parameter \beta. While the theoretical formulation is elegant, a close examination of the empirical validation reveals significant issues: the main baselines confound the proposed method with different training frameworks, the ablation studies suggest that conditional probability matching (rather than the proposed mass control) drives the performance, and severe response truncation rates cast doubt on the evaluation's reliability.

Critical strengths

Clean Theoretical Diagnosis: The decomposition of the token-level divergence cleanly identifies the mass inflation induced by top-k renormalization, mathematically proving that it acts as an implicit unit-mass target. This directly and elegantly motivates the need for a better objective.

Empirical Breadth: The authors undertook a comprehensive evaluation across multiple model scales (Qwen3-1.7B, Qwen3.5-2B, Gemma4-E2B) and on rigorous mathematics (MATH500, AIME) and code (HumanEval+, MBPP+) benchmarks.

Critical weaknesses

Confounded Baselines and Gain Attribution: The main comparison in Table 1 evaluates completely different training pipelines. ReTaCo uses the verl framework with a direct k3+ estimator, while EOPD is implemented in OpenRLHF with a PPO-style surrogate objective and entropy gating. Because the underlying frameworks, reverse objectives, and gating mechanisms differ, it is impossible to attribute the performance gains in Table 1 strictly to the residual-target control.

Misaligned Empirical Support (Conditional vs. Mass Control): The ablations in Table 2 reveal that the conditional-only variant (R+mCR + mC) performs almost as well as—and on AMC, outperforms, the full ReTaCo objective (R+B0+mCR + B_0 + mC). This strongly suggests that recovering underestimated selected tokens (the conditional part) drives the downstream improvements, rather than the paper's central thesis of explicit residual mass control (B0B_0).

Redundancy of the \beta Parameter: The experiments consistently show that β=0\beta=0 (preserving teacher mass) is the optimal setting across almost all tasks. If explicit mass trimming (β>0\beta > 0) is rarely beneficial, introducing β\beta as a tunable hyperparameter lacks practical utility, making the forward term functionally very similar to standard offline forward KL over k+1 classes.

Evaluation Reliability (Truncation Rates): Table 9 shows that roughly 47% to 50% of the ablation responses hit the 8,192-token generation limit. This severe truncation likely skews the performance metrics and makes it difficult to trust the ablation comparisons, as the differences in accuracy may simply reflect termination failures rather than true reasoning capabilities.

Questions for authors

Can you evaluate the ReTaCo target strictly within the original EOPD recipe (keeping the OpenRLHF framework, PPO-style reverse objective, and entropy gate fixed) to properly isolate the true effect of your forward target?

Given that the conditional-only objective (R + mC) matches or exceeds full ReTaCo on tasks like AMC (54.37% vs 52.56%), how do you justify the necessity of the mass control component?

Since Table 2 demonstrates that \beta=0 consistently outperforms \beta=0.5 and \beta=1, what is the practical value of formulating \beta as a tunable parameter rather than simply fixing it to preserve teacher mass?

How do you explain the ~50% truncation rate in Table 9, and could you provide evaluations at a significantly larger token budget (e.g., 32k) to ensure the results are not artifacts of generation length limits?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.