acceptodds
Under review as a conference paper at ICLR 2027

Beyond Mode-Covering and Mode-Seeking: A Closer Look at KL Divergence in On-Policy Distillation

11 views

Abstract

On-policy distillation (OPD) is a de facto recipe to enhancing the reasoning capability of student models by matching the teacher's distribution over self-generated trajectories, where the choice of divergence is a critical design decision. Empirical guidance relies on geometric intuitions — framing forward KL as *mode-covering* and reverse KL as *mode-seeking* — offering little insight into how these objectives actually steer optimization. Moving beyond this dichotomy, we shift our focus to optimization dynamics. Specifically, we analyze the logit-level gradient fields that different divergences produce during training. And we show that no -divergence guarantees that every logit update moves probability toward the teacher due to the Softmax coupling effect. Instead, effective optimization relies on coordinate-level sign alignment with the student-teacher probability gap. Under this framework, we prove that forward KL is the unique sign-aligned -divergence, while reverse KL actively pushes over-allocated tokens in the wrong direction — a structural pathology that convex combinations of the two cannot resolve. To settle this, we introduce m-RKL, a minimal correction of RKL's gradient field that restores sign alignment while retaining RKL's beneficial tail damping. Extensive experiments on Qwen3 family models demonstrate that m-RKL outperforms competitive KL-based objectives on a suite of math reasoning benchmarks without degrading the general capabilities.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, On-Policy Distillation, KL Divergence, Optimization Dynamics

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_H2TXOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies gradient fields induced by different KL divergences; FKL is sign aligned but RKL is provably misaligned, which widens teacher-student distribution gap. They further show that this pitfall cannot be overcome by just linearly interpolating FKL and RKL and propose m-RKL which corrects the gradient fields.

Critical strengths
  • Going beyond mode-averaging vs mode-seeking, this paper studies gradient-based optimization, and fixed RKL objectives, which is the de facto standard in practical OPD methods. m-RKL is plug-and-play and a clear contribution.
  • The custom autograd operator code is provided and the experimental results validates m-RKL on math (ID) and IF/Code (OOD) domains with probability error decreasing during training.
Critical weaknesses
  • I find no major weaknesses in this paper, but I would like the authors to clarify the following points.
Questions for authors
  • Could the authors provide the statistics of the logit gradients when the signs of the teacher and student logits are opposite? (Fig.1) I’m interested in the norm of the gradients in the sign-reversal case.
  • Is there any computational overhead in m-RKL compared to RKL? If so, could the authors discuss FLOPs/complexity or report wall-clock training times?
3 comments

We sincerely appreciate your recognition and positive assessment of our work. We have added quantitative analyses for both questions. In particular, we separate statistics at the generation-step level from those at the token-coordinate level, since a generation step occurring sign reversal can have only a subset of sign-reversal coordinates in its vocabulary distribution.

Answer to Q1: Following the definition in Theorem 2, we measure sign reversal on over-allocated tokens for which qk>pkq_k>p_k but the RKL logit gradient is negative. We report statistics at training checkpoints 5, 10, 15, 20, and 25, using the sampled on-policy trajectories.

Generation-step-level statistics‾.\underline{\text{Generation-step-level statistics}}. Table 4 compares normal generation steps with steps containing at least one sign-reversal coordinate. At sign-reversal steps, the average full-vocabulary gradient norm ranges from 0.4220.422 to 0.5180.518, compared with only 0.013∼0.0150.013\sim0.015 at normal steps — approximately 28∼3728\sim37 times larger. The average loss is also substantially higher at sign-reversal steps (0.845∼1.0840.845\sim1.084, versus 0.011∼0.0170.011\sim0.017 at normal steps).

The teacher distribution is considerably more uncertain at these steps: average teacher entropy ranges from 0.7850.785 to 0.9040.904 at sign-reversal steps, compared with 0.041∼0.0710.041\sim0.071 at normal steps. Thus, sign reversal is empirically associated with high-uncertainty generation contexts and larger gradient responses, rather than occurring predominantly in nearly deterministic contexts.

Table 4: Generation-step level statistics (normal positions vs. sign-reversal positions)

Metrics\text{Metrics}Position Subset\text{Position Subset}Step-5\text{Step-5}Step-10\text{Step-10}Step-15\text{Step-15}Step-20\text{Step-20}Step-25\text{Step-25}
Average LossNormal0.0120.0120.0150.0170.011
Sign-Reversal1.0840.9860.9250.8920.845
Average Gradient NormNormal0.0140.0130.0130.0150.014
Sign-Reversal0.5180.4740.4370.4260.422
Average Teacher EntropyNormal0.0410.0460.0600.0710.061
Sign-Reversal0.7850.7980.8600.9040.851

Table 5: Token-coordinate-level statistics

Metrics\text{Metrics}Step-5\text{Step-5}Step-10\text{Step-10}Step-15\text{Step-15}Step-20\text{Step-20}Step-25\text{Step-25}
Gradient Norm over All Coordinates0.5180.4740.4370.4260.422
Gradient Norm over Sign-Reversal Coordinates0.0650.0560.0520.0520.048
Student Probability Mass over Sign-Reversal Coordinates0.2740.2720.2700.2670.273

Token-coordinate-level statistics‾.\underline{\text{Token-coordinate-level statistics}}. Table 5 further examines the vocabulary coordinates that actually violate SLSM within the sign-reversal steps. The gradient norm over these coordinates ranges from 0.0480.048 to 0.0650.065, while the corresponding full-vocabulary gradient norm ranges from 0.4220.422 to 0.5180.518. The reported sign-reversal-coordinate norm is therefore approximately 11∼1311\sim13 \\% of the full-vocabulary norm and remains larger than the average full-vocabulary norm at normal generation steps.

Importantly, the sign-reversal coordinates collectively carry 26.7∼27.426.7\sim27.4 \\% of the student probability mass, and this proportion remains stable across the five checkpoints. Although the gradient norms decrease moderately from Step 5 to Step 25, the probability mass assigned to the affected coordinates does not vanish. These results indicate that sign reversal is not confined to negligible-probability tail tokens or coordinates with vanishingly small gradients.

Answer to Q2: m-RKL adds only two per-position vocabulary reductions to compute S+S^+ and S−S^-, requiring O(V)\mathcal{O}(V) additional work. It requires no additional forward or backward pass and no teacher-side modification. In our implementation, total training time changes from 1973 s for RKL to 1997 s for m-RKL, corresponding to approximately 1.221.22 \\% overhead. Throughput changes from 7.15 to 7.08 samples/s, a decrease of approximately 0.980.98 \\%, while both methods report 2.175×10132.175\times10^{13} total FLOPs. We have included these measurements in the appendix to make the practical cost explicit.

We thank the reviewer again for these suggestions, which helped us provide a more complete empirical characterization of both the sign-reversal phenomenon and the computational cost of m-RKL.

Thank you very much for additional experiments on logit analysis and clarifications on computation cost. I am leaning slightly toward accept. But, I am not an expert in this area and I will maintain my positive rating (3).

We sincerely thank the reviewer for maintaining the positive assessment. We greatly appreciate the reviewer's time and constructive comments, which have helped improved our paper.

Reviewer_uRaCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

In reverse KL, the direction of the update in logit space can sometimes be not optimal. The proposed method, m-RKL, addresses this issue. The authors evaluate m-RKL through experiments with Qwens.

Critical strengths
  1. The experiments on Qwen show large improvements over the baselines.
  2. The difference between logit space and probability space in distillation is novel and interesting.
Critical weaknesses

The experimental setting is limited to Qwen + Math

The experiments are limited to mathematical reasoning. It is unclear whether the proposed method also improves performance on other reasoning tasks. Also, the experiments only use Qwen families. It remains unclear whether the findings generalize to other model families, such as Llama and Gemma.

Logit space vs Weight space

The theoretical analysis focuses only on the dynamics in logit space. It is unclear whether the same dynamics actually holds if we consider the weight update.

Questions for authors
  1. Beyond the arithmetic mean, we can consider more general ways of combining distributions, such as the geometric mean and α-mean (e.g., AMiD [1]). In Line 24, the authors claim that there is "a structural pathology that convex combinations of the two cannot resolve." Does this issue hold when using other types of mixtures, such as those considered in AMiD?
  2. In Line 49, the authors claim that "LLMs are optimized via gradient descent, which directly affects the logits rather than the probabilities." However, LLMs are trained by updating their weights, not their logits directly. Even if the gradients with respect to the logits are modified and then backpropagated, it is not clear whether the weight updates by the gradient will induce the intended dynamics in logit space. For example, in linear networks, we need to use natural gradient descent to ensure that the dynamics in logit space correspond to GD [2]. Could the authors clarify how the proposed logit-space analysis relates to the dynamics induced by weight updates? For example, Proposition 2 appears to guarantee descent only for the dynamics in logit space. Is it possible to establish a similar result for the dynamics induced by weight updates?
  3. In the experiments, you are using AdamW but in the theoretical claims, you are considering GD in logit spaces. How much do the theoretical guarantees remain valaid when using other optimizers such as AdamW or Muon?
  4. There are follow-up works to AKL and MiniLLM that improve the divergence for distillation, such as ABKD [3] and TAID [4]. Is there any reason why these methods were not included as baselines? I am concerned that the current baselines may not be sufficiently strong to establish the advantage of the proposed method.

[1] Shin et al. "AMiD: Knowledge Distillation for LLMs with α-mixture Assistant Distribution." ICLR 2026.
[2] Bernacchia et al. "Exact Natural Gradient in Deep Linear Networks and Its Application to the Nonlinear Case." NeurIPS 2018.
[3] Wang et al. "ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via α-β-Divergence." ICML 2025.
[4] Shing et al. "TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Models." ICLR 2025.

4 comments
AuthorsOpenReview ↗

Part I of the reply

We sincerely thank you for thoroughly reviewing our paper and providing constructive comments. We will try our best to address your concerns below.

Response to W1 (limited to Qwen + Math only): We agree that the original experiments alone did not establish generality across model families and training domains. We have therefore added two sets of experiments: OLMo distillation on the same mathematical-reasoning dataset used in our original study, and code-domain training followed by code-generation evaluation.

A new model family with the same mathematics training recipe‾.\underline{\text{A new model family with the same mathematics training recipe}}. We distill an Olmo-3-7B-Instruct-SFT student using the corresponding Olmo-3-7B-Instruct teacher on DAPO-Math-17K, the same mathematical-reasoning training dataset used in our original Qwen experiments. The training and evaluation protocols are kept consistent across objectives. As shown in Table 2, m-RKL achieves the best average accuracy of 48.8%, compared with 47.2% for RKL and 47.7% for AMiD, the strongest baseline by average performance. It also achieves the best or tied-best result on every reported benchmark. These results indicate that the benefit of m-RKL on mathematical-reasoning distillation is not specific to Qwen models.

Tabel 2: Performance of different distilled Olmo-3-7B-Instruct-SFT students on math reasoning benchmarks.

Method\text{Method}MATH-500\text{MATH-500}Minerva\text{Minerva}AIME’24\text{AIME’24}AIME’25\text{AIME’25}HMMT’25\text{HMMT’25}HMMT’26\text{HMMT’26}Average\text{Average}
Olmo-3-7B-Instruct (Teacher)91.956.351.341.325.214.246.7
Olmo-3-7B-Instruct-SFT (Student)67.840.46.710.02.11.921.5
FKL87.253.156.444.023.114.246.3
RKL87.652.757.244.325.615.947.2
SFKL87.852.858.644.424.516.547.4
SRKL87.652.759.0‾\underline{59.0}43.225.815.547.3
Sym-KL87.553.557.544.625.415.947.4
JS88.053.758.544.225.215.047.4
AKL88.054.1‾\underline{54.1}55.845.2‾\underline{45.2}24.415.047.1
TAID88.653.256.743.926.716.8‾\underline{16.8}47.6
ABKD88.353.255.044.325.015.346.9
AMiD88.652.858.844.326.415.047.7‾\underline{47.7}
m-RKL88.654.459.446.526.717.348.8

Code-domain training and evaluation‾.\underline{\text{Code-domain training and evaluation}}. To examine a non-mathematical training domain, we further train the Qwen3-1.7B student on code data (Eurus-2-RL-Data), using Qwen3-4B-Instruct-2507 as the teacher, and evaluate the resulting models on LiveCodeBench-v6. As shown in Table 3, m-RKL obtains 49.3%, compared with 46.5% for RKL, corresponding to a 2.8-point improvement. It also outperforms JS (48.0%), the second-best method, by 1.3 points. This experiment directly extends our evaluation beyond mathematics training rather than merely evaluating mathematics-distilled models on code tasks.

Tabel 3: Performance of different distilled Qwen3-1.7B students on code generation benchmark.

Method\text{Method}LCB-v6\text{LCB-v6}
Qwen3-4B-Instruct-2507 (Teacher)54.7
Qwen3-1.7B (Student)26.5
FKL38.1
RKL46.5
SFKL46.8
SRKL47.4
Sym-KL42.2
JS48.0‾\underline{48.0}
AKL44.5
AMiD43.0
ABKD47.3
TAID43.3
m-RKL49.3

Stronger baselines‾.\underline{\text{Stronger baselines}}. Both new experiments include TAID, ABKD, and AMiD, using their official implementations adapted to our OPD framework under the same training budget and evaluation protocol. On OLMo, their average accuracies are 47.6%, 46.9%, and 47.7%, respectively. On code-domain training, their LCB-v6 scores are 43.3%, 47.3%, and 43.0%, respectively. m-RKL retains the best overall result after including these recent divergence-based and target-mixture methods. We will add these experiments to Section 4 and revise the introduction and conclusion to state the evaluated model families and training domains explicitly.

AuthorsOpenReview ↗

Part II of the reply

Response to Q1 (analysis of other types of mixtures): Thank you for highlighting the distinction between interpolating divergence losses and constructing a mixed target distribution. Our statement that convex combinations cannot eliminate sign reversal concerns combinations of FKL and RKL losses, rather than assistant distributions constructed from the teacher and student. We will revise Line 24 to make this scope unambiguous.

For a stop-gradient target r=M(p,q)r=M(p,q), the logit-level analysis remains applicable with rr replacing pp. In particular, ∇zDKL(r∥q)=q−r,\nabla_zD_{\mathrm{KL}}(r \parallel q)=q-r, and m-RKL can analogously be defined to align with q−rq-r. For an arithmetic mixture r=(1−λ)p+λqr=(1-\lambda)p+\lambda q, q−r=(1−λ)(q−p),q-r=(1-\lambda)(q-p), so FKL also preserves sign alignment with the original teacher–student gap for λ<1\lambda<1. For geometric and general α\alpha-mixtures, alignment with q−rq-r does not automatically imply alignment with q−pq-p, and we therefore do not claim that every original result transfers unchanged.

We have additionally evaluated AMiD empirically. On OLMo family models, AMiD is the strongest baseline by average accuracy at 47.7%, while m-RKL obtains 48.8%. On LCB-v6 with Qwen3 family models, AMiD obtains 43.0%, compared with 49.3% for m-RKL. These experiments complement the theoretical clarification without treating AMiD as merely a convex interpolation of FKL and RKL losses.

Response to W2 & Q2 (logit updates vs. weight updates): We agree that Proposition 2 is a logit-space result and does not automatically cover full-model parameter updates or arbitrary optimizers. We have added the Jacobian-based clarification described in our response to Reviewer 46Z8 regarding W2‾\underline{\text{Reviewer 46Z8 regarding W2}} and narrowed the theorem accordingly.

In brief, a parameter update introduces the factor JJ⊤JJ^\top, so the first-order FKL change depends on (q−p)⊤JJ⊤G(z)(q-p)^\top JJ^\top G(z), rather than only on (q−p)⊤G(z)(q-p)^\top G(z). Consequently, the positivity established by Proposition 2 does not generally extend to arbitrary deep networks. A restricted parameter-space extension does hold for a final linear decode layer z=Whz=Wh with fixed features hh. When only WW is trained using standard GD, the induced update is z+=z−η∥h∥22G(z)z^+=z-\eta\|h\|_2^2G(z), so Proposition 2 applies with a positive effective step size. This argument does not cover joint training of earlier layers. We have narrowed the theorem and its surrounding claims accordingly; the convergence curves under full-model AdamW training are presented only as empirical evidence.

Response to Q3 (theoretical guarantees for AdamW or Muon): Our general guarantee applies to the immediate logit-space gradient direction. A limited extension holds for a positive diagonal preconditioner PP without momentum: because every coordinate of Gm-RKL(z)G_\text{m-RKL}(z) has the same sign as q−pq-p, giving (q−p)⊤PG>0.(q-p)^\top PG>0. However, this argument does not directly cover AdamW’s momentum and decoupled weight decay, nor does it cover Muon’s non-diagonal update transformation. It also does not eliminate the parameter-space Jacobian coupling discussed above. We therefore do not claim optimizer-independent guarantees. Our AdamW results demonstrate the practical behavior of m-RKL, while a corresponding analysis for Muon remains an open extension.

Response to Q4 (follow-up works to AKL): We agree that these methods are important for establishing the advantage of m-RKL relative to recent distillation objectives. We have added TAID, ABKD, and AMiD using their official implementations adapted to our OPD framework, with the same student initialization, teacher checkpoint, training budget, and evaluation protocol across methods.

On OLMo, TAID, ABKD, and AMiD obtain average accuracies of 47.6%, 46.9%, and 47.7%, respectively, compared with 48.8% for m-RKL. On LCB-v6, they obtain 43.3%, 47.3%, and 43.0%, respectively, compared with 49.3% for m-RKL. Thus, m-RKL retains its advantage after including both recent divergence-based methods and target-mixture approaches. We will add the related-work and revise experimental sections to discuss these methods and their relationship to our analysis.

Thank you again. We hope these clarifications address your concerns and would appreciate reconsideration of the score if you find them sufficient.

Thank you for the detailed reply and the additional experiments.

My remaining concern is the difference between logit updates and weight updates. This feels like a very important weakness. Also, in the current manuscript, it is not clear to me how it was corrected. (If the fix above is already reflected in the current PDF, let me know.) In the reply, the notation is so rough and I still have doubts about its mathematical rigor.

For now, I will keep the score.

AuthorsOpenReview ↗

Follow-up on logit updates vs. weight updates

Thank you for keeping this discussion open and for pinpointing three concerns: whether the correction is reflected in the manuscript, the rigor of the derivation, and the status of the logit-space guarantee under weight updates. All are now addressed in the revised PDF, with exact locations below.

1.Where each change appears in the revised manuscript (All highlighted in blue)‾.1. \underline{\text{Where each change appears in the revised manuscript (All highlighted in blue)}}.

  • Proposition 2 (【Page 6, Lines 307–312】) is now explicitly scoped to direct logit-space updates; no parameter-space or batch-level descent is claimed.
  • New Section 3.5, Scope of the Analysis: From Logit Signals to Weight Updates (【Page 6, Lines 314–342】), contains the full derivation (Eqs. (19)–(20)) connecting logit-space fields to parameter updates.
  • New Proposition 3 (final-linear-layer extension) at 【Page 7, Lines 328–331】, proof in Appendix B.8.
  • New Section 3.6 with Figure 2 (【Page 7】): controlled probes of the realized dynamics.
  • Figure 4 is now presented strictly as empirical evidence

2.The derivation, as it appears in the manuscript‾.2. \underline{\text{The derivation, as it appears in the manuscript}}.

Fix a context and consider a single token position, write the logits as z=h(θ)z = h(\theta), where θ\theta is the student's parameters. We have the Jacobian Jθ(z)=∂z/∂θJ_{\theta}(z) = \partial z / \partial \theta representing the gradients of zz w.r.t. θ\theta. Our custom backward operator injects Gm-RKL(z)G_\text{m-RKL}(z) at the logit level, so backpropagation yields the parameter gradient Jθ(z)⊤Gm-RKL(z)J_{\theta}(z)^\top G_\text{m-RKL}(z) and the update θ←θ−ηJθ(z)⊤Gm-RKL(z)\theta \leftarrow \theta - \eta J_{\theta}(z)^\top G_\text{m-RKL}(z). Since ∇zDFKL(p∥q)=q−p\nabla_{z} D_{\mathrm{FKL}}(p \parallel q) = q - p, a first-order expansion gives Eq. (19):

ΔDFKL≈−η(q−p)⊤Jθ(z)Jθ(z)⊤Gm-RKL(z).\Delta D_\text{FKL} ≈ -\eta (q - p)^\top J_{\theta}(z) J_{\theta}(z)^\top G_\text{m-RKL}(z).

Three regimes follow precisely:

  • Direct logit updates (Jθ(z)=IJ_{\theta}(z) = I): Eq. (19) reduces to −η(q−p)⊤Gm-RKL(z)<0-\eta (q - p)^\top G_\text{m-RKL}(z) < 0 by Proposition 2 — a strict guarantee.
  • General deep networks: Jθ(z)Jθ(z)⊤J_{\theta}(z) J_{\theta}(z)^\top couples logit coordinates, so Proposition 2 does not imply negativity of ΔDFKL\Delta D_\text{FKL}. We claim nothing here, and say so explicitly.
  • Final linear decode layer (Proposition 3): with z=Whz = Wh, fixed hh, and only WW trained, the update induces z←z−η‖h‖2Gm-RKL(z)z \leftarrow z − η‖h‖²G_\text{m-RKL}(z) — an exact logit-space update with positive step size, to which Proposition 2 applies directly.

3.New empirical probe (Section 3.6, Figure 2)‾.3. \underline{\text{New empirical probe (Section 3.6, Figure 2)}}.

We measured the realized logit change Δzt\Delta z_t under controlled updates using gradients from exactly kk positions, updating only the final decode layer:

kkm-RKL\text{m-RKL}FKL (its own field q−p)\text{FKL}\ (\text{its own field}\ q − p)
10.99890.9997
100.54750.5486
1000.25100.2656
2048 (full-trajectory)0.08110.0967

(i) The analysis is exact where it claims to be. In the single-position, final-layer regime of Proposition 3, the realized logit update aligns with the theory-prescribed logit field G(z)G(z) up to numerical precision.
(ii) The attenuation is a property of the network Jacobian, not of our correction. Alignment decays smoothly in kk — exactly the cross-position mixing of Eq. (20) — and FKL's own field shows statistically identical alignment at every granularity. Since this filtering is objective-independent and all methods are trained under identical conditions, the consistent gains of m-RKL remain attributable to the per-position field, the only quantity that objective design controls.
(iii) Batch-level effectiveness is empirical. Despite mixing, both FKL and RKL decrease steadily throughout full-model AdamW training with our corrected field (Figure 4).

4.What we do not claim‾.4. \underline{\text{What we do not claim}}.

Our guarantees cover direct logit updates (Proposition 2) and the final-linear-layer regime (Proposition 3). They do not cover AdamW's momentum and decoupled weight decay, Muon's non-diagonal updates, or batch-level descent under position mixing; Section 3.5 and 3.6 state this explicitly, and batch-level guarantees for non-conservative fields under preconditioned optimizers are a concrete open problem.

We hope the revised statements, the complete derivation, and the new probe experiments resolve the remaining concern. If any step still appears insufficiently justified, we would be glad to know which one; otherwise we would be grateful for reconsideration of the score.

Reviewer_46Z8OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper points out that previous research on on-policy distillation (OPD) has focused solely on the resulting distribution shapes of forward KL (FKL) and reverse KL (RKL), without considering the impact of these divergences on optimization dynamics. This paper analyzes the characteristics of the logit-level and distribution-level gradients of FKL and RKL based on strict local sign monotonicity (SLSM). Through this analysis, this study demonstrates that FKL is the only divergence in the f-divergence family that satisfies SLSM, and proposes m-RKL, the gradient update method that satisfies SLSM by slightly modifying the logit gradient of RKL. This work experimentally validates the effectiveness of proposed method by the comparisons with various KL-based divergence methods.

Critical strengths
  1. To the best of my knowledge, this is the first work to reframe the choice of divergence in OPD as a question about the logit gradient field. SLSM-based analysis is novel and it is impressive that FKL has been theoretically shown to be the only divergence in the f-divergence family that satisfies the SLSM.

  2. The proposed method is simple and practically appealing: it is a drop-in replacement for the reverse-KL gradient, requires no additional forward or backward passes and no change on the teacher side.

Critical weaknesses
  1. This manuscript utilizes SLSM as the criterion to investigate the properties of KL divergences and propose the new method. However, I cannot find the sufficient evidences that coordinate-wise sign alignment in logit space is the core desideratum in OPD. The authors already demonstrate that sign alignment is neither necessary nor sufficient for the student’s probabilities to move toward the teacher (Line 190-192). Also, the gradient of RKL is simply the correct gradient to minimize the RKL objective. Labeling it a wrong direction or a structural pathology presupposes that the FKL coordinate-wise directions is what distillation should aim at. However, in my opinion, this claim is sufficiently justified. The misalignment of sign on some coordinates does not by itself prevent the overall divergence from decreasing.

  2. The descent direction of m-RKL is verified for direct updates in the logit space, while the actual training updates the model parameters. Since the corrected gradient field of the proposed method may not be the gradient of scalar loss function, a positive inner product in logit space does not still imply descent in terms of model parameters, tokens, sequences and instances in batch.

  3. The paper still utilizes the mode-covering and mode-seeking dichotomy it sets out to move beyond, without ever measuring it, and leaves unverified where m-RKL actually sits in distribution space.

Questions for authors
  1. I’m curious why we need to look beyond the shape of the resulting distribution and consider the underlying optimization process as well, especially in OPD scenarios.

  2. I’m wondering if there is any distribution-level analysis regarding the shape of the resulting distribution obtained using the proposed m-RKL methodology. Specifically, I’m interested in the trade-off between mode-covering and mode-seeking for models trained using FKL, RKL, and m-RKL.

  3. Recent distillation methods increasingly construct the target by mixing the teacher and student distributions rather than the losses [1,2,3]. I am curious whether the logit-level gradient analysis, which is the main claim of this paper, remains informative in this setting.

[1] DistiLLM: Towards Streamlined Distillation for Large Language Models

[2] TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Models

[3] AMiD: Knowledge Distillation for LLMs with α-mixture Assistant Distribution

5 comments
AuthorsOpenReview ↗

First 1/2 of the reply

We appreciate your careful reading and valuable feedback. We have seriously considered your comments and have addressed them below.

Response to W1: Thank you for identifying an important overstatement in our original presentation. We fully agree that the standard RKL gradient is the correct gradient for minimizing the RKL objective, and that a coordinate-wise sign mismatch does not prevent the overall RKL from decreasing. We also agree that SLSM is neither necessary nor sufficient for probability-space contraction or downstream performance.

Our intended claim is narrower: SLSM characterizes whether each raw logit update is locally aligned with the corresponding teacher–student probability gap. Under this criterion, RKL can increase the logits of tokens for which qk>pkq_k>p_k, whereas FKL and m-RKL cannot. We do not interpret this as evidence that RKL incorrectly optimizes its own objective. Instead, SLSM isolates a measurable local property whose practical relevance must be examined empirically.

Figure 2 in the manuscript provides such an examination: sign reversal occurs in over 30% of generation steps; although only approximately 5% of vocabulary tokens are affected at these steps, they carry over a quarter of the probability mass. On the affected tokens, RKL updates enlarge the measured probability gap, whereas FKL and m-RKL reduce it. We have clarified that this observation concerns the affected coordinates and does not establish a universal causal relationship between SLSM and task performance. In the final revision, we will replace “wrong direction” and “structural pathology” with the more precise terms “SLSM violation” and “sign-reversal phenomenon,” and will also remove statements implying that RKL is incorrectly optimizing its own objective.

Response to W2: We agree that Proposition 2 does not directly guarantee a decrease in FKL when the full model parameters are updated. Our original text did not sufficiently distinguish direct logit updates from parameter updates in a neural network.

To clarify this distinction, consider a fixed context and write the student logits as z=h(θ)z=h(\theta), with Jacobian J=∂z/∂θJ=\partial z/\partial\theta. Our custom backward operator supplies the m-RKL vector field G(z)G(z) at the logit level, so backpropagation gives the parameter gradient J⊤G(z)J^\top G(z). A standard gradient update is therefore θ+=θ−ηJ⊤G(z).\theta^{+}=\theta-\eta J^\top G(z). For FKL, ∇zDFKL=q−p\nabla_zD_{\mathrm{FKL}}=q-p, and hence its parameter gradient is J⊤(q−p)J^\top(q-p). A first-order expansion then gives the change in FKL after this update:

DFKL(θ+)−DFKL(θ)≈−η(q−p)⊤JJ⊤G(z).D_{\mathrm{FKL}}(\theta^{+})-D_{\mathrm{FKL}}(\theta) \approx -\eta(q-p)^\top JJ^\top G(z).

Proposition 2 proves that (q−p)⊤G(z)>0(q-p)^\top G(z)>0. If we update the logits directly, then J=IJ=I, and the expression above reduces to

ΔDFKL≈−η(q−p)⊤G(z)<0,\Delta D_{\mathrm{FKL}}\approx-\eta(q-p)^\top G(z)<0,

which is exactly the guarantee provided by Proposition 2. For a general neural network, however, JJ⊤JJ^\top couples and rescales different logit coordinates. Consequently, positivity of (q−p)⊤G(z)(q-p)^\top G(z) does not, in general, imply positivity of (q−p)⊤JJ⊤G(z)(q-p)^\top JJ^\top G(z). The same qualification applies when updates are aggregated across tokens, sequences, and examples in a batch. We have therefore narrowed Proposition 2 explicitly to direct logit-space dynamics and removed any implication that it constitutes a general parameter-space descent theorem.

Nevertheless, a restricted parameter-space extension is available for the final linear decode layer. Suppose z=Whz=Wh, where the features hh are fixed and only WW is trained. The custom backward operator induces the parameter update W+=W−ηG(z)h⊤W^+=W-\eta G(z)h^\top, which in turn gives z+=z−η∥h∥22G(z).z^+=z-\eta\|h\|_2^2G(z). Thus, in this special case, the parameter update is exactly a logit-space update with the positive step size η∥h∥22\eta\|h\|_2^2, and Proposition 2 applies directly for sufficiently small η\eta. If a trainable bias is included, the positive factor becomes ∥h∥22+1\|h\|_2^2+1. This argument does not extend to joint training of earlier layers, where hh also changes, nor does it directly cover AdamW or other preconditioned optimizers. We include it only to delineate precisely when the logit-space guarantee has a parameter-space counterpart.

AuthorsOpenReview ↗

Second 1/2 of the reply

Response to W3 and Q2: We agree that our original manuscript invoked mode-covering/mode-seeking terminology without directly measuring the resulting distributions. We have therefore added a distributional analysis of the final FKL-, RKL-, and m-RKL-trained students on matched base-student trajectories from 200 sampled MATH-500 prompts (covering all difficulty levels). At each generation step, all students and the teacher are evaluated on exactly the same context, so the comparison is not confounded by differences in trajectory content or length.

We further recognize that aggregate token-level averages can be dominated by near-deterministic positions where the teacher and students share the same confident top-1 prediction, we additionally analyze teacher-defined uncertainty subsets, including the highest-entropy decile and the lowest teacher top-1-probability decile. These subsets are selected exclusively from teacher statistics, independently of student predictions.

Table 1 Quantitative comparison of resulting distributions trained by FKL, RKL, and m-RKL

Teacher-defined Token Subset\text{Teacher-defined Token Subset}Objective\text{Objective}Hstu\mathcal{H}^\text{stu}Htea\mathcal{H}^\text{tea}Masstop-1stu\text{Mass}_\text{top-1}^\text{stu}Masstop-1tea\text{Mass}_\text{top-1}^\text{tea}DisagreeRatetop-1\text{DisagreeRate}_\text{top-1}TVD\text{TVD}Precisiontop-5stu\text{Precision}_\text{top-5}^\text{stu}Recalltop-20tea\text{Recall}_\text{top-20}^\text{tea}
FKL0.4200.1340.8690.9510.1030.1330.9830.986
All tokensRKL0.1230.1340.9540.9510.0890.0890.9850.998
m-RKL0.3030.1340.8990.9510.0970.1110.9840.993
FKL1.6701.0070.4830.5790.5500.4500.8620.932
Top 10% Htea\mathcal{H}^\text{tea}RKL0.6451.0070.7580.5790.5320.4720.8810.989
m-RKL1.3161.0070.5600.5790.5450.4320.8690.962
FKL1.5270.9720.5140.5780.5410.4190.8810.941
Bottom 10% Masstop-1tea\text{Mass}_\text{top-1}^\text{tea}RKL0.6010.9720.7700.5780.5230.4550.8960.990
m-RKL1.2070.9720.5860.5780.5340.4060.8870.967

The results are mixed and instructive. Over all positions, RKL achieves the strongest aggregate distributional measurements, while m-RKL generally lies between FKL and RKL. In the highest-entropy subset, however, m-RKL yields the smallest student–teacher entropy gap, the smallest top-1 probability-mass gap, and the lowest total variation distance (TVD). RKL remains numerically slightly better in teacher top-20 recall, student top-5 precision, and top-1 disagreement. Thus, m-RKL improves certain aspects of overall probability-mass matching in high-entropy contexts, but it does not uniformly improve candidate coverage, candidate precision, or argmax agreement.

Importantly, RKL’s stronger aggregate distributional measurements do not translate into stronger downstream reasoning performance. This dissociation supports our motivation for studying optimization dynamics in addition to final distribution geometry: neither average distributional similarity nor the mode-covering/mode-seeking dichotomy is sufficient to explain the downstream differences between objectives.

Answer to Q3: Thank you for highlighting the distinction between interpolating divergence losses and constructing a mixed target distribution. Our original impossibility statement concerns convex combinations of FKL and RKL losses, rather than target-distribution mixtures such as DistiLLM, TAID, and AMiD. We have made this scope explicit.

For a stop-gradient target r=M(p,q)r=M(p, q), the logit-level analysis remains applicable with rr replacing pp. In particular, FKL gives ∇zDKL(r∥q)=q−r\nabla_zD_{\mathrm{KL}}(r\parallel q)=q-r, and m-RKL can be defined relative to q−rq-r. For an arithmetic mixture r=(1−λ)p+λqr=(1-\lambda)p+\lambda q, we have q−r=(1−λ)(q−p)q-r=(1-\lambda)(q-p), so FKL also preserves alignment with the original teacher–student gap for λ<1\lambda<1. For geometric and general α\alpha-mixtures, however, alignment with q−rq-r does not automatically imply alignment with q−pq-p, and we do not claim a universal extension.

Thank you again. We hope the above clarifications address your concerns and would appreciate reconsideration of the score if you find them sufficient.

Thank you for providing the detailed clarification and additional experimental results. Although some concerns are resolved, two concerns remain.

First, I still cannot be convinced why do we need to consider the shape of resulting distribution and optimization process in OPD. This is my Q1 in the initial review, but I cannot find the corresponding authors’ response.

Also, the authors’ response clarify that the intention of the original claim is more narrow. Then, I am still curious why SLSM is important and why do we construct new method to satisfy it.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.