acceptodds
Under review as a conference paper at ICLR 2027

Divergence controls entropy in distillation

Abstract

Distillation has become a core primitive of large language model training, yet its properties are not yet well understood. We take an entropic perspective on distillation, studying how the entropy of the student depends on the data and the choice of divergence function that define the distillation objective. We prove that forward KL inflates the entropy of the student above that of the teacher. Since cross-entropy training is a special case, this yields an identity that we verify quantitatively in pretraining and supervised fine-tuning. Other divergences come with no such guarantee: reverse KL deflates entropy until the task gets too hard, and in between entropy smoothly changes early in training. The divergence therefore acts as an implicit entropy regularizer, whose role is clearest in self-distillation: as conditioning on privileged information deflates entropy, the divergence hyperparameters that work best are those that compensate for it.

Primary area:
foundation or frontier models, including LLMs
Keywords:
distillation, on-policy distillation, supervised fine-tuning, entropy, toy model

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_kCK8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper disentangles the choice of sampling distribution from the choice of divergence in LLM distillation, allowing these two components to be selected independently. This paper demonstrates that the choice of divergence implicitly controls the entropy of the student model and has a greater impact on entropy than the choice between on-policy and off-policy sampling. Specifically, forward KL increases the student's average entropy relative to the teacher's, whereas reverse KL generally decreases entropy but can also increase it when the distillation task becomes sufficiently challenging.

Critical strengths
  1. The paper is well written, and the figures are clear and easy to understand.
  2. By disentangling the sampling distribution from the divergence, which have often been coupled in previous approaches, the paper provides a clearer understanding of the mechanisms underlying LLM distillation.
Critical weaknesses

Variety of experiments

Most experiments focus on measuring entropy, while its effects on loss and downstream accuracy are not sufficiently investigated through experiments, except for the results presented in Figure 10. Moreover, the standard distillation experiments are limited to mathematical reasoning tasks using Qwen models. Other tasks and model families are not explored, except in the self-distillation experiments.

Presentation of the novelty

The novelty of the theoretical contributions is not sufficiently clear. Several of the main results are derived from classical results in information geometry, making it difficult to distinguish genuinely new theoretical findings from applications of existing results.

Questions for authors
  1. Some of the theoretical results appear to follow from well-established properties in information geometry. To better evaluate the novelty of this paper, I would appreciate a clearer distinction between new theoretical results and applications of existing results to LLM distillation. For example, Eq. 4~6 is based on a classical result from information geometry, and the local equivalence of divergences also appears to be well known. Does the paper establish any fundamentally new mathematical results, or does its main theoretical contribution lie in applying existing results to LLM distillation?
  2. The phase transitions observed in Figure 2 are interesting. Can these transitions be explained theoretically or these are purely the experimental results?
  3. What are the main practical suggestions arising from the proposed entropy analysis? Does the analysis offer practical suggestions on selecting divergences to improve training loss or downstream performance? While the paper establishes a clear relationship between divergence choice and entropy, it remains unclear how these findings translate into improvements in task performance.

minor

  1. How sensitive are the results to the choice of optimizer? Would the authors expect similar trends with optimizers other than AdamW, such as Muon?
  2. The analysis in Appendix B.5 appears to contain some new derivations. However, they rely on asymptotic approximations expressed using big-O notation, and for me, they seem not mathematically rigorous. For example, eq. 66 establishes an equality in expectation. Should the left-hand side of eq. 67 therefore also be expressed in terms of an expectation? Furthermore, the authors state that "the numbers of contexts and tokens are large enough". It is unclear how large these quantities need to be and whether a joint limit of β,n\beta, n and dd is being considered.
3 comments

We thank the reviewer for their thoughtful review, and for finding the paper well written and the decoupling of sampling and divergence useful. We address their concerns below, and hope that they will reconsider their score accordingly.

1. Novelty of the theoretical contributions

As detailed in the global answer (1.a), except for the derivation of App. B.5, our theoretical results apply known results from information geometry, and we now state it explicitly in the contributions. We see our theoretical contribution as mostly conceptual: finding a toy model of distillation that is simple enough for these results to apply directly, and rich enough to reproduce phenomena observed when distilling LLMs (Sec. 4), now also when the student generates its own contexts (global answer 2.a and answer to qw5G). Without such a model and the simplification work that comes with it, applying these classical results to LLM distillation is harder.

2. Can the phase transitions of Fig. 2 be explained theoretically?

Only partially. We tried several approaches, without much success. Our best characterization is the linearization around the uniform distribution of App. B.5: in the diffuse, high-load limit, every divergence of the generalized Jensen–Shannon family reduces to the same quadratic objective, which explains why reverse KL ends up inflating entropy like forward KL when the task is hard, and why this inflated regime looks the same for every divergence (Fig. 7). It therefore shows that there is a transition (given that reverse KL deflates entropy at smaller nn), but understanding where it happens is an open question; if such sharp transitions appear in LLM distillation, it would be very interesting to revisit this question.

3. Practical suggestions, loss and downstream performance

See global answer 1.b for the practical takeaways. We agree that the right entropy is not sufficient for good performance: a student can have the right entropy and still put its mass on the wrong tokens. It is however necessary, in the sense that entropy collapse and entropy blow-up are both problems.

More broadly, we see entropy as an observable that is both practically relevant and amenable to theory, and thus as a good place to start building a theory of distillation. It is a first step of this endeavor, not its end point.

Regarding the variety of experiments, we ran the equivalent of Figure 5 on OLMo 2, and are currently extending our GSM8K self-distillation results to OLMo too.

4. Sensitivity to the optimizer

In the toy model, we train until stationarity. The forward KL identity (Eq. 6) holds at any stationary point, whichever optimizer reaches it, and we reasonably expect the trends of Sec. 3 to be the same with other optimizers such as Muon. The optimizer could still matter where several stationary points coexist, for instance near the transitions of Fig. 2, as it can affect which one is reached. For LLMs, which are not trained to stationarity, we do not know, as the optimizer shapes the whole trajectory. Testing Muon for distillation at scale would be an interesting future work direction.

5. Rigor of the derivations of App. B.5

The reviewer has a point, and we revised the derivation (App. B.5). Eq. 66 is indeed an equality in expectation over the teacher logits, and so is now the left-hand side of Eq. 67, which then holds up to O(β3)O(\beta^3) without any assumption on the number of contexts; the asymptotic notation refers to the diffuse limit β→0\beta \to 0, all else fixed. The assumption that "the numbers of contexts and tokens are large enough" only serves to drop the expectation, and we now make it precise: the realized entropy gap concentrates around its expectation, with relative fluctuations of order (n−h)−1/2(n - h)^{-1/2}, in the limit of many contexts n→∞n \to \infty at fixed ratio h/nh / n and fixed vocabulary size dd. We also corrected the proof, which uses that the teacher logits are independent across contexts with zero mean, not that all their entries are independent, which centered logits cannot be. The result in itself is unchanged.

Thank you for the detailed responses, additional experiments, and revisions to the appendix.

I understand that the authors regard the toy model as a conceptual contribution. However, I think the value of a simplified model in ML generally lies in what it helps us understand or prove mathematically or theoretically. If most of the theoretical results are known results from information geometry, the toy models seem to mainly reproduce observed phenomena rather than explain why they occur or when they occur.

I find the approach interesting, but I am not convinced that reproducing these phenomena in small toy models, without some theoretical results, is enough to constitute a substantial contribution. (For example, to the best of my knowledge, papers that introduce toy models such as linear networks or logistic regression either construct closed-form dynamics or prove theorems using some kind of bound.)

For now, I will keep my score unchanged and defer to the other reviewers' judgment.

We thank the reviewer for engaging with our rebuttal.

We would like to offer a complementary view on the role of toy models in machine learning. Toy models are valuable for proving careful mathematical results, but we believe their usefulness goes further.

  1. They make extensive hypothesis testing possible. Toy models are many orders of magnitude cheaper to run than the systems they abstract. This lets us test hypotheses systematically, which would be infeasible at scale. They also surface behaviors that motivate further theoretical analysis, even when that analysis is not yet tractable. An example in our work is the phase transition under reverse KL, for which we do not yet have an analytical account.
  2. They show which components are necessary for a behavior and which are not. Building the simplest model that still reproduces a phenomenon tells us what is needed for it. Anything stripped away without losing the behavior is not required, and anything whose removal breaks it is. We show that a simple MLP captures the entropy dynamics of distillation. So neither scale nor sequential structure is necessary for this behavior (with stop-gradients in the on-policy setting). Like any model in science, ours will break down when pushed past what it can capture. Those failures are informative too, because they point to the additional ingredients other behaviors require.
Reviewer_34j2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The work studies how the divergence used for language-model distillation shapes student entropy, separately from the distribution used to generate training prefixes. It derives an entropy–cross-entropy identity at forward-KL stationary points under matched context weighting, and uses a shared linear-softmax model to investigate reverse KL, generalized Jensen–Shannon divergences, and top-k restrictions.

Critical strengths

The forward-KL identity gives a simple, quantitative connection between an optimization condition and a routinely logged observable. Moreover, the toy model experiment provides a concrete categorical setting in which representation constraints can be varied.

Critical weaknesses
  1. The entropy definition and theoretical guarantee use inconsistent averaging conventions. Equation (2) averages length-normalized entropy across sequences, whereas Equation (1) and Appendix B.3 use token-summed weights. For variable-length completions, the latter does not imply Equation (6) under the former definition. Appendix C.3 also acknowledges that the two conventions can yield opposite entropy trends. The identity remains valid under consistent weighting, but the theorem, metric, and empirical interpretation need reconciliation.
  2. Although the paper includes multiple model families, cross-model support is uneven across its empirical claims. The divergence vs. sampling comparison uses only a Qwen3-0.6B-Base student, and varying teacher sizes does not establish robustness across students (e.g., other model family). Self-distillation uses OLMo on Chemistry and Qwen on GSM8K, changing model and task together. These experiments provide useful evidence within the tested settings, but leave the generality of several empirical conclusions insufficiently established.
Questions for authors
  1. Which averaging convention is intended in Equation (6) and used in each experiment? On the same frozen prefixes, do the main entropy comparisons retain their qualitative ordering under both pooled-token and per-sequence averaging?

  2. Is there evidence that the relative effects of divergence and sampling persist beyond the Qwen3-0.6B-Base student, under otherwise comparable settings?

  3. How do the empirical insights differ from EOPD [1], which already studies reverse-KL diversity loss and introduces forward KL to preserve diversity during on-policy distillation?

[1] Entropy-Aware On-Policy Distillation of Language Models, arxiv: 2603.07079

1 comment

Response to 34j2

We thank the reviewer for their feedback on our work. As the weaknesses mentioned by the reviewer are fully addressed below, we hope that the reviewer will update their score to better reflect the

1. Effect of entropy averaging convention on Figure 5 results

The equations were indeed inconsistent, and we thank the reviewer for pointing it out. We now average over tokens throughout and have updated the paper accordingly (global answer 1.c). All terms of Eq. 6 are token-weighted, and the identity then holds exactly for completions of variable length; with a loss normalized by the length of each completion, it would instead hold for the per-sequence average (App. B.3). The results of Figure 5 are robust to the averaging convention: with per-sequence averaging, on both frozen teacher completions and student completions, forward KL still ends above reverse KL for every teacher and sampling distribution, and the effect of the divergence remains two to four times larger than that of the sampling distribution (new Figure 13). This holds for both seeds.

Regarding the sentence of App. C.3 on the two conventions yielding opposite trends: it referred to the change in entropy since the start of training, on the student's own completions. Our claims, however, are about relative values and these are unaffected by the convention. On the same frozen teacher completions, the four objectives keep the same ordering (on-policy forward KL > off-policy forward KL > on-policy reverse KL > off-policy reverse KL) for every teacher under both conventions. As this sentence was confusing, we removed it.

2. Extension to other students

We repeat the grid of Fig. 5 with an OLMo 2 student and OLMo 2 teachers, distilling the 1B, 7B and 13B instruct models into the 1B base model (global answer 2.c). The results remain the same.

3. Difference with EOPD (Jin et al., 2026)

We thank the reviewer for the pointer, and now cite this work in Sec. 4.2 (global answer 1.d). Our findings are consistent with EOPD on the role of the divergence and complementary: we additionally study the role of sampling data, provide some theoretical insights on the role of divergence, and consider this question in a larger set of setups.

Reviewer_qw5GOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors discuss the consequences of design choices in model distillation with respect to the entropy of teacher and student distributions. Importantly, they untangle the selection of sampling distribution from the choice of divergence by highlighting a token-level objective. They show that, while forward KL is likely to inflate the entropy of the student with respect to the teacher, reverse KL may deflate it. Furthermore, the paper studies the interpolation in the form of a generalized JS divergence and considers top-k methods to both steer and render the objective more resource-light. Their reasoning is empirically supported by experiments with a comprehensive toy model as well as a selection of modern large language models.

Critical strengths

Contribution to understanding distillation

Distillation is a heavily used technique but far from perfectly understood. The paper studies the process from a viewpoint of entropy. The authors both theoretically and empirically show the influence of design choices (on vs off-policy, forward vs reverse KL vs JS) on the learned student distribution. This serves as a guide for anyone interested in distilling a model, informing which sampling strategies and which divergences to use to render the learned student distribution such as is sought after. The paper thereby supplies the community with important tools for a sophisticated distillation process. While the authors consider the case of LLMs, it may be, that their approach is even more broadly applicable.

Clear and stringent argumentation

Besides being clearly written, the submission is very cogent. All claims are thoroughly discussed and substantiated by sound reasoning and empirical data. This makes it not only easy to follow along but also to understand the paper's objective clearly, thereby facilitating other researchers to build up on its findings.

Critical weaknesses

Validity in actual implementations

It seems to me, that the transferability to widely-in-use LLMs is questionable:

  • "The toy model has no sequence modelling component." The authors argue (l. 166-169), that this is not much of a problem with the distillation objective mainly aiming at next-token prediction. I would appreciate, if the authors could explain why they deem the sequence nature of less importance in the distillation process. If feasible, a small experiment with a toy model that does incorporate a sequence modelling aspect should not qualitatively change the paper's findings.
  • "Divergence choice does not matter around optimality." Especially at start of distillation procedure it's hard for me to imagine, that the student distribution can sufficiently match the teacher's one. Have the author's actually checked the differences between teacher and student distributions at different points of distillation? As the authors note themselves in l. 1118-1122, for divergence choices to not matter, not only would have the divergence between teacher and student distributions to be small, but πθ(j)−π∗(j)\pi_{\theta}(j) - \pi^{\ast}(j) for every token jj.

Recommendation

Considered in total, I'd rate the paper with a weak accept. It is a good paper that tackles an interesting problem, motivates its treatment and indeed does contribute new knowledge to the community. I am, however, unsure to what degree the findings are practically relevant (see my first stated weakness).

I would be willing to increase my score if the author's are able to (1) address my concerns and equip their experimental validation with a toy sequence model and compare teacher with student distributions (not only in their entropy) to solidify their divergence invariance claims and (2) submit code for verification.

Questions for authors
  • The difference between sequence- and token-wise loss computation seems central to the disentanglement of sampling probability and divergence choice. Yet I find it barely explained in the main text and somewhat opaquely in the Appendix. Can the authors explain/derive simply and clearly the mechanism that fixes the sampling probability when using sequence probability divergences?

  • Can the authors explain more clearly what they mean by the sentence "The divergence therefore acts as an implicit entropy regularizer, whose role is clearest in self-distillation: as conditioning on privileged information deflates entropy, the divergence hyperparameters that work best are those that compensate for it." (l. 018-022)? The same phrasing comes up in l. 052-053, when speaking about the "entropy drop that conditioning on privileged information induces."

  • How general do the authors think their findings are? As distillation is not specific to LLMs, can the entropy findings be naively extended to where student distributions are learned from teacher ones by minimizing a divergence?

3 comments

We thank the reviewer for their insightful review, their positive feedback, and for clearly stating what would change their mind. We answer their questions below and extend our toy model to sequential data. If this addresses their concerns, we would appreciate a score update, as offered in their review, and remain available for further clarification.

1. Why the sequence component matters little in practice

In practice, gradients are not backpropagated through sampling, so the sampling distribution is implicitly treated as fixed at every step. Intuitively, distillation then amounts to "matching the teacher on the given contexts", and not also to "changing the sampling so that the student visits contexts on which it is close to the teacher", which is what the gradient through sampling would do.

Following the reviewer's recommendation, we extend the toy model to sequences: the teacher is a 3-gram model, whose table of next-token distributions plays the role of the static teacher mapping, and the student is a 2-layer transformer. Ablating the divergence and on- vs. off-policy sampling with the stop-gradient yields the same findings as Fig. 5 (new App. B.8, Fig. 9). To test the above intuition, we also train without the stop-gradient on sampling: on-policy distillation then behaves qualitatively differently, lowering the student's entropy on its own completions for both divergences.

In short, because of one implementation detail, the stop-gradient, LLM distillation can be well understood while ignoring the sequence component. Our results are tailored to that regime.

2. Divergence around optimality

We see the result of Sec. 3.2, that different divergences lead to similar learning dynamics close to optimality, as a conceptual point worth knowing for researchers in LLM distillation, though likely of little practical relevance there: it only applies when the student is at least as large as the teacher and already very close to it, e.g. when distilling a model into a finetuned version of itself, which is rarely useful. This shared local geometry may matter more for KL regularization, where learning starts at the reference distribution the regularization pulls towards; this is outside the paper's scope.

To confirm that, we checked how far the student is from the teacher in the experiments of Fig. 5: they remain large and different throughout distillation as shown in Fig. 12.

3. Code

The code to reproduce our experiments, including the new toy sequence model, is available here.

4. Why a sequence-level loss fixes the sampling distribution

The decomposition is indeed central to our argument; it already appears in previous work that we follow (Czarnecki et al., 2019; Agarwal et al., 2024). The mechanism is the chain rule of the KL divergence (Eqs. 14–15): a sequence-level KL is a token-level KL averaged over prefixes sampled from the first argument of the divergence. For the reverse KL, this is the student: sequence-level reverse KL forces on-policy data. For the forward KL, it is the teacher: sequence-level forward KL forces off-policy data. Choosing a sequence-level divergence thus also chooses the sampling distribution. Writing the objective at the token level from the start (Eq. 1) removes this constraint: any token-level divergence can be paired with any sampling distribution, e.g. forward KL on student completions, which lets us vary the two independently.

5. Sentence clarification

"The divergence acts as an implicit entropy regularizer." None of our objectives contains an entropy term, yet the choice of divergence determines whether the student ends up with more or less entropy, much like an implicit entropy bonus or penalty.

"Its role is clearest in self-distillation." There, the teacher is the student itself, which creates a feedback loop: anything that lowers (or raises) the student's entropy also lowers (or raises) the teacher's, which in turn pushes the student's entropy further in the same direction. Conditioning the teacher on privileged information is one such mechanism, as it can lower its entropy (Nicolicioiu et al., 2026). The divergence can amplify this loop, as reverse KL does until entropy collapses, or counteract it, and the hyperparameters that work best are those that avoid entropy collapse or explosion (Fig. 15).

6. How general are the findings?

Our theoretical and toy model results only rely on neural networks predicting categorical distributions, of finite size and limited capacity, which is at the root of what we observe both in the toy model and in LLMs. Our results therefore a priori extend beyond LLMs to any setting with these properties. We know of no other practical setting that has them, hence our focus on LLMs; we would appreciate pointers to relevant literature.

I thank the authors for their substantial yet targeted responses which have clarified my questions about the divergence around optimality and the fixation of the sampling distribution due to a sequence-level loss. Also, I appreciate the addition of the sequence-aware toy model and making available their code for reproducibility. I'll therefore raise my score.

We thank the reviewer for their thoughtful feedback and for updating their score!

Reviewer_3BPaOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper analyzes distillation through its effect on predictive entropy, separating prefix sampling from the choice of token-level divergence. Its main theoretical result states that, under a fixed context distribution, an admissible logit-temperature scaling direction, and stationarity, forward-KL distillation makes the student’s average entropy equal to the teacher’s average entropy plus the residual KL divergence. The paper also uses a linear-softmax model to investigate reverse KL, generalized Jensen–Shannon divergences, and top-k approximations. Experiments on pretraining, supervised fine-tuning, teacher–student distillation, and self-distillation examine how these effects appear in language models.

Critical strengths
  1. The central identity is interpretable and directly testable. The temperature-direction stationarity argument connects predictive entropy, cross-entropy loss, and approximation error. The paper carefully distinguishes entropy inflation relative to the teacher from an increase in entropy during training.

  2. The analysis goes beyond standard mode-covering and mode-seeking intuition. It identifies a restricted regime in which reverse KL inflates entropy, examines how tail handling changes top-k behavior, and studies interactions between teacher updates and entropy collapse in self-distillation.

Critical weaknesses
  1. Some broad empirical conclusions rest on limited large-model evidence. The teacher–student experiments in Figure 5 and the main Chemistry experiments primarily use a single seed, with limited student scales and task coverage. The three-seed GSM8K experiments and learning-rate sweeps are useful, but do not fully replace repetition of the main comparisons.

  2. The entropy averaging convention needs to be made consistent. Equation (1) sums losses within each sequence, whereas Equation (2) first normalizes by each sequence’s length. The proof in Appendix B.3 weights individual tokens, and Appendix C.3 states that experiments use token-pooled entropy and that sequence-averaged trends can even differ in sign. These quantities are not equivalent for variable-length sequences.

Questions for authors
  1. Are all terms in Equation (6) token-weighted or sequence-weighted? Can the authors align Equations (1) and (2), the proof, and the experimental measurements, and state the precise result for variable-length sequences?

  2. At the pretrained and SFT checkpoints, how large are the entropy–cross-entropy discrepancy and the derivative along the temperature direction? This would clarify the relationship between the exact identity and finite-training approximations.

  3. For the key comparisons in Figure 5, does divergence continue to have a larger effect than sampling across training seeds and averaging conventions?

1 comment

We thank the reviewer for their careful review and positive feedback on our work. The weaknesses they raise are fully addressed below; we hope that the reviewer will reconsider their score accordingly.

1. Limited number of seeds

As mentioned in the global answer (2.b), we ran one extra seed for every arm of Figures 5 and 6 (26 runs), and all findings hold. Given how little entropy varies across seeds in all our experiments, and the compute each extra seed requires, we believe that two seeds strike a good balance between cost and precision.

To broaden the student and task coverage of Figure 5, we also repeat its grid with an OLMo 2 student and OLMo 2 teachers on GSM8K (global answer 2.c, new Fig. 14), with the same conclusions.

2. How large is the gap between entropy and cross-entropy?

It is small: after pretraining, entropy is within 5% of the cross-entropy (global answer 2.d). The new Fig. 11 reports the gap for every pretrained and finetuned checkpoint of Figs. 4 and 10. As the derivative of the loss along the temperature direction is exactly CE−H\mathrm{CE} - H (App. B.3), the gap also measures the distance to stationarity along that direction. Finite training leaves some gap, larger after supervised finetuning, which is a much shorter run, but the identity holds well overall.

3. Effect of the entropy averaging convention

The equations were indeed inconsistent, and we thank the reviewer for pointing it out. We now average over tokens throughout and have updated the paper accordingly (global answer 1.c). All terms of Eq. 6 are token-weighted, and the identity then holds exactly for completions of variable length; with a loss normalized by the length of each completion, it would instead hold for the per-sequence average (App. B.3).

The results of Figure 5 are robust to the averaging convention: with per-sequence averaging, on both frozen teacher completions and student completions, forward KL still ends above reverse KL for every teacher and sampling distribution, and the effect of the divergence remains two to four times larger than that of the sampling distribution (new Figure 13). This holds for both seeds.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global rebuttal

We thank the reviewers for their careful reading and suggestions.

Reviewers find the paper "tackles an interesting problem" and "does contribute new knowledge to the community" (qw5G), is "well written" (kCK8) and "very cogent" (qw5G). They highlight:

  • Decoupling sampling from divergence. "By disentangling the sampling distribution from the divergence, which have often been coupled in previous approaches, the paper provides a clearer understanding of the mechanisms underlying LLM distillation" (kCK8); "This serves as a guide for anyone interested in distilling a model" (qw5G).
  • A toy model that captures the phenomena. It "provides a concrete categorical setting in which representation constraints can be varied" (34j2), the reasoning "is empirically supported by experiments with a comprehensive toy model" (qw5G), "the phase transitions observed in Figure 2 are interesting" (kCK8), and the analysis "goes beyond standard mode-covering and mode-seeking intuition" (3BPa).
  • Theory tied to observables. "The central identity is interpretable and directly testable" (3BPa) and "gives a simple, quantitative connection between an optimization condition and a routinely logged observable" (34j2).

Most concerns request clarifications and experiments, which strengthen and better position our contributions. We believe the points below address all of them, and hope reviewers will reconsider their scores.

1. Clarifications

  • a. Novelty of the theory (kCK8). Except for reverse KL inflating entropy at large nn (App. B.5), our theoretical results are classical in information geometry: the local equivalence of divergences (Sec. 3.2, App. B.1) and the forward KL identity (Eq. 6, App. B.3). To our knowledge, they had not been applied to LLMs; one of our main contributions is leveraging them to understand distillation in the toy model (Sec. 3) and LLMs (Sec. 4). We now say so in the contributions (l.44).

  • b. Practical takeaways (kCK8, qw5G). We expand them in the conclusion:

    • Choose sampling and divergence separately, using the divergence to set entropy: reverse KL, not on-policy sampling, lowers entropy (Fig. 5).
    • With cross-entropy (or forward KL), a converged model's entropy equals its loss, and a model that cannot match its teacher becomes less confident, not overconfident: pretrained LLMs tend to leak probability outside the data support (Fig. 4).
    • In self-distillation, entropy is an early signal for picking hyperparameters, less noisy than accuracy, whose trends transfer across setups (Figs. 6, 16). Frans et al. (2026; post-deadline, now cited) observe similarly in agentic RL with self-distillation that an entropy spike, triggered by an overweighted distillation loss, predicts collapse.
  • c. Token vs. sequence averaging (3BPa, 34j2). This distinction only matters for variable-length sequences: the conventions coincide in the toy model (no sequences) and in pretraining (fixed length). Eq. 2 was inconsistent with our experiments; we replace it with the token-pooled average Hˉ(π)=E[∑tHt]/E[T]\bar H(\pi) = \mathbb{E}[\sum_t H_t] / \mathbb{E}[T], under which Eq. 6 holds exactly for variable lengths. The conclusions of Fig. 5 hold under both (Fig. 13).

  • d. Missing relevant work (34j2). Jin et al. (2026) is relevant; we now cite it in Sec. 4.2 (after l.373).

2. Additional experiments

  • a. Sequential toy model (qw5G). We add a sequential component to our toy model (new App. B.8, Fig. 9): a 3-gram teacher (each token depends on the two preceding ones) and a 2-layer transformer student. With the stop-gradient used in practice, it reproduces Fig. 5; with the full gradient, on-policy distillation instead lowers the student's entropy on its own completions, for both divergences. Our results thus hinge on the stop-gradient on sampling.

  • b. More seeds (3BPa). We add a second seed to every arm of Figs. 5 (12 runs) and 6 (14 runs). We stop at two because of cost (one Fig. 6 seed takes ~230 H100-hours) and as entropy barely varies across seeds.

  • c. Second model family (34j2, kCK8). We repeat the grid of Fig. 5 in the OLMo 2 family, distilling 1B, 7B and 13B instruct teachers into the 1B base model (new App. C.3, Fig. 14). As this student solves only a few percent of MATH500, we use GSM8K, learning rate 5e-6 and one seed. The ordering and trends of Fig. 5 carry over.

  • d. Distance to stationarity (3BPa). For a softmax head, the loss derivative along the temperature direction is exactly CE − H; new Fig. 11 (App. C.1) shows the relative gap (H − CE) / CE for all checkpoints of Figs. 4 and 10. It stays within 5% after pretraining and is larger after SFT: models trained long enough on a data mixture are near-stationary along that direction.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.