- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper disentangles the choice of sampling distribution from the choice of divergence in LLM distillation, allowing these two components to be selected independently. This paper demonstrates that the choice of divergence implicitly controls the entropy of the student model and has a greater impact on entropy than the choice between on-policy and off-policy sampling. Specifically, forward KL increases the student's average entropy relative to the teacher's, whereas reverse KL generally decreases entropy but can also increase it when the distillation task becomes sufficiently challenging.
Critical strengths
- The paper is well written, and the figures are clear and easy to understand.
- By disentangling the sampling distribution from the divergence, which have often been coupled in previous approaches, the paper provides a clearer understanding of the mechanisms underlying LLM distillation.
Critical weaknesses
Variety of experiments
Most experiments focus on measuring entropy, while its effects on loss and downstream accuracy are not sufficiently investigated through experiments, except for the results presented in Figure 10. Moreover, the standard distillation experiments are limited to mathematical reasoning tasks using Qwen models. Other tasks and model families are not explored, except in the self-distillation experiments.
Presentation of the novelty
The novelty of the theoretical contributions is not sufficiently clear. Several of the main results are derived from classical results in information geometry, making it difficult to distinguish genuinely new theoretical findings from applications of existing results.
Questions for authors
- Some of the theoretical results appear to follow from well-established properties in information geometry. To better evaluate the novelty of this paper, I would appreciate a clearer distinction between new theoretical results and applications of existing results to LLM distillation. For example, Eq. 4~6 is based on a classical result from information geometry, and the local equivalence of divergences also appears to be well known. Does the paper establish any fundamentally new mathematical results, or does its main theoretical contribution lie in applying existing results to LLM distillation?
- The phase transitions observed in Figure 2 are interesting. Can these transitions be explained theoretically or these are purely the experimental results?
- What are the main practical suggestions arising from the proposed entropy analysis? Does the analysis offer practical suggestions on selecting divergences to improve training loss or downstream performance? While the paper establishes a clear relationship between divergence choice and entropy, it remains unclear how these findings translate into improvements in task performance.
minor
- How sensitive are the results to the choice of optimizer? Would the authors expect similar trends with optimizers other than AdamW, such as Muon?
- The analysis in Appendix B.5 appears to contain some new derivations. However, they rely on asymptotic approximations expressed using big-O notation, and for me, they seem not mathematically rigorous. For example, eq. 66 establishes an equality in expectation. Should the left-hand side of eq. 67 therefore also be expressed in terms of an expectation? Furthermore, the authors state that "the numbers of contexts and tokens are large enough". It is unclear how large these quantities need to be and whether a joint limit of and is being considered.
3 comments
Thank you for the detailed responses, additional experiments, and revisions to the appendix.
I understand that the authors regard the toy model as a conceptual contribution. However, I think the value of a simplified model in ML generally lies in what it helps us understand or prove mathematically or theoretically. If most of the theoretical results are known results from information geometry, the toy models seem to mainly reproduce observed phenomena rather than explain why they occur or when they occur.
I find the approach interesting, but I am not convinced that reproducing these phenomena in small toy models, without some theoretical results, is enough to constitute a substantial contribution. (For example, to the best of my knowledge, papers that introduce toy models such as linear networks or logistic regression either construct closed-form dynamics or prove theorems using some kind of bound.)
For now, I will keep my score unchanged and defer to the other reviewers' judgment.
We thank the reviewer for engaging with our rebuttal.
We would like to offer a complementary view on the role of toy models in machine learning. Toy models are valuable for proving careful mathematical results, but we believe their usefulness goes further.
- They make extensive hypothesis testing possible. Toy models are many orders of magnitude cheaper to run than the systems they abstract. This lets us test hypotheses systematically, which would be infeasible at scale. They also surface behaviors that motivate further theoretical analysis, even when that analysis is not yet tractable. An example in our work is the phase transition under reverse KL, for which we do not yet have an analytical account.
- They show which components are necessary for a behavior and which are not. Building the simplest model that still reproduces a phenomenon tells us what is needed for it. Anything stripped away without losing the behavior is not required, and anything whose removal breaks it is. We show that a simple MLP captures the entropy dynamics of distillation. So neither scale nor sequential structure is necessary for this behavior (with stop-gradients in the on-policy setting). Like any model in science, ours will break down when pushed past what it can capture. Those failures are informative too, because they point to the additional ingredients other behaviors require.
We thank the reviewer for their thoughtful review, and for finding the paper well written and the decoupling of sampling and divergence useful. We address their concerns below, and hope that they will reconsider their score accordingly.
1. Novelty of the theoretical contributions
As detailed in the global answer (1.a), except for the derivation of App. B.5, our theoretical results apply known results from information geometry, and we now state it explicitly in the contributions. We see our theoretical contribution as mostly conceptual: finding a toy model of distillation that is simple enough for these results to apply directly, and rich enough to reproduce phenomena observed when distilling LLMs (Sec. 4), now also when the student generates its own contexts (global answer 2.a and answer to
qw5G). Without such a model and the simplification work that comes with it, applying these classical results to LLM distillation is harder.2. Can the phase transitions of Fig. 2 be explained theoretically?
Only partially. We tried several approaches, without much success. Our best characterization is the linearization around the uniform distribution of App. B.5: in the diffuse, high-load limit, every divergence of the generalized Jensen–Shannon family reduces to the same quadratic objective, which explains why reverse KL ends up inflating entropy like forward KL when the task is hard, and why this inflated regime looks the same for every divergence (Fig. 7). It therefore shows that there is a transition (given that reverse KL deflates entropy at smaller n), but understanding where it happens is an open question; if such sharp transitions appear in LLM distillation, it would be very interesting to revisit this question.
3. Practical suggestions, loss and downstream performance
See global answer 1.b for the practical takeaways. We agree that the right entropy is not sufficient for good performance: a student can have the right entropy and still put its mass on the wrong tokens. It is however necessary, in the sense that entropy collapse and entropy blow-up are both problems.
More broadly, we see entropy as an observable that is both practically relevant and amenable to theory, and thus as a good place to start building a theory of distillation. It is a first step of this endeavor, not its end point.
Regarding the variety of experiments, we ran the equivalent of Figure 5 on OLMo 2, and are currently extending our GSM8K self-distillation results to OLMo too.
4. Sensitivity to the optimizer
In the toy model, we train until stationarity. The forward KL identity (Eq. 6) holds at any stationary point, whichever optimizer reaches it, and we reasonably expect the trends of Sec. 3 to be the same with other optimizers such as Muon. The optimizer could still matter where several stationary points coexist, for instance near the transitions of Fig. 2, as it can affect which one is reached. For LLMs, which are not trained to stationarity, we do not know, as the optimizer shapes the whole trajectory. Testing Muon for distillation at scale would be an interesting future work direction.
5. Rigor of the derivations of App. B.5
The reviewer has a point, and we revised the derivation (App. B.5). Eq. 66 is indeed an equality in expectation over the teacher logits, and so is now the left-hand side of Eq. 67, which then holds up to O(β3) without any assumption on the number of contexts; the asymptotic notation refers to the diffuse limit β→0, all else fixed. The assumption that "the numbers of contexts and tokens are large enough" only serves to drop the expectation, and we now make it precise: the realized entropy gap concentrates around its expectation, with relative fluctuations of order (n−h)−1/2, in the limit of many contexts n→∞ at fixed ratio h/n and fixed vocabulary size d. We also corrected the proof, which uses that the teacher logits are independent across contexts with zero mean, not that all their entries are independent, which centered logits cannot be. The result in itself is unchanged.