- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper asks why on-policy distillation of LLMs keeps slowing even though every student rollout gets abundant teacher supervision. Its answer is a local curvature of the loss, the teacher-student residual decays eigenmodewise under that operator, so strong modes decay first, and the residual is supported on weak eigen directions. It shows every smooth -divergence used for training objective shares the same local eigendirections up to the scalar , and hence the query distribution, not the -divergence, sets hard directions.
Critical strengths
The paper's main point is that optimally choosing a fixed number of prompts from a larger dataset is training horizon-dependent. Gradient-energy criteria (largest early gradient) win short horizon, and choosing prompts that updates lowest eigenvalue most criteria win long horizon (Theorem 3.3). Concurrent empirical work has established that OPD needs very few prompts yet absorbs slowly, for example, arXiv:2609.04172 ``Rethinking On-Policy Distillation of LLMs II: One Training Example'' - the current paper sits well with those findings.
Critical weaknesses
- The theorems are repackaged well-known facts, nothing specific to the particular application, and hence, theoretical novelty is thin. Theorem 3.2 is gradient descent on a fixed quadratic: results of linear iterative methods. It appears throughout the spectral-bias/linearized-dynamics literature, for example, arXiv:1710.03667, arXiv:2002.02561, arXiv:1806.08734 - none of which is cited. Fisher/NTK-based data selection exists, see for example arXiv:2106.09675, arXiv:2206.12569.
- The natural conclusion of theoretical study is not to choose prompts once and keep them. Instead, re-choose them as the remaining gap from the teacher shifts, possibly also switching the selection rule. Yet every experiment selects prompts once, at a single common checkpoint, and trains on that fixed set for all updates. Given a dataset, if one just trains on it with random prompts vs if one carefully chooses prompts during training, it is not clear that the second method will give lower final loss. At a compute matched setup, the second method is unlikely to beat the first either. Neither of these two is analyzed in the paper.
Questions for authors
See weaknesses.