acceptodds
Under review as a conference paper at ICLR 2027

Spectral Starvation in On Policy Distillation

Abstract

On-policy distillation can improve from a small selected query set while absorbing teacher supervision progressively more slowly. We study this phenomenon through the spectrum induced by student-generated prefixes. Specifically, for every smooth -divergence over tokens, the local OPD operator is an occupancy-weighted pullback of categorical Fisher geometry. The exact cross-entropy Hessian separates this Gauss-Newton operator from a teacher-gap-weighted network-curvature remainder. Under this local geometry, the linearized dynamics show that the step-to-step decline in absorption equals a normalized variance of active contraction factors, while the continuous log-rate derivative equals minus four times the active eigenvalue variance. Fast modes disappear first and leave the residual concentrated on weak directions. Kernel components form an invariant floor, and the smallest active eigenvalue gives the finite-time envelope and limiting rate. Consequently, equal-budget query choice is horizon dependent, while changing-operator bounds characterize the validity window of a frozen spectral forecast. Experimentally, in 160 unclipped Transformer runs, the initial student-Fisher predictor explains short-horizon gap reduction with and rank correlation 0.925. Across 64 matched replays, tangent drift tracks frozen-forecast error with rank correlation 0.504 and seed-cluster interval . The broader 1,088-run study spans GSM8K and ARC-Challenge; all eight ARC selector and training-mode conditions exhibit positive starvation strength with intervals bounded away from zero. Consistent with this local-validity picture, direct measurements find order-one occupancy-induced operator replacement by update 64. Finally, after a common 32-candidate screen and at equal 256 online teacher calls, four selected Residual E prompts absorb more maximum teacher gap than sixteen selected random prompts. Explicit accounting specifies screening, online-trajectory, and token-position cost.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
on-policy distillation, knowledge distillation, spectral analysis, Fisher information geometry, query selection, optimal experimental design

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_PdZZOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks why on-policy distillation of LLMs keeps slowing even though every student rollout gets abundant teacher supervision. Its answer is a local curvature of the loss, the teacher-student residual decays eigenmodewise under that operator, so strong modes decay first, and the residual is supported on weak eigen directions. It shows every smooth ff-divergence used for training objective shares the same local eigendirections up to the scalar f′′(1)f''(1), and hence the query distribution, not the ff-divergence, sets hard directions.

Critical strengths

The paper's main point is that optimally choosing a fixed number of prompts from a larger dataset is training horizon-dependent. Gradient-energy criteria (largest early gradient) win short horizon, and choosing prompts that updates lowest eigenvalue most criteria win long horizon (Theorem 3.3). Concurrent empirical work has established that OPD needs very few prompts yet absorbs slowly, for example, arXiv:2609.04172 ``Rethinking On-Policy Distillation of LLMs II: One Training Example'' - the current paper sits well with those findings.

Critical weaknesses
  • The theorems are repackaged well-known facts, nothing specific to the particular application, and hence, theoretical novelty is thin. Theorem 3.2 is gradient descent on a fixed quadratic: results of linear iterative methods. It appears throughout the spectral-bias/linearized-dynamics literature, for example, arXiv:1710.03667, arXiv:2002.02561, arXiv:1806.08734 - none of which is cited. Fisher/NTK-based data selection exists, see for example arXiv:2106.09675, arXiv:2206.12569.
  • The natural conclusion of theoretical study is not to choose prompts once and keep them. Instead, re-choose them as the remaining gap from the teacher shifts, possibly also switching the selection rule. Yet every experiment selects prompts once, at a single common checkpoint, and trains on that fixed set for all updates. Given a dataset, if one just trains on it with random prompts vs if one carefully chooses prompts during training, it is not clear that the second method will give lower final loss. At a compute matched setup, the second method is unlikely to beat the first either. Neither of these two is analyzed in the paper.
Questions for authors

See weaknesses.

Reviewer_Y2RVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper explains why on-policy distillation (OPD) improves from a small selected query set yet absorbs teacher supervision progressively more slowly. The authors develop a local spectral account: for every smooth f-divergence over tokens, the local OPD operator is an occupancy-weighted pullback of categorical Fisher geometry (Thm 3.1), and the exact cross-entropy Hessian separates this Gauss–Newton operator from a teacher-gap-weighted network-curvature remainder (Prop 3.2). Under the linearized dynamics, the step-to-step decline in absorption equals a normalized variance of active contraction factors, and the continuous log-rate derivative equals −4 times the active eigenvalue variance; fast modes disappear first, kernel components form an invariant floor, and the smallest active eigenvalue governs the finite-time envelope and limiting rate (Thm 3.2). Consequences: equal-budget query selection is horizon dependent (gradient energy vs. E-optimality can reverse rankings, Thm 3.3), and a drifting-operator bound characterizes the validity window of a frozen spectral forecast (Thm 3.4). Empirically, the paper closes an unusually tight theory–measurement loop: an exact finite system verifies the modal identities to machine precision; across 160 unclipped Transformer runs the initial student-Fisher predictor explains short-horizon gap reduction with R²=0.906 and rank correlation 0.925; 64 matched replays link tangent drift to frozen-forecast error (rank correlation 0.504, interval [0.392, 0.623]); a 1,088-run study spans GSM8K and ARC-Challenge with positive starvation strength in all eight selector×mode conditions; and direct measurement shows order-one operator replacement by update 64 (~89° principal angle). At equal 256 online teacher calls, four Residual-E prompts beat sixteen random prompts on maximum teacher-gap absorption (+0.078, [0.009, 0.152]), with explicit screening/online/token cost accounting.

Critical strengths
  1. An unusually tight, and unusually honest, theory–measurement closure. The mathematics itself is clean but standard; the real contribution is converting a vague engineering observation ("OPD slows down") into a falsifiable, measurable object. The theory-to-measurement map (Table 4), machine-precision finite certificates, matched replay that separates forecast error from replay error (4.9×10⁻⁴), and direct operator-drift measurement at hash-verified archived adapters make spectral starvation an independently tested prediction rather than a post-hoc narrative. Most commendably, the authors directly measure where their own model breaks (order-one operator replacement by update 64). Statistical discipline (seed as the unit of replication, 10,000 paired bootstrap draws, exact 2¹⁶ sign-flip inference, four-level cost accounting) is exemplary.

  2. A practically operable reconceptualization of query selection. Reframing query choice as occupancy-weighted spectral experimental design — with the four-coverage taxonomy, the kernel-intersection test (Prop B.10), replication invariance (Prop B.11), and the equal-budget horizon reversal (Thm B.14) — gives practitioners a computable diagnostic vocabulary (kernel dimension, smallest active eigenvalue, residual energy, spectral variance) rather than another black-box selector, and §E.8 turns it into a concrete workflow. The matched-cost result (four selected queries outperform sixteen random ones at equal online teacher calls, with screening cost stated separately) demonstrates that the geometry matters at fixed budget.

Critical weaknesses
  1. A scope gap between the frozen-operator theory and the practical claims. The paper's own hardest data delimit the theory: the quantitative predictor covers only the first 16 of 256 updates, and only under deliberately theory-aligned unclipped SGD; in the practically relevant fully on-policy AdamW regime, all quantitative evidence degrades to rank-level qualitative trends (absorption–time association attenuates from −0.649 to −0.301, with 32/160 positive runs), while direct measurements show the operator is replaced order-one by update 64 (~89° principal angle) — i.e., the frozen model is essentially invalid in the practical setting within the first quarter of training. Meanwhile the headline consequence — horizon-dependent query selection — is demonstrated mainly under fixed-prefix SGD and in a two-dimensional certificate; the "four selected beat sixteen random" result is a single fixed-horizon (256 updates), single-model-pair comparison, and the selector itself is computed once at the common checkpoint, never realizing the adaptive energy↔E-optimal scheduling envisioned in §E.4. The conclusion's claim to establish "a predictive theory of OPD dynamics" outruns the demonstrated validity window.

  2. All primary conclusions rest on a proxy metric, at limited scale. Every primary outcome is teacher-gap reduction on a 32-candidate pool (max/mean/heldout); no downstream task accuracy is reported anywhere. This is not nitpicking: Props B.16–B.17 prove that continuous teacher-gap reduction can leave argmax decisions untouched until decision boundaries are crossed, i.e., gap reduction does not imply task improvement. Having so precisely characterized that disconnect, the experiments never test when or whether it is crossed, so the practical value of the "query design" remains an inner-loop quantity. Moreover, the setting is rank-8 LoRA, ≤64-token rollouts, top-16 truncated soft targets, and 128–384-token prompts; the 4B/9B panel still uses 16 candidates, 8 seeds, and 128 updates. Whether spectral-conditioning effects survive full fine-tuning, longer rollouts, and candidate pools too large for exact subset enumeration is entirely untested.

Questions for authors
  1. The predictor's R² is 0.810 at update 1 but 0.930 at update 2 (pooled 0.906). What drives the low one-step value — cross-horizon aggregation, measurement noise of the design-pool gap, or a still-significant network-curvature remainder (second term of Prop B.2) one step away from the common checkpoint?

  2. Starvation attenuates under fully on-policy AdamW (−0.649 → −0.301; 32/160 positive). Is this consistent with the commuting-path picture (Prop B.19), i.e., occupancy churn repeatedly re-injecting strong modes? Could you compute modal exposure sequences from the measured A_t paths to quantitatively account for the attenuation?

  3. Given that Props B.16–B.17 precisely characterize the gap-reduction/decision disconnect, can you report downstream accuracy for even one condition (e.g., on the heldout pool), or at least an empirical timescale for when gap reduction flips decoding decisions?

  4. Since the operator is replaced order-one by update 64, does periodic re-selection (refreshing the spectral design at geometric checkpoints) beat one-shot selection? This would directly test the scheduling implication of §E.4.

  5. Does the top-16 teacher truncation plus renormalization introduce zero-curvature directions (unobserved tokens) that artificially inflate the kernel component? How sensitive is the invariant-floor conclusion to this?

  6. Under full fine-tuning rather than rank-8 LoRA, how do the pullback operator's condition number and starvation strength change? Is the low-rank subspace a prerequisite for the strong starvation observed?

Reviewer_6GeuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies diminishing returns in on-policy distillation (OPD) and how to select training prompts. It uses a local analysis of the Fisher spectrum to explain the slowdown in convergence and proposes selection criteria. Numerical tests and Qwen3.5 experiments show that the local model can predict short-term changes in the teacher gap and that some selection methods outperform random selection.

Critical strengths
  1. Under fixed prefixes and SGD without gradient clipping, the local model predicts changes in the neural model's teacher gap well through update 16 (pooled R2=0.906R^2=0.906). This gives the approximation practical predictive value in a controlled, short training window.
  2. The authors measure curvature drift and prediction error, and use comparisons with the same optimizer to examine the effect of changing prefixes. These measurements provide empirical evidence on the local model's range of validity.
Critical weaknesses
  1. The theory contains an incorrect formula and does not fully establish how local spectral analysis supports practical query selection.

    Using local spectral properties to guide OPD query selection requires showing that the local model describes the training states being studied. The paper establishes the Fisher approximation near teacher–student agreement, but does not adequately control the omitted network-curvature term at general checkpoints. The error guarantee near agreement therefore does not directly apply to the checkpoints used in practice.

    Even within this local approximation, some conclusions need correction. Equation (10) gives a positive limiting fractional absorption rate even with a nonzero irreducible residual, although this rate should tend to zero. Long-run behavior thus depends on the residual floor as well as the smallest positive eigenvalue.

    A more consequential issue arises when comparing query designs: the paper treats the gap-space operator as linear in the query weights, although this linearity generally holds for the corresponding parameter-space operator. Switching to the latter requires re-establishing how the residual, null space, and common evaluation objective correspond. Otherwise, better spectral coverage need not reduce the current teacher gap, and accurate predictions of individual training curves need not give reliable rankings of query designs. The central step of using spectral properties to guide practical selection therefore remains insufficiently justified.

  2. The experiments do not isolate the gains from spectral information, or fully establish when they hold and whether they justify the cost.

    To demonstrate the practical value of spectral selection, the paper needs to show what it adds beyond simple gradient information. Over the reported training horizons, Residual E and gradient-norm selection yield similar reductions in the maximum teacher gap over the candidate pool, but their direct difference is not statistically assessed. The local predictor performs well, but uses both gradients and curvature, without a comparison that removes curvature or replaces it with a simpler approximation. These results leave the additional predictive and selection value of spectral structure unclear.

    Even if an added benefit is established, its dependence on training stage needs to be tested. The paper proposes different criteria for different training horizons, but the neural experiments do not directly demonstrate predictable reversals between short- and long-horizon rankings. The measured Fisher drift shows that the initial geometry changes, but the study does not test when to re-estimate it or whether reselecting queries helps. Some gains from the initial selection persist, so inaccurate frozen predictions do not by themselves show that the selection has become ineffective. What remains untested is whether prediction error or operator drift can indicate changes in selection benefit and when reselection is worthwhile.

    If refreshing is needed, its value depends on whether the gains outweigh re-estimation and reselection costs. The paper accounts for teacher calls, but does not fully compare selector-specific precomputation, probe, and search costs, or test whether the gains are stable across probe counts and sketch dimensions. The experiments also focus on a single model family, short rollouts, and short LoRA training runs. The evidence supports local prediction and some selection gains in those settings, but does not yet establish effectiveness and cost efficiency across broader OPD settings.

Questions for authors
  1. How does the gap-space operator in Eq. (6) satisfy the linear combination in Eq. (14)? If the latter refers to the parameter-space operator, how do the residual and common evaluation objective map between the two spaces?
  2. What justifies omitting the network-curvature term at the checkpoints used in practice? Do the error guarantees near teacher–student agreement apply to these training states?
  3. What is the paired difference, with its confidence interval, between Residual E and Gradient norm in the reduction of the maximum teacher gap over the candidate pool? With the other prediction steps unchanged, how does prediction accuracy change when curvature is removed or replaced with a simpler approximation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.