acceptodds
Under review as a conference paper at ICLR 2027

Training-Sufficient Supervision: What Distillation Caches Must Preserve

Abstract

A distillation cache can be built to reconstruct the teacher's prediction, or to reproduce the training a student would have performed. These are different requirements. For cross-entropy under a frozen affine head the teacher enters the update only through , so targets differing elsewhere on the simplex induce identical gradients. We call a representation training-sufficient when it preserves the gradient field over the whole declared parameter domain rather than at one iterate, and characterize it: a fixed linear cache serves every target exactly when the reachable space of centered logit changes lies in its column space, and the least number of coordinates it can use is the dimension of that space. Two constructions follow, one for a frozen head and one for output directions opened on a schedule, the second identifying what must be appended when training reaches directions the cache does not cover. Controlled interventions confirm the predicted separation and its repair. Trained at a matched byte budget against renormalized Top-k and a low-rank reconstruction of the teacher distribution, the two exact representations hold gradient error four to five orders of magnitude lower along the whole trajectory, and store the supervision in under 5% of a full target's bytes, which on one pair yields a measured per-step rate. Sufficiency remains a property of the declared loss and training family, not of the teacher alone: reversing the divergence, with the student untouched, raises the requirement to coordinates.

Primary area:
learning theory
Keywords:
knowledge distillation, training sufficiency, gradient-field equivalence, supervision caching, trainable subspaces, offline distillation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_DngMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates which teacher information should be cached in offline distillation so that the student can obtain the same training gradients as those obtained using the full teacher distribution. To this end, the authors mathematically define a criterion called “training sufficiency” and propose MOMENT and MOMENT+ for two settings: freezing the student output head and opening some new training directions. Experiments on three main model pairs show that MOMENT can substantially reduce storage and achieves lower gradient error than conventional methods.

Critical strengths
  1. It provides a thorough characterization of “training sufficiency” and explains, for a general family of student training parameters, what kind of linear caching can preserve gradients over the entire allowed training range.
  2. It proposes two new caching schemes, MOMENT and MOMENT+, which achieve certain empirical benefits.
Critical weaknesses
  1. The introduction to the research background and prerequisite knowledge is almost absent or only mentioned in passing.
  2. The writing is poorly organized; the abstract and the first section mix specific methods with background introduction.
  3. The experimental section does not clearly describe the experimental setup, data conditions, etc. Some figures are poorly drawn. For example, Figure 2 places the gradient difference between two teachers, the gradient error of the cache relative to the full target, and another set of staged training results in the same figure. The comparison objects and experimental sources switch, so readers must read the caption carefully to distinguish them.
  4. The novelty of the proposed MOMENT is limited. An identical approach is used in [1]. The paper should consider rephrasing their claims of contributions and focus on the training sufficiency and MOMENT+.

[1] Jonathan Chang, "KL Cache Trick for Full-Vocab Distillation", url: https://jonathanc.net/blog/kl-cache-trick.

Questions for authors
  1. Please clarify the research background and contribution hierarchy of this paper: which parts belong to existing foundations, and which parts constitute the new contributions of this paper?
  2. What specific corpus is used for the 128 steps of student training in Table 2? Please specify the dataset name, version, split, sample size, and how the training prefixes and supervision positions are constructed.
6 comments

Thank you for recognizing the characterization of training sufficiency. We would like to clarify which results build on existing foundations, which are our contributions, and how we will address the presentation concerns.

1. Contribution hierarchy and the fixed-head construction

Chang's note [1] describes the same fixed-head projection μ=W0⊤q\mu=W_0^{\top}q and the corresponding loss reformulation. We will cite it explicitly in Section 2.2. This overlap concerns the fixed-head construction; it does not establish the general characterization or the cache-expansion results.

The submission already introduces Eq. (1) as an identity. We will make its role as the starting example clearer and foreground the following results:

  • General criterion and minimum dimension (Theorem 1). Under the stated assumptions, we characterize when two targets induce the same gradient field throughout a declared parameter domain. We then establish necessary and sufficient conditions for a fixed linear cache to preserve that field for every target, and identify the minimum number of independent linear coordinates required. When one basis serves multiple prefixes, the relevant dimension is that of their joint reachable centered-logit space. This is a result about universally sufficient fixed linear caches, not arbitrary encodings.

  • Expansion and reuse (Section 2.3). For W=W0+UAW=W_0+UA, with W0W_0 and UU fixed and AA trainable, we identify when the original cache already covers the newly trainable directions and when additional statistics ν=U⊤q\nu=U^{\top}q are required. The analysis predicts where gradients separate when information is missing and how augmenting the cache restores each target's Full-gradient reference.

These are separate claims from the fixed-head identity. We agree with making this hierarchy more explicit in the abstract and introduction, rather than presenting the constructions without sufficiently distinguishing their foundations.

2. Background, organization, and Figure 2

The task is offline next-token distillation: for each text prefix, a teacher supplies a vocabulary distribution, and the student later trains against cached supervision using cross-entropy. We retain the input prefixes and supervision positions and ask what teacher information must be stored for the specified student training family.

We will establish this setting before the first gradient equation, defining the teacher target, student hidden state, output distribution, loss, and trainable parameters. The introduction will then distinguish the fixed-head example from the general criterion and expansion results.

Figure 2 was intended to connect the mechanism and its implementation: target information can be invisible under one training family, become relevant when new directions open, and be recovered by cache augmentation. The scientific comparison is intentional, but the current layout mixes different quantities. We will separate: (a) gradient separation between the constructed target pair; (b) cache-versus-Full error for the controlled intervention; and (c) cache fidelity in the separate staged-training run.

Each panel will identify the parameter block, reference, error definition, and experimental source. The predicted-versus-observed comparison for Eq. (13) will remain with the controlled mechanism experiment.

3. Table 2 data protocol

The submission does not provide a sufficiently explicit description of the original Table 2 training corpus. We will answer this question separately using the original experiment records, specifying the dataset identifier and version, configuration, split, sample count and selection, prefix construction, and supervision-position masks. We will also provide the actual optimizer-step and token counts, together with the corrected checkpoint evaluations.

This protocol will be stated directly in the experimental setup and appendix. The held-out WikiText evaluation description is not a substitute for identifying the training data.

[1] Jonathan Chang, "KL Cache Trick for Full-Vocab Distillation" (2026).

Follow-up on the Table 2 training data

For Table 2, all three model pairs (Q1, S1, and P2) use the official training split of WikiText-103, raw-v1 (Salesforce/wikitext, wikitext-103-raw-v1).

We selected a fixed subset of training text and tokenized it separately for each model family. The text was divided into non-overlapping 512-token blocks within individual text segments, without padding or adding special tokens. We retained 1,024 blocks per family, giving 524,288 input tokens.

Within each block, we sampled 32 of the 511 valid next-token positions uniformly without replacement and kept these positions fixed throughout training. At each selected position, the teacher provides its next-token distribution conditioned on the block prefix up to that position. This gives 32,768 distinct cached teacher targets per family. All methods within a model pair use the same token sequences, supervision positions, and underlying teacher targets; only the cached representation changes.

Held-out NLL is evaluated on blocks constructed from the official test split. Unlike the sampled positions used for distillation, evaluation covers all 511 valid next-token positions in each block, using the reference tokens.

We will add these details to the experimental setup and Appendix B so that the training corpus, prefix construction, and supervision scheme are stated together.

A note on how this project started

We would also like to explain the motivation behind this work. While comparing supervised fine-tuning (SFT) and on-policy distillation (OPD), we ran into the storage cost of caching teacher distributions. This brought to mind Godey and Artzi's “Lost in Backpropagation: The LM Head is a Gradient Bottleneck” [1], which studies how the output head restricts the gradient signal reaching the model backbone.

That connection prompted a different question: if some distinctions between teacher targets never affect the student's updates, why should the cache preserve them? What acts as a bottleneck for gradient transmission can become a principle for compressing supervision.

We formulate this question in the offline, forward-cross-entropy setting. With a fixed affine head, the teacher influences the backbone gradient only through its projection onto the student head. Caching that projection does not introduce an additional approximation to the specified training objective; it stores the teacher information that this training family actually uses.

The next question was what makes this idea work beyond that example. Which target information must remain available as the student parameters change? What is the smallest sufficient linear representation? If new output directions become trainable, can the existing cache still be used, or must it be extended? These questions led to the general criterion, the minimum-dimension result, and MOMENT+.

This is the connection we should have made clearer in the introduction: the work starts from a practical storage problem, uses gradient geometry to identify what can be discarded, and develops that observation into a rule for designing and reusing distillation caches.

[1] Nathan Godey and Yoav Artzi. “Lost in Backpropagation: The LM Head is a Gradient Bottleneck.” 2026.

Reviewer_kWpaOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper investigates, whether it is possible to more efficiently compress the data of a teacher that will be used to distill its performance to a known student architecture.

Critical strengths

.

Critical weaknesses
  • W1: I gave a soundness score of 1, only because I had such a hard time following the paper. Perhaps the experiments and results are sound, but I did not manage to get a clear understanding of them.
  • W2: There is a community around dataset distillation, which tries to find the optimal small dataset to train a student (e.g. https://arxiv.org/abs/1811.10959). I think this relates to the efficiency goals of the saved data from this work, but was not mentioned.
  • W3: The paper does not follow the ordinary structure of a conference paper. There is no specific discussion of the results or concrete contributions, the related work is separated. This contributed to being difficult to read.
  • W4: The exposition, including the abstract, uses many terms that are unclearly defined in this context. (L33: Offline distillation of what? L34: training prefix of what? L35 writes the cache - why? caching what exactly?) It goes on similarly from there. We do not know what hh is in equation (1), what the architecture of the models is, what loss we are even deriving ...
Questions for authors

I believe this work needs substantial improvement.

As a first question, I would like to understand the relevance of the problem of compressing the distillation data for the specific student model.

Nonetheless, I do not see myself recommending acceptance based on the state of this work even after the discussion period - it would be too major of a rewrite.

4 comments

Thank you for explaining where the paper was difficult to follow. Let us clarify the task, its practical relevance, and the contribution beyond the fixed-head example.

1. What is being distilled, and why use a student-specific cache?

We study next-token distillation between language models. For each text prefix (the tokens preceding a supervised position), a frozen teacher produces a vocabulary distribution qq. The student trains against this target using cross-entropy. Offline distillation computes and stores teacher supervision in advance, avoiding repeated teacher-backbone computation during student training. The cache stores supervision, not a replacement for the input text.

For the frozen-head setting, let hθh_{\theta} be the student's hidden state, W0W_0 its fixed output matrix, and p=softmax⁡(W0hθ)p=\operatorname{softmax}(W_0h_{\theta}). The loss and hidden-state gradient are

\qquad \nabla_h L=W_0^{\top}p-W_0^{\top}q.$$ The teacher therefore enters the gradient only through $\mu=W_0^{\top}q$. Equivalently, $$L(\theta;q)=\operatorname{LSE}(W_0h_{\theta})-h_{\theta}^{\top}\mu,$$ where LSE denotes log-sum-exp. Storing $\mu$ preserves this loss and all gradients with respect to the trainable internal parameters in exact arithmetic. In the main matched comparisons, these parameters are LoRA adapters. Floating-point discrepancies are measured separately. The storage benefit is concrete: Q1 has 151,936 vocabulary entries and a student hidden width of 1,024. At FP32, a full target takes 607,744 bytes per position, versus 4,096 for the projection, approximately a 148-fold reduction. This is a supervision-storage reduction, not a claim of equivalent total-training speedup. Specialization is a deliberate trade-off. When the student training family is known, teacher information that cannot affect its updates need not be stored. A teacher-state cache permits reconstruction of the teacher distribution and broader reuse across student configurations; our representation avoids reconstruction but is tied to the declared training family. Section 4 discusses this distinction, which we will bring forward in the introduction. **2. Relation to dataset distillation** Wang et al. [1] is relevant and will be included. It learns a small synthetic dataset intended to approximate the outcome of training on a larger dataset. We keep the input prefixes and supervision positions unchanged and compress the teacher distribution attached to each position. Thus, the two approaches reduce different parts of the training data. Our guarantee concerns preservation of a gradient field throughout a declared parameter domain, rather than similarity of outcomes obtained from a smaller synthetic dataset. **3. Contributions beyond the fixed-head identity** The submission explicitly introduces Eq. (1) as an identity. Our main theoretical result is the general characterization in Theorem 1. For fixed teacher targets and continuously differentiable logits over a connected open parameter domain, it characterizes when two targets induce identical gradient fields. It also gives a necessary and sufficient condition for a fixed linear cache to preserve those fields for every target, and the minimum number of independent linear coordinates required. That minimum is the dimension of the reachable centered-logit space: the span of possible logit changes after removing uniform shifts, which leave softmax unchanged. The result concerns universally sufficient fixed linear caches, not arbitrary nonlinear encodings. Proofs are in Appendices A.1-A.2. MOMENT+ addresses changes in the training family. With $W=W_0+UA$, $W_0$ and $U$ are fixed and $A$ is trainable. The analysis identifies when an existing cache already covers newly trainable directions and when additional coordinates $U^{\top}q$ are required. Section 2.3 and Appendices A.4-A.5 establish these reuse and augmentation conditions. **4. Presentation** The opening introduces the criterion before sufficiently explaining the task. We will define the prefix, target, model, loss, and trainable parameters before the first gradient equation, and make the contribution hierarchy explicit. The contribution statement at the end of Section 1 and the results discussion in Sections 3.4 and 4 should also be easier to locate. We will separate background from the analysis and consolidate the experimental protocol, including data construction, training budgets, references, and metric definitions. The explanations above specify the claims and their conditions. If a particular assumption, proof step, or experimental comparison remains a concern, please identify it so that we can address the technical issue directly. [1] Wang, Zhu, Torralba, and Efros. "Dataset Distillation." arXiv:1811.10959.

Dear authors, thank you for your reply. Given how your reply is formulated, it is unclear whether you already fixed, or promise to fix the presentation of the work. From the current version of the pdf-file, I still find it difficult to follow. I will therefore maintain my score.

Dear Reviewer,

Thank you for your follow-up. You were right that our previous reply did not make clear which changes were already in the PDF and which were still planned. I am sorry for that confusion. We have now uploaded a revised manuscript with a substantially different organization.

I would also like to explain more directly how this project started for me. While comparing supervised fine-tuning and on-policy distillation, I ran into the storage cost of teacher distributions. This reminded me of Godey and Artzi's paper, “Lost in Backpropagation: The LM Head is a Gradient Bottleneck.” Their perspective made me wonder whether the same bottleneck could be useful for caching: if some differences between teacher targets cannot affect the student's updates, why do we need to store those differences?

That was the motivation for a student-specific cache. Under a specified loss and trainable parameter family, the aim is not to approximate information the student needs, but to avoid storing information that its training cannot use. We studied this in the offline setting, where the prefixes and teacher targets are fixed before student training.

The fixed-head case gives a concrete example. It then raises questions that the example alone does not answer: what must remain in the cache as the student parameters change, how small can a sufficient linear cache be, and what happens when new output directions become trainable? Those questions led to the general characterization, the minimum-dimension result, and MOMENT+.

Looking back at the original presentation, we moved into the formalism before giving the reader enough reason to care about these questions. Our earlier reply explained some of that missing context, but the paper needed to do this work itself.

We have therefore rewritten the Introduction around this motivation and placed the task, model, loss, and notation before the first gradient equation. Related Work is now in the main text, including the dataset-distillation work you suggested and the existing fixed-head construction. Sections 3 and 4 separate the general theory from the concrete cache constructions.

We also reorganized the experiments. Section 5.1 and Table 1 state the protocols together. Sections 5.4 and 5.5 separately examine whether a cache reproduces its matched Full-target training and how much performance is affected by restricting the output head. The paper now ends with separate Discussion and Conclusion sections.

I still believe the question is worth studying, and I want the paper to make that case clearly rather than ask readers to reconstruct it from the equations. I would appreciate another look at the revised manuscript, particularly the Introduction, Related Work, and experimental setup, and your assessment of whether these changes address the difficulties you raised.

Thank you for taking the time to read and discuss the work.

Dear Reviewer kWpa,

Just a quick note that we've uploaded another revision of the PDF. Sorry for the repeated updates. Following feedback from the other reviewers, we further improved the figures and expanded the explanation of Theorem 1 in the main text.

The overall structure we revised in response to your comments remains unchanged. Thank you again for taking the time to read our paper. We really appreciate it.

Reviewer_mMv6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes training-sufficient distillation cache for offline knowledge distillation to meet the requirement of reproducing the training a student would have performed instead reconstructing the teacher’s prediction. The cache is based on the established theoretical criterion for training sufficiency and characterize the minimum dimensionality required for a fixed linear cache. Two construction methods MOMENT and MOMENT+, for frozen affine heads and predefined trainable output directions are introduced respectively. Experiments across several language model families, primarily involving 64–128 training updates, validate the theoretical properties and demonstrate obvious reductions in cache storage and gradient discrepancies compared with lossy compression methods.

Critical strengths
  1. Clear motivation and theoretical basis, the paper presents a well-motivated perspective on offline knowledge distillation, with a clearly formulated training-sufficiency criterion and a theoretical characterization of the minimum dimensionality required for a fixed linear cache.

  2. The storage and computational overhead associated with caching teacher predictions is a practical concern in offline knowledge distillation, particularly for language models with large vocabularies. The proposed approach addresses a meaningful efficiency problem.

  3. Coherent connection between theory and methodology. The MOMENT and MOMENT+ methods are closely aligned with the theoretical analysis.

Critical weaknesses
  1. The proposed methods rely on frozen affine output heads or predefined trainable output directions, which differ from common distillation settings where the student's output head is fully trainable. Although MOMENT+ partially relaxes this restriction, its updates remain confined to a predefined subspace. The experiments do not provide a direct comparison with conventional distillation using fully trainable output heads, making it difficult to assess the practical applicability and potential performance trade-offs of these assumptions.

  2. Despite evaluating multiple language model families, the main experiments involve only 64–128 training updates, with an additional 256-update exploratory experiment. Although downstream evaluations are included, these short runs provide limited evidence of performance over complete distillation trajectories.

  3. In Q1 (Table 2), MOMENT achieves substantially lower gradient error than Top-k, yet their NLL differences are relatively close. This raises questions about the practical benefits of near-exact gradient preservation and whether numerical discrepancies may accumulate over longer training.

Questions for authors
  1. Can MOMENT or MOMENT+ be extended to distillation settings with fully trainable output heads? If so, could the authors provide experimental results to demonstrate their effectiveness in such settings?
  2. Can the authors provide longer training experiments to demonstrate that MOMENT and MOMENT+ maintain comparable performance to full-target distillation over realistic training horizons?
  3. Could the authors explain why MOMENT achieves orders-of-magnitude lower gradient error than Top-k on Q1, yet yields only a modest improvement in NLL? How does this discrepancy evolve over longer training trajectories?
1 comment

Dear Reviewer, We have now uploaded the revised manuscript, incorporating the additional experiments and corrections discussed in our earlier responses. Thank you again for your time and feedback.

Comments on the paper

1 comment

Dear Reviewer mMv6,

Thank you for clarifying your remaining concerns. Given our limited GPU resources and the time remaining in the discussion period, we prioritized the comparison with unrestricted-head distillation across all three model families.

We have now completed matched experiments on Qwen3, SmolLM2, and Pythia, each with 2,048 updates, 33.6M input tokens, and 2.10M supervised positions. We compare unrestricted-head Full, data-selected rank-256 subspace Full, and MOMENT+ using the same subspace.

The results (test / validation NLL) are:

Q1 (Qwen3): Unrestricted Full: 2.64974 / 2.66665; Subspace Full: 2.65035 / 2.67216; MOMENT+: 2.65050 / 2.67223.

S1 (SmolLM2): Unrestricted Full: 2.82965 / 2.85515; Subspace Full: 2.82546 / 2.85349; MOMENT+: 2.82546 / 2.85349.

P2 (Pythia): Unrestricted Full: 2.66699 / 2.68269; Subspace Full: 2.65928 / 2.67762; MOMENT+: 2.65927 / 2.67762.

MOMENT+ closely reproduces its matched subspace Full reference across all three families. The largest test-NLL difference is only 1.5e-4 nats/token, with same-point relative gradient errors around 1e-6. Compared with unrestricted-head training, Q1 shows a small NLL gap, while S1 and P2 achieve lower NLL on both test and validation splits. An additional Q1 seed shows a similar pattern.

We also want to emphasize why this problem matters at scale. The training-sufficiency guarantee and the cache-size reduction do not depend on the number of training positions. For example, caching 100 million supervised positions in Q1 would require approximately 60.8 TB of dense FP32 teacher distributions, compared with 512 GB for MOMENT+, plus a small shared basis.

A BF16 teacher-state cache has the same per-position size in this configuration, but requires reconstructing teacher distributions during training. MOMENT+ avoids this reconstruction, although end-to-end speed gains depend on the implementation.

Our central contribution is a characterization of what teacher information student training actually requires. The mathematical guarantee holds throughout the declared training family, while our experiments examine its numerical realization and the practical cost of restricting the output head.

We agree that substantially longer training would provide further evidence about final model quality. Given the remaining time and compute budget, we chose these cross-family comparisons to directly address the questions of applicability and competitiveness with unrestricted-head training.

We hope these new results clarify both the theoretical contribution and its practical relevance. We will incorporate them into the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.