- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper investigates which teacher information should be cached in offline distillation so that the student can obtain the same training gradients as those obtained using the full teacher distribution. To this end, the authors mathematically define a criterion called “training sufficiency” and propose MOMENT and MOMENT+ for two settings: freezing the student output head and opening some new training directions. Experiments on three main model pairs show that MOMENT can substantially reduce storage and achieves lower gradient error than conventional methods.
Critical strengths
- It provides a thorough characterization of “training sufficiency” and explains, for a general family of student training parameters, what kind of linear caching can preserve gradients over the entire allowed training range.
- It proposes two new caching schemes, MOMENT and MOMENT+, which achieve certain empirical benefits.
Critical weaknesses
- The introduction to the research background and prerequisite knowledge is almost absent or only mentioned in passing.
- The writing is poorly organized; the abstract and the first section mix specific methods with background introduction.
- The experimental section does not clearly describe the experimental setup, data conditions, etc. Some figures are poorly drawn. For example, Figure 2 places the gradient difference between two teachers, the gradient error of the cache relative to the full target, and another set of staged training results in the same figure. The comparison objects and experimental sources switch, so readers must read the caption carefully to distinguish them.
- The novelty of the proposed MOMENT is limited. An identical approach is used in [1]. The paper should consider rephrasing their claims of contributions and focus on the training sufficiency and MOMENT+.
[1] Jonathan Chang, "KL Cache Trick for Full-Vocab Distillation", url: https://jonathanc.net/blog/kl-cache-trick.
Questions for authors
- Please clarify the research background and contribution hierarchy of this paper: which parts belong to existing foundations, and which parts constitute the new contributions of this paper?
- What specific corpus is used for the 128 steps of student training in Table 2? Please specify the dataset name, version, split, sample size, and how the training prefixes and supervision positions are constructed.
6 comments
Follow-up on the Table 2 training data
For Table 2, all three model pairs (Q1, S1, and P2) use the official training split of WikiText-103, raw-v1 (Salesforce/wikitext, wikitext-103-raw-v1).
We selected a fixed subset of training text and tokenized it separately for each model family. The text was divided into non-overlapping 512-token blocks within individual text segments, without padding or adding special tokens. We retained 1,024 blocks per family, giving 524,288 input tokens.
Within each block, we sampled 32 of the 511 valid next-token positions uniformly without replacement and kept these positions fixed throughout training. At each selected position, the teacher provides its next-token distribution conditioned on the block prefix up to that position. This gives 32,768 distinct cached teacher targets per family. All methods within a model pair use the same token sequences, supervision positions, and underlying teacher targets; only the cached representation changes.
Held-out NLL is evaluated on blocks constructed from the official test split. Unlike the sampled positions used for distillation, evaluation covers all 511 valid next-token positions in each block, using the reference tokens.
We will add these details to the experimental setup and Appendix B so that the training corpus, prefix construction, and supervision scheme are stated together.
A note on how this project started
We would also like to explain the motivation behind this work. While comparing supervised fine-tuning (SFT) and on-policy distillation (OPD), we ran into the storage cost of caching teacher distributions. This brought to mind Godey and Artzi's “Lost in Backpropagation: The LM Head is a Gradient Bottleneck” [1], which studies how the output head restricts the gradient signal reaching the model backbone.
That connection prompted a different question: if some distinctions between teacher targets never affect the student's updates, why should the cache preserve them? What acts as a bottleneck for gradient transmission can become a principle for compressing supervision.
We formulate this question in the offline, forward-cross-entropy setting. With a fixed affine head, the teacher influences the backbone gradient only through its projection onto the student head. Caching that projection does not introduce an additional approximation to the specified training objective; it stores the teacher information that this training family actually uses.
The next question was what makes this idea work beyond that example. Which target information must remain available as the student parameters change? What is the smallest sufficient linear representation? If new output directions become trainable, can the existing cache still be used, or must it be extended? These questions led to the general criterion, the minimum-dimension result, and MOMENT+.
This is the connection we should have made clearer in the introduction: the work starts from a practical storage problem, uses gradient geometry to identify what can be discarded, and develops that observation into a rule for designing and reusing distillation caches.
[1] Nathan Godey and Yoav Artzi. “Lost in Backpropagation: The LM Head is a Gradient Bottleneck.” 2026.
Thank you for recognizing the characterization of training sufficiency. We would like to clarify which results build on existing foundations, which are our contributions, and how we will address the presentation concerns.
1. Contribution hierarchy and the fixed-head construction
Chang's note [1] describes the same fixed-head projection μ=W0⊤q and the corresponding loss reformulation. We will cite it explicitly in Section 2.2. This overlap concerns the fixed-head construction; it does not establish the general characterization or the cache-expansion results.
The submission already introduces Eq. (1) as an identity. We will make its role as the starting example clearer and foreground the following results:
General criterion and minimum dimension (Theorem 1). Under the stated assumptions, we characterize when two targets induce the same gradient field throughout a declared parameter domain. We then establish necessary and sufficient conditions for a fixed linear cache to preserve that field for every target, and identify the minimum number of independent linear coordinates required. When one basis serves multiple prefixes, the relevant dimension is that of their joint reachable centered-logit space. This is a result about universally sufficient fixed linear caches, not arbitrary encodings.
Expansion and reuse (Section 2.3). For W=W0+UA, with W0 and U fixed and A trainable, we identify when the original cache already covers the newly trainable directions and when additional statistics ν=U⊤q are required. The analysis predicts where gradients separate when information is missing and how augmenting the cache restores each target's Full-gradient reference.
These are separate claims from the fixed-head identity. We agree with making this hierarchy more explicit in the abstract and introduction, rather than presenting the constructions without sufficiently distinguishing their foundations.
2. Background, organization, and Figure 2
The task is offline next-token distillation: for each text prefix, a teacher supplies a vocabulary distribution, and the student later trains against cached supervision using cross-entropy. We retain the input prefixes and supervision positions and ask what teacher information must be stored for the specified student training family.
We will establish this setting before the first gradient equation, defining the teacher target, student hidden state, output distribution, loss, and trainable parameters. The introduction will then distinguish the fixed-head example from the general criterion and expansion results.
Figure 2 was intended to connect the mechanism and its implementation: target information can be invisible under one training family, become relevant when new directions open, and be recovered by cache augmentation. The scientific comparison is intentional, but the current layout mixes different quantities. We will separate: (a) gradient separation between the constructed target pair; (b) cache-versus-Full error for the controlled intervention; and (c) cache fidelity in the separate staged-training run.
Each panel will identify the parameter block, reference, error definition, and experimental source. The predicted-versus-observed comparison for Eq. (13) will remain with the controlled mechanism experiment.
3. Table 2 data protocol
The submission does not provide a sufficiently explicit description of the original Table 2 training corpus. We will answer this question separately using the original experiment records, specifying the dataset identifier and version, configuration, split, sample count and selection, prefix construction, and supervision-position masks. We will also provide the actual optimizer-step and token counts, together with the corrected checkpoint evaluations.
This protocol will be stated directly in the experimental setup and appendix. The held-out WikiText evaluation description is not a substitute for identifying the training data.
[1] Jonathan Chang, "KL Cache Trick for Full-Vocab Distillation" (2026).