acceptodds
Under review as a conference paper at ICLR 2027

Offline Hidden-State Distillation from Compact Teacher Caches

Abstract

For language models, logit distillation is now routinely performed offline: the teacher's next-token distributions are computed once, ahead of training, and then cached for use in subsequent training runs. Storing the full distributions would be prohibitive, but truncation or sampling compresses the cache by orders of magnitude without significantly impacting performance. Hidden-state distillation, which additionally uses the teacher's hidden states to supervise the student's intermediate computation, has no compact offline counterpart. Existing methods either perform teacher inference within the training loop, slowing down training significantly, or require enormous amounts of storage. We show that a per-token random projection of the teacher's hidden states, coarsely quantized, lets hidden-state distillation run from a cache of the same order of size as the logit cache, at the training throughput of logit distillation. With the hidden states compressed 32, students distilled from four teachers of 1.5B to 8B parameters need at most about 2% more training tokens than students trained on the full hidden states to reach the same held-out loss at 1B tokens. Over training runs of up to 15B tokens, they need about 1% more tokens than the full-state students and match their downstream accuracy, while a student distilled from logits alone needs 1.4 to 1.7 times as many.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Knowledge distillation, hidden-state distillation, feature distillation, offline distillation, teacher caching, random projections, activation compression, quantization, unbiased estimation, language model pre-training, efficient training

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_qJCKOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes offline hidden-state distillation from compact teacher caches. The method stores coarsely quantized random projections of teacher hidden states together with their norms and uses a variance-reduced loss estimator. Experiments show that at 32x compression, students closely match full-state students in held-out loss and downstream accuracy, while training at the throughput of logit distillation.

Critical strengths
  1. The proposed method substantially reduces hidden-state cache storage while maintaining performance close to full-state distillation.
  2. The paper provides theoretical analysis and useful ablation studies to clarify the roles of its key components.
Critical weaknesses
  1. As acknowledged in the limitations, the experiments are limited to at most 15B training tokens and a single pass over the data. It remains unclear whether the proposed method continues to match full-state distillation over longer training runs or multiple epochs. Additional experiments with longer training runs and multiple epochs would strengthen the evidence that these benefits persist beyond the evaluated setting.

  2. The online teacher baseline recomputes both logits and hidden states at every training step. Although the paper discusses using cached logits and running the teacher only up to the matched intermediate layer, this setting is not evaluated. It remains unclear how much of the reported training-speed advantage would remain under this comparison.

Questions for authors

Does the proposed method also preserve the benefits of full-state distillation when applied to post-training with a pretrained student?

2 comments

We thank the reviewer for the thoughtful comments. We are pleased that the reviewer finds the theoretical analysis and the ablation studies useful.

We have run new experiments on multiple epochs and on the truncated online teacher, and added them to the paper (Appendix C.7 and Table 9). We were not able to run longer training or post-training, and we discuss both below.

Weakness 1

As acknowledged in the limitations, the experiments are limited to at most 15B training tokens and a single pass over the data. It remains unclear whether the proposed method continues to match full-state distillation over longer training runs or multiple epochs. Additional experiments with longer training runs and multiple epochs would strengthen the evidence that these benefits persist beyond the evaluated setting.

Multiple epochs. With four passes over the data, the sketch stays as close to full states as with a single pass.

The student makes four passes over a 250M-token subset, for 1B training tokens in total. Every pass reuses the same cached record for each token, in a new order. Qwen3-1.7B teacher, one seed, held-out cross-entropy in nats.

four passes over 250M tokenssingle pass over 1B tokens (Table 2)
logits only2.4542.428
full states2.3502.336
our 32× sketch2.3522.337
sketch − full states+0.0020+0.0018

Repeating the data slightly hurts every method, as expected, but the gap between our sketch and full states is the same up to noise. At four passes, reusing each token's record therefore has no measurable cost, and it needs no extra storage. We have not tested more passes than this.

Longer training. We agree that 15B tokens is small for pretraining, and we state this in our limitations. Unfortunately, due to resource constraints, we were not able to extend these experiments further.

Within the range we covered, we see no sign of the gap growing. At both teachers, a single token ratio of 1.01 fits all five checkpoints of the 32× student from 1B to 15B tokens (Appendix C.2), and at 15B the sketch is 0.001 nat behind full states, which is at the level of our seed noise. The estimate is also unbiased at every step, so we do not expect its error to accumulate over a longer run. We have not tested this beyond 15B tokens, however.

Weakness 2

The online teacher baseline recomputes both logits and hidden states at every training step. Although the paper discusses using cached logits and running the teacher only up to the matched intermediate layer, this setting is not evaluated. It remains unclear how much of the reported training-speed advantage would remain under this comparison.

We measured this baseline at both teachers. It recovers about half of the gap between an online teacher and our cache: a step is 1.24 to 1.25 times as slow as logit distillation, against 1.00 to 1.01 times for our sketch.

The teacher runs only to the matched layer at half depth, and the logits come from the same cache that our method uses. The table gives the step time on one H100 relative to logit distillation from cached logits (Table 9).

methodQwen3-1.7B → 0.77BQwen3-4B → 1.65B
logits only (cached)1.00×1.00×
our 32× sketch (cached)1.01×1.00×
truncated online teacher, cached logits1.24×1.25×
full online teacher1.54×1.58×

With this optimization, the cost depends on which layers are matched. If one matches at a late layer, as Muralidharan et al. (2024) do, or matches several layers, most of the teacher must still be run. The cost also grows with the size of the teacher relative to the student. In contrast, the step-time savings of our method do not depend on the layer matching strategy or on the size of the teacher. A truncated teacher must also still be held in memory beside the student during training, which a cache avoids.

Question

Does the proposed method also preserve the benefits of full-state distillation when applied to post-training with a pretrained student?

We have not evaluated this setting. All our students start from random initialization, and the paper's experiments are scoped to pretraining. Our guarantee does carry over, since it holds for any student state and so does not depend on how the student was initialized. Whether the training results also carry over is untested, and we have added this to our limitations.

We are grateful for the points raised by the reviewer, and we believe that they have helped us strengthen our work. We would be happy to discuss anything further.

Thank you for the additional experiments and clarifications. I will maintain my positive rating.

Reviewer_GLMpOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes offline hidden-state distillation from a compact cache. Each teacher hidden state is projected with a subsampled randomized Hadamard transform, with column signs randomly redrawn per token. The projection is then quantized to 3 bits and stored. During distillation, the student hidden states are mapped to the teacher space and projected with the same transform, and the cosine loss is estimated from the sketched residual between the two. At 32x compression, the students' downstream accuracy mathces that of the full-state students, while training at close to the speed of logit distillation.

Critical strengths
  • The method seems novel, and the design choice of the algorithm is theoretically well justified.
  • The method is effective, and the performance at 32x compression is almost equal to the full state distillation.
Critical weaknesses
  • It would be better to also show results where logit distillation involves the ground-truth CE loss. Since the CE term often improves the overall performance, the marginal benefit of hidden state supervision may be smaller than it currently seems under that setting.
  • The main experiments primarily perform single-layer supervision (middle-to-middle). There are various ways layers of the student and teacher can be matched, but the authors only briefly discuss this and never test multi-layer supervision.
  • The training budget is quite limited. the longest training run is on 15B tokens, which is rather small for a pretraining experiment.
Questions for authors

Could you please provide additional experimental results raised in the weaknesses section?

2 comments

We thank the reviewer for the constructive feedback. We are glad that the reviewer finds the method effective and its design well justified by the theory.

We have run new experiments for the first two weaknesses and added them to the paper (Appendix C.7). We respond to the third below.

Weakness 1

It would be better to also show results where logit distillation involves the ground-truth CE loss. Since the CE term often improves the overall performance, the marginal benefit of hidden state supervision may be smaller than it currently seems under that setting.

In our setting, the CE term does not reduce the benefit of hidden-state supervision, which stays between 0.14 and 0.16 nat at every weight we tried.

We added the ground-truth cross-entropy with a weight β\beta, and swept β\beta for both logit distillation and full-state distillation. We trained our 32× sketch at β=0.1\beta = 0.1. Qwen3-1.7B teacher, 300M training tokens (the setting of Table 1), one seed, held-out cross-entropy in nats. The β=0\beta=0 row is taken from Table 1.

β\betalogits onlyfull statesour 32× sketchbenefit kept by the sketch
02.7052.5632.56896%
0.12.7042.5632.56599%
0.32.7102.565––
12.7562.596––
32.8182.662––

A small weight leaves both methods unchanged within noise, and larger weights hurt both. This agrees with recent work on distilling large language models (Busbridge et al., 2025; Muralidharan et al., 2024). With the CE term, the sketch stays as close to full states as without it.

Weakness 2

The main experiments primarily perform single-layer supervision (middle-to-middle). There are various ways layers of the student and teacher can be matched, but the authors only briefly discuss this and never test multi-layer supervision.

Matching three layers improves on one layer by 0.023 nat with full states, and our sketch keeps that improvement.

We matched three layers, placed as close as possible to one sixth, one half and five sixths of the depth in both models, and kept the total weight of the hidden-state term the same as with one layer. Same setting as above, one seed. Logit distillation alone reaches 2.705, and the one-layer row is taken from Table 1.

matched layersfull statesour sketch, 32×benefit keptour sketch, 128×benefit kept
one2.5632.56896%2.58684%
three2.5402.54398%2.56187%

With three layers the sketch retains at least as much of the full-state benefit as with one layer, at both compression levels. The differences of 2 to 3 points are too small to distinguish from seed noise with a single run. Multi-layer supervision also need not cost more storage: three layers at 128× take 90 bytes per token, less than one layer at 32× (130 bytes), and reach a lower loss (2.561 against 2.568 nat).

Weakness 3

The training budget is quite limited. the longest training run is on 15B tokens, which is rather small for a pretraining experiment.

We agree that 15B tokens is small for pretraining, and we state this in our limitations. Unfortunately, due to resource constraints, we were not able to extend these experiments further.

Within the range we covered, we see no sign of the gap growing. At both teachers, a single token ratio of 1.01 fits all five checkpoints of the 32× student from 1B to 15B tokens (Appendix C.2), and at 15B the sketch is 0.001 nat behind full states, which is at the level of our seed noise. The estimate is also unbiased at every step, so we do not expect its error to accumulate over a longer run. We have not tested this beyond 15B tokens, however.

We are grateful for the feedback, and we believe the suggested experiments have strengthened our work. We would be glad to clarify anything further.

Dear authors,

thank you for your thoughtful response. My remaining concerns are largely resolved. While it is unfortunate that the training budget is overall quite limited, I think the experimental results sufficiently suggest the strength of the proposed method. I believe the paper is worth being presented at ICLR, and I will maintain my positive score of 3.

Reviewer_i6tROpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper is about hidden-state distillation for language models, using a caching mechanism. The idea is to replace the complete storage of the teacher’s hidden states with a very compact summary, which is nevertheless sufficient to compute the distillation loss. The compact summary is built from a low-dimensional random projection of the teacher state. The paper introduces a clear formalization of the proposed distillation scheme and some theoretical guarantees. An experimental validation is conducted on several teacher/student models.

Critical strengths
  • The paper is well written and well motivated.
  • The formalization of the proposed approach is also clear and sound
  • The proposed approach is positioned in an original way at the intersection of several lines of work: Hidden layer distillation, offline KD, and cache compression.
  • The main claim has some theoretical guarantees given in the paper: the loss computed from the sketch is an unbiased estimator of the true loss, and its variance is controlled.
  • The experimental validation is well done, with 4 Qwen teachers showing the interest of the proposed method, followed by a detailed discussion of the results.
Critical weaknesses
  • The main technical novelty of the paper should be discussed more specifically compared to Guigon et al. 2026) or Ma et al., 2026.
  • The dependence of the proposed result on the chosen KD loss should be discussed.
Questions for authors
  • How does the paper compare to Ma et al., 2026?
  • What about teacher and student models with different vocabularies?
  • What about other KD losses, such as contrastive KDs, for instance?
1 comment

We thank the reviewer for the careful review and for the questions. We are encouraged that the reviewer finds the positioning original, the formalization clear and sound, and the experimental validation well done.

We have expanded the comparison with prior work and added new experiments with two other losses (Section 4, Appendices B and C.7).

Weakness 1 and Question 1

The main technical novelty of the paper should be discussed more specifically compared to Guigon et al. 2026 or Ma et al., 2026 (https://openreview.net/pdf?id=IcVSKhVpKu).

The link provided opens Dasgupta & Cohn (ICLR 2025) rather than Ma et al. (2026). In case either was intended, we address both, along with Guigon et al. The revised Section 4 has the full comparison.

  • Guigon et al. cache one teacher layer uncompressed. They note that the truncation that makes logit caches compact cannot be applied to hidden states, and they give storage as one reason for matching a single layer. Our technical contribution is the compressed record that was missing: a residual estimator with exact norms, per-token random signs, and dithered 3-bit quantization, which together give an unbiased estimate of the loss whose projection noise shrinks with the loss. At 32× compression of such a cache, our students need 1 to 2% more training tokens than students trained on full states, and match their downstream accuracy (Tables 2 and 3).
  • Ma et al. (ForeTime-VLA) cache a learned 64-dimensional code of pooled video features, which serves as a new target for a robot policy. Our record differs in three ways: it is a training-free random projection, it gives an unbiased estimate of the loss on the full hidden state with a bound on its variance, and it targets per-token hidden states of language models, where storage is the obstacle.
  • Dasgupta & Cohn replace the learned map and the per-token loss of hidden-state distillation with a CKA loss between the two models' states over a batch. Their work changes which loss is matched, while ours changes how the teacher's targets are stored, so the two are complementary. We now cite it, and in one experiment our sketch tracked full states under an adaptation of their loss, below.

Weakness 2 and Question 3

The dependence of the proposed result on the chosen KD loss should be discussed.

What about other KD losses, such as contrastive KDs, for instance?

Proposition 1 is stated for the cosine loss, but its argument applies to any loss that is a squared Euclidean distance between suitably scaled student and teacher states (new paragraph in Appendix B). Contrastive losses are outside this family: computed from our sketch, they would be biased. We have not tested how much this bias matters in training, so this remains future work.

We ran new experiments with two other losses, one inside the family and one outside it:

  • Normalized MSE, where both states are divided by the norm of the teacher's state. This loss is covered by the guarantee.
  • An adaptation of the CKA loss of Dasgupta & Cohn to our setting: computed on states normalized to unit length, with a constant weight of 1. This loss is not covered by the guarantee, so this is an empirical test only.

Without these normalizations, both losses did worse than logit distillation alone in our setting, even with full hidden states.

Qwen3-1.7B teacher, 300M training tokens (the setting of Table 1), one seed, held-out cross-entropy in nats. Logit distillation alone reaches 2.705.

hidden-state lossfull statesour 32× sketchgap (nat)benefit kept by the sketch
cosine (Table 1)2.5632.5680.00596%
normalized MSE2.5822.5850.00397%
CKA2.6892.6900.00194%

The sketch keeps essentially the same share of the benefit with normalized MSE as with cosine. With CKA the sketch also tracks full states. In our adaptation, however, CKA improves on logit distillation by only 0.016 nat, against 0.142 nat for cosine, and the 0.001 nat gap is at the level of our seed noise, so its percentage is a weak measurement. This reflects our adaptation to a pre-norm decoder-only teacher, and is not a statement about their method in its original setting. Details are in Appendix C.7.

Question 2

What about teacher and student models with different vocabularies?

Like prior work on hidden-state distillation (DistilBERT; TinyBERT; Minitron), we assume that the student shares the teacher's tokenizer, because our loss compares hidden states position by position. Distillation across tokenizers requires an alignment between the two sequences, which is a challenge of its own that we have not studied. Like a cache of teacher logits, our cache is tied to the teacher's tokenization. We have added this to our limitations.

We thank the reviewer again, and we are happy to answer any further questions.

Comments on the paper

1 comment

Dear Reviewers,

The authors have posted their responses to the reviews. Please go through their rebuttal at your earliest convenience, and engage in a discussion or leave any follow-up comments.

The discussion period ends on November 18.

Thanks,
Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.