- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes offline hidden-state distillation from compact teacher caches. The method stores coarsely quantized random projections of teacher hidden states together with their norms and uses a variance-reduced loss estimator. Experiments show that at 32x compression, students closely match full-state students in held-out loss and downstream accuracy, while training at the throughput of logit distillation.
Critical strengths
- The proposed method substantially reduces hidden-state cache storage while maintaining performance close to full-state distillation.
- The paper provides theoretical analysis and useful ablation studies to clarify the roles of its key components.
Critical weaknesses
-
As acknowledged in the limitations, the experiments are limited to at most 15B training tokens and a single pass over the data. It remains unclear whether the proposed method continues to match full-state distillation over longer training runs or multiple epochs. Additional experiments with longer training runs and multiple epochs would strengthen the evidence that these benefits persist beyond the evaluated setting.
-
The online teacher baseline recomputes both logits and hidden states at every training step. Although the paper discusses using cached logits and running the teacher only up to the matched intermediate layer, this setting is not evaluated. It remains unclear how much of the reported training-speed advantage would remain under this comparison.
Questions for authors
Does the proposed method also preserve the benefits of full-state distillation when applied to post-training with a pretrained student?
2 comments
Thank you for the additional experiments and clarifications. I will maintain my positive rating.
We thank the reviewer for the thoughtful comments. We are pleased that the reviewer finds the theoretical analysis and the ablation studies useful.
We have run new experiments on multiple epochs and on the truncated online teacher, and added them to the paper (Appendix C.7 and Table 9). We were not able to run longer training or post-training, and we discuss both below.
Weakness 1
Multiple epochs. With four passes over the data, the sketch stays as close to full states as with a single pass.
The student makes four passes over a 250M-token subset, for 1B training tokens in total. Every pass reuses the same cached record for each token, in a new order. Qwen3-1.7B teacher, one seed, held-out cross-entropy in nats.
Repeating the data slightly hurts every method, as expected, but the gap between our sketch and full states is the same up to noise. At four passes, reusing each token's record therefore has no measurable cost, and it needs no extra storage. We have not tested more passes than this.
Longer training. We agree that 15B tokens is small for pretraining, and we state this in our limitations. Unfortunately, due to resource constraints, we were not able to extend these experiments further.
Within the range we covered, we see no sign of the gap growing. At both teachers, a single token ratio of 1.01 fits all five checkpoints of the 32× student from 1B to 15B tokens (Appendix C.2), and at 15B the sketch is 0.001 nat behind full states, which is at the level of our seed noise. The estimate is also unbiased at every step, so we do not expect its error to accumulate over a longer run. We have not tested this beyond 15B tokens, however.
Weakness 2
We measured this baseline at both teachers. It recovers about half of the gap between an online teacher and our cache: a step is 1.24 to 1.25 times as slow as logit distillation, against 1.00 to 1.01 times for our sketch.
The teacher runs only to the matched layer at half depth, and the logits come from the same cache that our method uses. The table gives the step time on one H100 relative to logit distillation from cached logits (Table 9).
With this optimization, the cost depends on which layers are matched. If one matches at a late layer, as Muralidharan et al. (2024) do, or matches several layers, most of the teacher must still be run. The cost also grows with the size of the teacher relative to the student. In contrast, the step-time savings of our method do not depend on the layer matching strategy or on the size of the teacher. A truncated teacher must also still be held in memory beside the student during training, which a cache avoids.
Question
We have not evaluated this setting. All our students start from random initialization, and the paper's experiments are scoped to pretraining. Our guarantee does carry over, since it holds for any student state and so does not depend on how the student was initialized. Whether the training results also carry over is untested, and we have added this to our limitations.
We are grateful for the points raised by the reviewer, and we believe that they have helped us strengthen our work. We would be happy to discuss anything further.