- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces RAPID, a knowledge distillation method for LLM which partitions the teacher and student models into layers and distils layer-wise representation with random projectors. The experiments across Gemma3 and Qwen2.5 models on the OpenThoughts and SmolTalk datasets show that RAPID achieves comparable performances with the distillation baseline using learnable projectors, while reducing the number of parameters by up to 21.1%.
Critical strengths
- The paper incorporates randomized algorithm into the representation-based knowledge distillation with heterogeneous feature dimensionalities, and demonstrates its potentials under modern LLM architectures.
- The paper presents a method to bypass the learnable projectors for representation distillation, which is a substantial overhead when matching multiple activations across many layers.
- The paper presents a simple yet promising approach to distil internal knowledge more efficiently. The proposed approach with fixed random matrix is straightforward to implement.
Critical weaknesses
Major weaknesses
- Limited novelty. The core design of random projection matrix for internal distillation is similar to [1], particularly their Definition 3.1. The paper should explicitly discuss the distinction and include the comparison with this baseline.
- Unclear layer partition strategy. The paper should provide more details about the layer partition strategy for both student and teacher models. Line 216 to 221 are unclear. This is critical to the reproducibility. In addition, ablation study is required to explore how layer partition strategy affects the performance of RAPID.
- Insufficient evidence and potential overclaim for efficiency improvement. Table 2 only shows the parameter savings in comparison with learned projector. This cannot be directly referred as efficiency improvement. The paper has to include efficiency-related metrics, such as FLOPs, throughput, latency, and training time, to show that the proposed random projectors actually improve the efficiency of internal distillation.
- Limited quantitative performance. Table 1 shows that the method degrades the quantitative performance across both teacher-student pairs on the OpenThoughts dataset and achieves marginal improvements (+0.11% and +0.01%) on the SmolTalk dataset.
- Missing ablation on projection distribution. Since the core contribution of RAPID lies in the random projectors, the paper should also discuss how projection matrices from different distributions, such as normalized Gaussian distribution or random orthogonal projections, affect the distillation performance.
- Inconsistent tables. Table 2 shows the comparisons of number of parameters with Llama3.1-Llama3.2 and Olmo3-Olmo3 model pairs, while their performances are not reported in Table 1.
[1] Gupta, Shivam, and Sushrut Karmalkar. "Efficient knowledge distillation via curriculum extraction." arXiv preprint arXiv:2503.17494 (2025).
Minor weaknesses
- Insufficient coverage in related work. Distillation using intermediate representations is a longstanding approach, dating at least to FitNets in ICLR 2015. The paper does not include sufficient literature survey in its related work section.
- Limited student-teacher pairing. Experiments cover only two pairings, each within the same model family. This raises a concern about RAPID’s effectiveness in cross-family distillation and limits the applications of RAPID to a narrow audience.
- Limitation to open-weight models. Internal distillation is limited to open-weight models. The paper does not properly address the pros and cons of internal distillation and logit-based distillation.
- Presentation issues. I think many concepts and preliminary contents in Sec. 2 and 3 overlap. The author should consider merge these two sections together. Hence, the paper has extra space for ablation studies. In addition, Figure 1 does not provide an informative overview of the paper, where the core component of this work, i.e. the random projector denoted as , is not clearly explained. The notations are also inconsistent, where Figure 1 uses and the text uses and .
Questions for authors
- Why does RAPID adopt the binary projection matrix with uniform distribution instead of projectors from other distributions?
- How does RAPID react to different layer partition strategy? Does it get better or worse with more distilled activations?
- Why are Llama3 and Olmo parameter comparisons reported in Table 2, while the performances are not reported in Table 1? Is RAPID still effective in these two model pairs?
Authors' Response
We would like to sincerely thank the reviewers for spending the time and energy to carefully read and evaluate our work.
There were some consistent recommendations across the reviews which we concur would increase the strength of our paper: (1) more comprehensive ablations, (2) real-world, practical metrics for measuring efficiency, and (3) improving task performance results.
Additionally, we received useful feedback regarding other areas of the paper which could be further improved, including: clarity regarding the layer partitioning strategy, the motivation for only performing within-family distillation, and a handful of presentation issues.
In particular, we would like to thank reviewer fvJn for pointing out a mathematical error we made in the design of our objective function. We are currently working on a fix for this, as well as incorporating the rest of the feedback to improve our paper.
We again thank all of the reviewers for their time, consideration, and feedback.