acceptodds
Under review as a conference paper at ICLR 2027

RAPID: Increasing Efficiency of Internal Distillation via Randomized Methods

Abstract

Knowledge distillation is a technique used to transfer the expertise of large teacher models into small, efficient student models. Recent advances have integrated the distillation of internal activations, more directly training students to learn similar representations as the teacher. One of the core challenges of internal distillation is the dimensional incompatibility between the student and teacher internal representations. Existing methods add learnable matrices to project the larger teacher activations into the smaller representation space of the student. This is a notable point of inefficiency, as learning additional parameters for the projectors increases training costs, only for them to be discarded immediately after training. We introduce RAPID (RAndomized Projection for Internal Distillation) a novel distillation methodology which eliminates the wasteful projection learning by leveraging randomized methods. RAPID utilizes an efficient transform construction based on the Johnson-Lindenstrauss lemma to compress the teacher representations into the student embedding space with strong theoretical guarantees of structural preservation. RAPID also employs a combination of exponential moving average (EMA) and homoscedastic uncertainty weighting in a novel multi-objective function designed for internal distillation. Our experiments in post-training for reasoning demonstrate that RAPID increases training efficiency by as much as 21.1% while maintaining performance comparable to existing methods.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
machine learning, knowledge distillation, randomization, representation learning, dynamic objective

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ZaGJOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces RAPID, a knowledge distillation method for LLM which partitions the teacher and student models into layers and distils layer-wise representation with random projectors. The experiments across Gemma3 and Qwen2.5 models on the OpenThoughts and SmolTalk datasets show that RAPID achieves comparable performances with the distillation baseline using learnable projectors, while reducing the number of parameters by up to 21.1%.

Critical strengths
  1. The paper incorporates randomized algorithm into the representation-based knowledge distillation with heterogeneous feature dimensionalities, and demonstrates its potentials under modern LLM architectures.
  2. The paper presents a method to bypass the learnable projectors for representation distillation, which is a substantial overhead when matching multiple activations across many layers.
  3. The paper presents a simple yet promising approach to distil internal knowledge more efficiently. The proposed approach with fixed random matrix is straightforward to implement.
Critical weaknesses

Major weaknesses

  1. Limited novelty. The core design of random projection matrix for internal distillation is similar to [1], particularly their Definition 3.1. The paper should explicitly discuss the distinction and include the comparison with this baseline.
  2. Unclear layer partition strategy. The paper should provide more details about the layer partition strategy for both student and teacher models. Line 216 to 221 are unclear. This is critical to the reproducibility. In addition, ablation study is required to explore how layer partition strategy affects the performance of RAPID.
  3. Insufficient evidence and potential overclaim for efficiency improvement. Table 2 only shows the parameter savings in comparison with learned projector. This cannot be directly referred as efficiency improvement. The paper has to include efficiency-related metrics, such as FLOPs, throughput, latency, and training time, to show that the proposed random projectors actually improve the efficiency of internal distillation.
  4. Limited quantitative performance. Table 1 shows that the method degrades the quantitative performance across both teacher-student pairs on the OpenThoughts dataset and achieves marginal improvements (+0.11% and +0.01%) on the SmolTalk dataset.
  5. Missing ablation on projection distribution. Since the core contribution of RAPID lies in the random projectors, the paper should also discuss how projection matrices from different distributions, such as normalized Gaussian distribution or random orthogonal projections, affect the distillation performance.
  6. Inconsistent tables. Table 2 shows the comparisons of number of parameters with Llama3.1-Llama3.2 and Olmo3-Olmo3 model pairs, while their performances are not reported in Table 1.

[1] Gupta, Shivam, and Sushrut Karmalkar. "Efficient knowledge distillation via curriculum extraction." arXiv preprint arXiv:2503.17494 (2025).


Minor weaknesses

  1. Insufficient coverage in related work. Distillation using intermediate representations is a longstanding approach, dating at least to FitNets in ICLR 2015. The paper does not include sufficient literature survey in its related work section.
  2. Limited student-teacher pairing. Experiments cover only two pairings, each within the same model family. This raises a concern about RAPID’s effectiveness in cross-family distillation and limits the applications of RAPID to a narrow audience.
  3. Limitation to open-weight models. Internal distillation is limited to open-weight models. The paper does not properly address the pros and cons of internal distillation and logit-based distillation.
  4. Presentation issues. I think many concepts and preliminary contents in Sec. 2 and 3 overlap. The author should consider merge these two sections together. Hence, the paper has extra space for ablation studies. In addition, Figure 1 does not provide an informative overview of the paper, where the core component of this work, i.e. the random projector denoted as HRAPIDH_{RAPID}, is not clearly explained. The notations are also inconsistent, where Figure 1 uses HRAPIDH_{RAPID} and the text uses PϕP_\phi and HfixH_{fix}.
Questions for authors
  1. Why does RAPID adopt the binary projection matrix with uniform distribution instead of projectors from other distributions?
  2. How does RAPID react to different layer partition strategy? Does it get better or worse with more distilled activations?
  3. Why are Llama3 and Olmo parameter comparisons reported in Table 2, while the performances are not reported in Table 1? Is RAPID still effective in these two model pairs?
Reviewer_wcemOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RAPID, an internal knowledge distillation method that replaces trainable alignment projectors with fixed, scaled Rademacher random projections from teacher representations into the student’s hidden space. The experiments distill Gemma 3 4B into Gemma 3 270M and Qwen 2.5 3B into Qwen 2.5 0.5B using 40M training tokens from either OpenThoughts or SmolTalk, proving that this distillation method is more efficient than original learned projections.

Critical strengths
  1. The proposed method is simple, useful and has theoretical guarantee as the fixed projector can reduce actual resource use.
  2. The experimental results report overall averages differ by at most approximately 0.13 percentage points between RAPID and the learned-projector baseline, showing that it is unnecessary to train learned projectors.
Critical weaknesses
  1. The gain is limited or even worse from the table, especially for Qwen2.5.
  2. Equation (14) and Table 2 quantify the fraction of trainable parameters removed, without reporting end-to-end training time, throughput, or peak memory.
Questions for authors
  1. Can the authors report training throughput or end-to-end time and peak GPU memory under matched hardware, token budget, batch configuration, and precision for the residual-only and expanded-activation settings?
  2. Can the authors trained models with Qwen3.5-9B or much larger to show the effectiveness?
Reviewer_fvJnOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a framework for internal distillation and addresses the problem of teacher-student internal dimensional incompatibility. To address this, it uses fixed random downward projections from the teacher’s dimension to the students. The projection approach is based on the Johnson-Lindenstrauss Lemma. The fixed randomized downward projections eliminate the need of training learnable upward projectors, and therefore, reduce training costs. The framework also proposes a multi-objective loss function combining exponential moving average (EMA) and homoscedastic uncertainty weighting for distillation.

Critical strengths

A simple framework is proposed to reduce trainable overhead in internal distillation. The established geometric-preservation properties of random projections provide a principled motivation for this design.

RAPID preserves benchmark performance across 4 evaluation tasks in 2 model families and 2 datasets compared to learned projectors showing consistency across tested configurations.

Critical weaknesses

In the multi-objective loss function, it’s unclear how much gains are from SFT and output-distribution distillation. It would provide a useful baseline to understand the performance gains of both learnable projectors and RAPID.

The learned projector is trained using an equally-weighted objective. However, RAPID uses dynamic weighted objectives. It would be a useful ablation to include dynamic weighting for the learned projector and use an equally-weighted objective for RAPID.

I am not sure if I found a proper quantification of practical efficiency benefit. I think if the authors provide training throughput, GPU memory usage and training time change, it would help. Does RAPID reach decent performance faster compared to learned projectors?

Will reducing the number of layers for distillation have a similar impact for both learned projectors and RAPID? I think obtaining similar performance with fewer layers (eg. 3 layers) can impact the training efficiency gains.

Questions for authors

Can this be extended to different model families?

Section 4.4 and Appendix D state that detaching the weights gives (\partial L_{\mathrm{total}}/\partial s_i=1). How can (s_i) learn task-specific uncertainty under this constant gradient? Under ordinary gradient descent, this update decreases each active (s_i), so why does Appendix D says that it eliminates downward drift? Does the implementation use an additional update rule that I cannot find in the paper?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Authors' Response

We would like to sincerely thank the reviewers for spending the time and energy to carefully read and evaluate our work.

There were some consistent recommendations across the reviews which we concur would increase the strength of our paper: (1) more comprehensive ablations, (2) real-world, practical metrics for measuring efficiency, and (3) improving task performance results.

Additionally, we received useful feedback regarding other areas of the paper which could be further improved, including: clarity regarding the layer partitioning strategy, the motivation for only performing within-family distillation, and a handful of presentation issues.

In particular, we would like to thank reviewer fvJn for pointing out a mathematical error we made in the design of our objective function. We are currently working on a fix for this, as well as incorporating the rest of the feedback to improve our paper.

We again thank all of the reviewers for their time, consideration, and feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.