- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes EvaKD, an intermediate-layer knowledge distillation method for LLMs. It first does sparse prediction projection that passes intermediate hidden states of the teacher and student through each model's own final norm and LM head, restricted to a per-token candidate vocabulary. With such projection from each layer, it describes how each model's prediction changes from layer to layer, and uses optimal transport to find a soft many-to-many alignment between teacher and student stages. The alignment is further filtered by keeping only the teacher signals that push the student toward the teacher's final prediction. The filtered alignment weights an extra intermediate distillation loss, added on top of standard fine-tuning and KD. Experiments that distill Llama3.1-8B→Llama3.2-1B and Qwen3-4B→Qwen3-0.6B show that EvaKD outperform baselines
Critical strengths
- The paper proposes a simple yet effective method to utilize the information from intermediate layers, which does not involve extra trainable parameters. The presentation is easy to follow.
- The authors conduct comprehensive experiments and ablation studies to demonstrate the effectiveness of the proposed method.
Critical weaknesses
- One major concern is that the evolution step used in practice. From Sec 3.4, I assume the evolution step is layer-wise. However, according to Figure 4, there are only 4 evolution steps for both teacher and student, which contradicts the description in the method section. From Appendix B.4, the authors mentions "we use four representative evolution stages for both the teacher and student. ". But these 4 evolution stages are not clearly defined.
a. Does it simply divided layers into 4 chunks and use 4 4 transport mass instead of using ? If so, layer position is already pre-aligned before OT. Combined with uniform marginals and the depth cost, this may make PEOT close to a smoothed proportional mapping, which is the kind of predefined correspondence the paper aims to avoid.
b. How is 4 chosen? In other words, how would the number of stages affect the performance?
c. Besides, ablation studies on the evolution pairs is under-explored. For example, what is the performance if we only focus the teacher-student prediction discrepancy of the last K layers?
-
In the experiment, the authors use LoRA rank=8 for student training. However, it could greatly limit the learning ability given that the student model size is small. Would the conclusion still hold when full-parameter finetuning?
-
The experiment in the main text focus on the same-tokenizer same-architecture setting. For Llama3.1-8B→Qwen3-0.6B, Appendix C.3 compares EvaKD only against vanilla KD and gives a marginal gain (+0.34). So applicability beyond same-family pairs is weakly supported. I wonder if the method is applicable to MoE-to-dense setting, e.g. Qwen3-MoE to Qwen3 dense model.
Questions for authors
- See weakness 1,2,3.
- In the ablation, w/o PEOT I assume is just using SFT and KD loss. But the performance does not match the KD row in Table 1, and even lower under Qwen3-4B → Qwen3-0.6B (29.54 vs 31.55). What is w/o PEOT exactly and why this happens?
6 comments
Response to Reviewer 1YAh (PART II)
Furthermore, we reuse the depth-cost sensitivity results from our response to Reviewer 33UR (Table C) to demonstrate that the evolution cost provides an independent contribution to the learned correspondence. Table C further quantifies the depth term's contribution to evolution-aware cross-model alignment.
Table C: Sensitivity analysis of the depth-cost coefficient (Average ROUGE-L).
| Llama3.2-1B | Qwen3-0.6B | |
|---|---|---|
| 0 | 28.14 | 31.47 |
| 0.1 | 28.84 | 31.63 |
| 0.5 | 29.12 | 32.18 |
| 1.0 (Default) | 29.71 | 32.85 |
| 2.0 | 29.37 | 31.91 |
When the depth cost is completely removed (), PEOT still achieves 28.14 and 31.47 average ROUGE-L, respectively. This indicates that prediction-evolution similarity itself provides meaningful alignment signals, rather than the transport plan being determined solely by relative depth. The default () further achieves the best performance on both model pairs.
Third, regarding the choice of four evolution stages, we agree that this choice should be empirically justified. We therefore reuse the stage-number sensitivity experiments from our response to Reviewer 8VJP (Table D), comparing different numbers of representative stages while keeping the remaining configurations unchanged. The all-stage setting covers all adjacent-layer transitions among Transformer layers 1 through .
Table D: Sensitivity analysis of the number of evolution stages (Average ROUGE-L).
| Number of Teacher / Student Stages | Llama3.2-1B | Qwen3-0.6B |
|---|---|---|
| 2 / 2 | 28.96 | 31.87 |
| 4 / 4 (Default) | 29.71 | 32.85 |
| 8 / 8 | 29.53 | 32.79 |
| All / All () | 29.64 | 32.76 |
The results show that 4/4 stages achieve the best performance on both model pairs. Increasing the number of stages to 8 or using all layer-wise evolution transitions does not yield further improvements, while using only two stages results in noticeable performance degradation. These results suggest that four representative stages provide an effective balance between evolution coverage and computational efficiency, rather than being an arbitrarily selected configuration.
Finally, regarding the contribution of late-layer evolution, we further conduct a suffix-only ablation to investigate whether the improvements can be obtained by exclusively transferring knowledge from the final portions of the teacher and student networks.
Specifically, we retain only the last 25%, 50%, or 75% of the layer-wise evolution transitions between Transformer layers 1 and in each model, independently reconstruct the corresponding PEOT cost matrix, and apply DAT to the resulting transport plan. The original normalized depth coordinates are preserved, and all other training configurations remain unchanged.
Table E: Ablation of late-layer evolution distillation (Average ROUGE-L).
| Evolution Range | Llama3.2-1B | Qwen3-0.6B |
|---|---|---|
| Last 25% | 28.72 | 31.88 |
| Last 50% | 28.91 | 32.03 |
| Last 75% | 29.06 | 32.21 |
| Last 100% (All transitions) | 29.64 | 32.76 |
| Four representative stages (EvaKD) | 29.71 | 32.85 |
The results show that performance consistently improves as the evolution range expands from the last 25% to all layer-wise transitions, increasing from 28.72 to 29.64 on Llama3.2-1B and from 31.88 to 32.76 on Qwen3-0.6B. Moreover, the four-stage configuration achieves the best results, outperforming all-transition supervision by +0.07 and +0.09 points, respectively. These findings suggest that useful prediction evolution is distributed across network depths, while representative-stage aggregation provides a more effective supervision granularity. We will incorporate these results into the revised appendix.
Overall, these experiments clarify the stage construction, the flexibility of cross-model correspondence, and the impact of evolution-stage selection. We sincerely thank the reviewer for these helpful suggestions.
Response to Reviewer 1YAh (PART III)
W2: Effect of Full-Parameter Fine-Tuning
Response. We appreciate this concern. Since LoRA may constrain a small student's adaptation capacity, full-parameter fine-tuning is necessary to test whether EvaKD's gains depend on parameter-efficient optimization. We therefore compare KD and EvaKD under both strategies.
From a methodological perspective, EvaKD modifies the distillation objective rather than the parameterization of the student model. Specifically, PEOT establishes evolution-aware correspondence between teacher and student predictions, while DAT selectively transfers teacher evolution according to the student's remaining prediction deficiency.
Neither component introduces additional trainable projection modules or relies on LoRA-specific operations. Consequently, the proposed supervision mechanism is compatible with both parameter-efficient and full-parameter optimization. Thus, the same PEOT and DAT objectives supervise LoRA-adapted and fully trainable students.
To investigate this issue, we additionally compare KD and EvaKD under full-parameter fine-tuning, using the original LoRA-based results as references. Both methods use identical training data and optimization configurations within each training strategy.
Table F: Comparison between LoRA and full-parameter fine-tuning (Average ROUGE-L).
| Model Pair | Training Strategy | KD | EvaKD | Improvement |
|---|---|---|---|---|
| Llama3.1-8B → Llama3.2-1B | LoRA (r=8) | 22.87 | 29.71 | +6.84 |
| Full-Parameter FT | 22.94 | 29.83 | +6.89 | |
| Qwen3-4B → Qwen3-0.6B | LoRA (r=8) | 31.55 | 32.85 | +1.30 |
| Full-Parameter FT | 31.61 | 32.93 | +1.32 |
The results show that EvaKD consistently outperforms KD under both training strategies. Under full-parameter fine-tuning, EvaKD achieves 29.83 and 32.93 average ROUGE-L on the two model pairs, outperforming KD by +6.89 and +1.32 points, respectively. These gains are comparable to those under LoRA (+6.84 and +1.30), demonstrating that EvaKD's effectiveness does not rely on parameter-efficient fine-tuning and remains consistent when all student parameters are trainable. We will include these comparisons and clarify the compatibility of EvaKD with different student fine-tuning strategies in the revised manuscript.
W3: Applicability to Heterogeneous Architectures and MoE-to-Dense Distillation
Response. We thank the reviewer for pointing out the limited experimental evidence for heterogeneous architecture settings.
In Appendix C.3, we evaluate distillation from Llama3.1-8B to Qwen3-0.6B. EvaKD improves the average ROUGE-L from 30.83 to 31.17, yielding a gain of +0.34 over vanilla KD. Although this result demonstrates that EvaKD can operate across different model families, we acknowledge that further evaluation is needed to establish its applicability to different architectural configurations, particularly MoE-to-Dense distillation.
From a methodological perspective, EvaKD does not require identical internal architectures or expert-level correspondence. Its prediction-evolution representation is constructed by projecting each model's intermediate hidden states through its own normalization module and LM head. For an MoE teacher, the expert routing and expert computations are already incorporated into the resulting hidden states. These hidden states can therefore be projected into the prediction space in the same manner as those of a dense model.
Consequently, EvaKD does not need to align teacher and student experts, expert indices, or routing decisions. Instead, PEOT operates on their prediction changes, while DAT evaluates whether the transferred teacher evolution complements the student's remaining prediction deficiency.
To further investigate this setting, we conduct an additional experiment using Qwen3-30B-A3B (MoE) as the teacher and Qwen3-0.6B (Dense) as the student. We follow the same domain-agnostic evaluation protocol as the original experiments.
Table G: Results of MoE-to-Dense distillation on domain-agnostic benchmarks (ROUGE-L).
| Method | Dolly | SelfInst | S-NI | UnNI | Vicuna | Avg. |
|---|---|---|---|---|---|---|
| KD | 28.47 | 22.87 | 43.58 | 43.61 | 19.24 | 31.55 |
| EvaKD | 29.39 | 24.43 | 43.48 | 45.45 | 20.92 | 32.73 |
For the MoE-to-Dense setting, EvaKD achieves 32.73 average ROUGE-L, outperforming vanilla KD (31.55) by +1.18 points. It consistently improves performance on four out of five benchmarks. These results provide further empirical evidence that EvaKD generalizes to heterogeneous architectures involving sparse-expert teachers and dense students.
Response to Reviewer 1YAh (PART I)
Comment: We sincerely thank the reviewer for the positive assessment of EvaKD, particularly its simple yet effective design and comprehensive evaluation. We appreciate the insightful concerns regarding the practical construction of evolution stages, the influence of parameter-efficient fine-tuning, and the applicability of EvaKD to heterogeneous architectures. We address each weakness and question separately below.
Response. We appreciate the reviewer for pointing out the insufficient clarification of the four representative evolution stages. We agree that the original manuscript should better distinguish the general layer-wise formulation from its practical implementation.
First, regarding the construction of the four evolution stages, the general formulation in Sec. 3.4 characterizes prediction evolution through changes between consecutive Transformer-layer outputs. For computational efficiency, our implementation aggregates these layer-wise prediction changes into four representative depth intervals for both the teacher and student, excluding the embedding output.
Specifically, let LX denote the number of Transformer layers in model X. We select five depth boundaries independently for each model:
m_r^{\mathcal X}=\left\lfloor 1+\frac{r(L^{\mathcal X}-1)}{4}\right\rfloor, \quad r=1,\ldots,4.$$ The prediction evolution of the (r)-th representative stage is then defined as: $\Delta \mathbf z_{t,r}^{\mathcal X} = \mathbf z_{t,m_{r}^{\mathcal X}}^{\mathcal X} - \mathbf z_{t,m_{r-1}^{\mathcal X}}^{\mathcal X}.$ Here, node 1 denotes the output of the first Transformer layer, while node $L^{\mathcal X}$ denotes the final Transformer-layer output. The embedding output is excluded. Each representative evolution vector therefore captures the cumulative prediction change between the two endpoint Transformer layers of its interval, rather than the change between only two immediately adjacent layers. The exact selected depth intervals are summarized below. > **Table A: Selected evolution stages in the two main teacher-student pairs.** | Model Pair | Model | Selected Depth Intervals | | ------------------------- | ------- | ------------------------ | | Llama3.1-8B → Llama3.2-1B | Teacher | 1→8, 8→16, 16→24, 24→32 | | | Student | 1→4, 4→8, 8→12, 12→16 | | Qwen3-4B → Qwen3-0.6B | Teacher | 1→9, 9→18, 18→27, 27→36 | | | Student | 1→7, 7→14, 14→21, 21→28 | **Second, regarding whether uniform depth partition introduces predefined cross-model alignment**, we clarify that the partition only determines the evolution units within each individual model. It does not establish any teacher-student correspondence in advance. Specifically, PEOT independently calculates the evolution cost for all 16 teacher-student stage pairs and optimizes a $4\times4$ transport plan based on both prediction-evolution similarity and normalized depth difference. Although uniform marginals constrain the total transport mass of each row and column, they do not impose a diagonal or one-to-one correspondence. The resulting transport plan can therefore distribute mass across different teacher and student stages. We further reuse the fixed-mapping comparison from our response to Reviewer 33UR (Table B), which directly isolates the contribution of flexible evolution correspondence. > **Table B: Comparison between fixed relative-depth mapping and PEOT (ROUGE-L %).** | Llama3.1-8B → Llama3.2-1B | Dolly | SelfInst | S-NI | UnNI | Vicuna | Avg. | | ------------------------------- | --------- | ------------ | --------- | --------- | ---------- | --------- | | Fixed relative-depth (w/o PEOT) | 28.46 | 21.32 | 36.43 | 36.55 | 16.36 | 27.82 | | **PEOT (EvaKD)** | **30.02** | **23.23** | **38.61** | **38.19** | **18.49** | **29.71** | | **Qwen3-4B → Qwen3-0.6B** | **Dolly** | **SelfInst** | **S-NI** | **UnNI** | **Vicuna** | **Avg.** | | Fixed relative-depth (w/o PEOT) | 27.57 | 19.72 | 38.79 | 41.58 | 20.05 | 29.54 | | **PEOT (EvaKD)** | **29.56** | **24.77** | **43.07** | **45.63** | **21.23** | **32.85** | As shown in Table B, PEOT improves average ROUGE-L by **+1.89** and **+3.31** points over fixed relative-depth mapping on the two model pairs, respectively. These improvements demonstrate that the gains are not simply obtained from introducing uniformly partitioned intermediate supervision.