acceptodds
Under review as a conference paper at ICLR 2027

Evolution-Aware Distillation via Optimal Transport for Large Language Models

Abstract

Knowledge distillation (KD) has become a widely adopted approach for compressing large language models (LLMs) for efficient deployment and inference. Existing LLM distillation methods can be broadly categorized into black-box and white-box paradigms. Black-box distillation transfers teacher-generated outputs but cannot fully exploit internal knowledge, while white-box distillation leverages logits or intermediate representations for finer-grained knowledge transfer. However, existing intermediate distillation methods usually rely on static layer-wise alignment and still suffer from two major limitations. First, predefined layer correspondence ignores that teacher and student predictions may evolve at different rates across model depth, causing layers at mismatched evolution stages to be aligned. Second, even with appropriate correspondence, indiscriminately transferring teacher evolution signals overlooks whether such knowledge actually complements the deficiencies of the student model. To address these issues, we propose an **ev**olution-**a**ware **k**nowledge **d**istillation framework (**EvaKD**) via optimal transport for LLMs, which dynamically aligns prediction evolution between teacher and student models and selectively transfers beneficial knowledge. Specifically, first, to overcome rigid layer alignment, we introduce prediction-evolution optimal transport, which projects intermediate representations into the prediction space, characterizes depthwise evolution through prediction changes between adjacent states, and establishes a soft many-to-many correspondence between teacher and student evolution via optimal transport. Second, to avoid indiscriminate knowledge transfer, we further propose deficiency-aware truncated distillation, which evaluates transported teacher evolution according to the remaining prediction deficiency of the student and retains only transport signals that contribute to closing this deficiency. Extensive experiments across multiple benchmarks and LLM families show that our method consistently outperforms strong LLM distillation baselines, improving Rouge-L scores by up to 7.01% and accuracy scores by 7.4%. Our code is at https://anonymous.4open.science/r/EvaKD-CFCF/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Knowledge Distillation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_1YAhOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes EvaKD, an intermediate-layer knowledge distillation method for LLMs. It first does sparse prediction projection that passes intermediate hidden states of the teacher and student through each model's own final norm and LM head, restricted to a per-token candidate vocabulary. With such projection from each layer, it describes how each model's prediction changes from layer to layer, and uses optimal transport to find a soft many-to-many alignment between teacher and student stages. The alignment is further filtered by keeping only the teacher signals that push the student toward the teacher's final prediction. The filtered alignment weights an extra intermediate distillation loss, added on top of standard fine-tuning and KD. Experiments that distill Llama3.1-8B→Llama3.2-1B and Qwen3-4B→Qwen3-0.6B show that EvaKD outperform baselines

Critical strengths
  1. The paper proposes a simple yet effective method to utilize the information from intermediate layers, which does not involve extra trainable parameters. The presentation is easy to follow.
  2. The authors conduct comprehensive experiments and ablation studies to demonstrate the effectiveness of the proposed method.
Critical weaknesses
  1. One major concern is that the evolution step used in practice. From Sec 3.4, I assume the evolution step is layer-wise. However, according to Figure 4, there are only 4 evolution steps for both teacher and student, which contradicts the description in the method section. From Appendix B.4, the authors mentions "we use four representative evolution stages for both the teacher and student. ". But these 4 evolution stages are not clearly defined.

a. Does it simply divided layers into 4 chunks and use 4 ×\times 4 transport mass instead of using NT×NSN_T \times N_S? If so, layer position is already pre-aligned before OT. Combined with uniform marginals and the depth cost, this may make PEOT close to a smoothed proportional mapping, which is the kind of predefined correspondence the paper aims to avoid.

b. How is 4 chosen? In other words, how would the number of stages affect the performance?

c. Besides, ablation studies on the evolution pairs is under-explored. For example, what is the performance if we only focus the teacher-student prediction discrepancy of the last K layers?

  1. In the experiment, the authors use LoRA rank=8 for student training. However, it could greatly limit the learning ability given that the student model size is small. Would the conclusion still hold when full-parameter finetuning?

  2. The experiment in the main text focus on the same-tokenizer same-architecture setting. For Llama3.1-8B→Qwen3-0.6B, Appendix C.3 compares EvaKD only against vanilla KD and gives a marginal gain (+0.34). So applicability beyond same-family pairs is weakly supported. I wonder if the method is applicable to MoE-to-dense setting, e.g. Qwen3-MoE to Qwen3 dense model.

Questions for authors
  1. See weakness 1,2,3.
  2. In the ablation, w/o PEOT I assume is just using SFT and KD loss. But the performance does not match the KD row in Table 1, and even lower under Qwen3-4B → Qwen3-0.6B (29.54 vs 31.55). What is w/o PEOT exactly and why this happens?
6 comments
AuthorsOpenReview ↗

Response to Reviewer 1YAh (PART I)

Comment: We sincerely thank the reviewer for the positive assessment of EvaKD, particularly its simple yet effective design and comprehensive evaluation. We appreciate the insightful concerns regarding the practical construction of evolution stages, the influence of parameter-efficient fine-tuning, and the applicability of EvaKD to heterogeneous architectures. We address each weakness and question separately below.

W1: Practical Evolution-Stage Construction and the Effect of Stage Selection

Response. We appreciate the reviewer for pointing out the insufficient clarification of the four representative evolution stages. We agree that the original manuscript should better distinguish the general layer-wise formulation from its practical implementation.

First, regarding the construction of the four evolution stages, the general formulation in Sec. 3.4 characterizes prediction evolution through changes between consecutive Transformer-layer outputs. For computational efficiency, our implementation aggregates these layer-wise prediction changes into four representative depth intervals for both the teacher and student, excluding the embedding output.

Specifically, let LXL^{\mathcal X} denote the number of Transformer layers in model X\mathcal X. We select five depth boundaries independently for each model:

m_r^{\mathcal X}=\left\lfloor 1+\frac{r(L^{\mathcal X}-1)}{4}\right\rfloor, \quad r=1,\ldots,4.$$ The prediction evolution of the (r)-th representative stage is then defined as: $\Delta \mathbf z_{t,r}^{\mathcal X} = \mathbf z_{t,m_{r}^{\mathcal X}}^{\mathcal X} - \mathbf z_{t,m_{r-1}^{\mathcal X}}^{\mathcal X}.$ Here, node 1 denotes the output of the first Transformer layer, while node $L^{\mathcal X}$ denotes the final Transformer-layer output. The embedding output is excluded. Each representative evolution vector therefore captures the cumulative prediction change between the two endpoint Transformer layers of its interval, rather than the change between only two immediately adjacent layers. The exact selected depth intervals are summarized below. > **Table A: Selected evolution stages in the two main teacher-student pairs.** | Model Pair | Model | Selected Depth Intervals | | ------------------------- | ------- | ------------------------ | | Llama3.1-8B → Llama3.2-1B | Teacher | 1→8, 8→16, 16→24, 24→32 | | | Student | 1→4, 4→8, 8→12, 12→16 | | Qwen3-4B → Qwen3-0.6B | Teacher | 1→9, 9→18, 18→27, 27→36 | | | Student | 1→7, 7→14, 14→21, 21→28 | **Second, regarding whether uniform depth partition introduces predefined cross-model alignment**, we clarify that the partition only determines the evolution units within each individual model. It does not establish any teacher-student correspondence in advance. Specifically, PEOT independently calculates the evolution cost for all 16 teacher-student stage pairs and optimizes a $4\times4$ transport plan based on both prediction-evolution similarity and normalized depth difference. Although uniform marginals constrain the total transport mass of each row and column, they do not impose a diagonal or one-to-one correspondence. The resulting transport plan can therefore distribute mass across different teacher and student stages. We further reuse the fixed-mapping comparison from our response to Reviewer 33UR (Table B), which directly isolates the contribution of flexible evolution correspondence. > **Table B: Comparison between fixed relative-depth mapping and PEOT (ROUGE-L %).** | Llama3.1-8B → Llama3.2-1B | Dolly | SelfInst | S-NI | UnNI | Vicuna | Avg. | | ------------------------------- | --------- | ------------ | --------- | --------- | ---------- | --------- | | Fixed relative-depth (w/o PEOT) | 28.46 | 21.32 | 36.43 | 36.55 | 16.36 | 27.82 | | **PEOT (EvaKD)** | **30.02** | **23.23** | **38.61** | **38.19** | **18.49** | **29.71** | | **Qwen3-4B → Qwen3-0.6B** | **Dolly** | **SelfInst** | **S-NI** | **UnNI** | **Vicuna** | **Avg.** | | Fixed relative-depth (w/o PEOT) | 27.57 | 19.72 | 38.79 | 41.58 | 20.05 | 29.54 | | **PEOT (EvaKD)** | **29.56** | **24.77** | **43.07** | **45.63** | **21.23** | **32.85** | As shown in Table B, PEOT improves average ROUGE-L by **+1.89** and **+3.31** points over fixed relative-depth mapping on the two model pairs, respectively. These improvements demonstrate that the gains are not simply obtained from introducing uniformly partitioned intermediate supervision.
AuthorsOpenReview ↗

Response to Reviewer 1YAh (PART II)

Furthermore, we reuse the depth-cost sensitivity results from our response to Reviewer 33UR (Table C) to demonstrate that the evolution cost provides an independent contribution to the learned correspondence. Table C further quantifies the depth term's contribution to evolution-aware cross-model alignment.

Table C: Sensitivity analysis of the depth-cost coefficient β\beta (Average ROUGE-L).

β\betaLlama3.2-1BQwen3-0.6B
028.1431.47
0.128.8431.63
0.529.1232.18
1.0 (Default)29.7132.85
2.029.3731.91

When the depth cost is completely removed (β=0\beta=0), PEOT still achieves 28.14 and 31.47 average ROUGE-L, respectively. This indicates that prediction-evolution similarity itself provides meaningful alignment signals, rather than the transport plan being determined solely by relative depth. The default (β=1.0\beta=1.0) further achieves the best performance on both model pairs.

Third, regarding the choice of four evolution stages, we agree that this choice should be empirically justified. We therefore reuse the stage-number sensitivity experiments from our response to Reviewer 8VJP (Table D), comparing different numbers of representative stages while keeping the remaining configurations unchanged. The all-stage setting covers all LX−1L^{\mathcal X}-1 adjacent-layer transitions among Transformer layers 1 through LXL^{\mathcal X}.

Table D: Sensitivity analysis of the number of evolution stages (Average ROUGE-L).

Number of Teacher / Student StagesLlama3.2-1BQwen3-0.6B
2 / 228.9631.87
4 / 4 (Default)29.7132.85
8 / 829.5332.79
All / All (NT/NSN^{\mathcal T}/N^{\mathcal S})29.6432.76

The results show that 4/4 stages achieve the best performance on both model pairs. Increasing the number of stages to 8 or using all layer-wise evolution transitions does not yield further improvements, while using only two stages results in noticeable performance degradation. These results suggest that four representative stages provide an effective balance between evolution coverage and computational efficiency, rather than being an arbitrarily selected configuration.

Finally, regarding the contribution of late-layer evolution, we further conduct a suffix-only ablation to investigate whether the improvements can be obtained by exclusively transferring knowledge from the final portions of the teacher and student networks.

Specifically, we retain only the last 25%, 50%, or 75% of the layer-wise evolution transitions between Transformer layers 1 and LXL^{\mathcal X} in each model, independently reconstruct the corresponding PEOT cost matrix, and apply DAT to the resulting transport plan. The original normalized depth coordinates are preserved, and all other training configurations remain unchanged.

Table E: Ablation of late-layer evolution distillation (Average ROUGE-L).

Evolution RangeLlama3.2-1BQwen3-0.6B
Last 25%28.7231.88
Last 50%28.9132.03
Last 75%29.0632.21
Last 100% (All transitions)29.6432.76
Four representative stages (EvaKD)29.7132.85

The results show that performance consistently improves as the evolution range expands from the last 25% to all layer-wise transitions, increasing from 28.72 to 29.64 on Llama3.2-1B and from 31.88 to 32.76 on Qwen3-0.6B. Moreover, the four-stage configuration achieves the best results, outperforming all-transition supervision by +0.07 and +0.09 points, respectively. These findings suggest that useful prediction evolution is distributed across network depths, while representative-stage aggregation provides a more effective supervision granularity. We will incorporate these results into the revised appendix.

Overall, these experiments clarify the stage construction, the flexibility of cross-model correspondence, and the impact of evolution-stage selection. We sincerely thank the reviewer for these helpful suggestions.

AuthorsOpenReview ↗

Response to Reviewer 1YAh (PART III)

W2: Effect of Full-Parameter Fine-Tuning

Response. We appreciate this concern. Since LoRA may constrain a small student's adaptation capacity, full-parameter fine-tuning is necessary to test whether EvaKD's gains depend on parameter-efficient optimization. We therefore compare KD and EvaKD under both strategies.

From a methodological perspective, EvaKD modifies the distillation objective rather than the parameterization of the student model. Specifically, PEOT establishes evolution-aware correspondence between teacher and student predictions, while DAT selectively transfers teacher evolution according to the student's remaining prediction deficiency.

Neither component introduces additional trainable projection modules or relies on LoRA-specific operations. Consequently, the proposed supervision mechanism is compatible with both parameter-efficient and full-parameter optimization. Thus, the same PEOT and DAT objectives supervise LoRA-adapted and fully trainable students.

To investigate this issue, we additionally compare KD and EvaKD under full-parameter fine-tuning, using the original LoRA-based results as references. Both methods use identical training data and optimization configurations within each training strategy.

Table F: Comparison between LoRA and full-parameter fine-tuning (Average ROUGE-L).

Model PairTraining StrategyKDEvaKDImprovement
Llama3.1-8B → Llama3.2-1BLoRA (r=8)22.8729.71+6.84
Full-Parameter FT22.9429.83+6.89
Qwen3-4B → Qwen3-0.6BLoRA (r=8)31.5532.85+1.30
Full-Parameter FT31.6132.93+1.32

The results show that EvaKD consistently outperforms KD under both training strategies. Under full-parameter fine-tuning, EvaKD achieves 29.83 and 32.93 average ROUGE-L on the two model pairs, outperforming KD by +6.89 and +1.32 points, respectively. These gains are comparable to those under LoRA (+6.84 and +1.30), demonstrating that EvaKD's effectiveness does not rely on parameter-efficient fine-tuning and remains consistent when all student parameters are trainable. We will include these comparisons and clarify the compatibility of EvaKD with different student fine-tuning strategies in the revised manuscript.

W3: Applicability to Heterogeneous Architectures and MoE-to-Dense Distillation

Response. We thank the reviewer for pointing out the limited experimental evidence for heterogeneous architecture settings.

In Appendix C.3, we evaluate distillation from Llama3.1-8B to Qwen3-0.6B. EvaKD improves the average ROUGE-L from 30.83 to 31.17, yielding a gain of +0.34 over vanilla KD. Although this result demonstrates that EvaKD can operate across different model families, we acknowledge that further evaluation is needed to establish its applicability to different architectural configurations, particularly MoE-to-Dense distillation.

From a methodological perspective, EvaKD does not require identical internal architectures or expert-level correspondence. Its prediction-evolution representation is constructed by projecting each model's intermediate hidden states through its own normalization module and LM head. For an MoE teacher, the expert routing and expert computations are already incorporated into the resulting hidden states. These hidden states can therefore be projected into the prediction space in the same manner as those of a dense model.

Consequently, EvaKD does not need to align teacher and student experts, expert indices, or routing decisions. Instead, PEOT operates on their prediction changes, while DAT evaluates whether the transferred teacher evolution complements the student's remaining prediction deficiency.

To further investigate this setting, we conduct an additional experiment using Qwen3-30B-A3B (MoE) as the teacher and Qwen3-0.6B (Dense) as the student. We follow the same domain-agnostic evaluation protocol as the original experiments.

Table G: Results of MoE-to-Dense distillation on domain-agnostic benchmarks (ROUGE-L).

MethodDollySelfInstS-NIUnNIVicunaAvg.
KD28.4722.8743.5843.6119.2431.55
EvaKD29.3924.4343.4845.4520.9232.73

For the MoE-to-Dense setting, EvaKD achieves 32.73 average ROUGE-L, outperforming vanilla KD (31.55) by +1.18 points. It consistently improves performance on four out of five benchmarks. These results provide further empirical evidence that EvaKD generalizes to heterogeneous architectures involving sparse-expert teachers and dense students.

Reviewer_8VJPOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EvaKD, an intermediate-layer knowledge distillation framework for LLMs. EvaKD addresses two limitations of existing intermediate LLM distillation methods: predefined layer correspondence and indiscriminate knowledge transfer. The experiments cover multiple LLMs, including Llama and Qwen at different scales, distilled on Dolly and on medical/legal data. The promising results support the effectiveness of the method.

Critical strengths
  1. EvaKD achieves the best average scores across both model families and both domain-agnostic and domain-specific settings, outperforming a broad set of output-layer baselines and intermediate-layer baselines.

  2. Reusing each model’s own norm head avoids learned projectors and hidden-dimension alignment. Candidate-vocabulary restriction is theoretically justified as exact conditioning. The small 4×4 OT problem further supports practical adoption.

Critical weaknesses
  1. It's unclear where the gain come from, and concerns about the ablation. The main text defines $(N^{\mathcal{T}}=L^{\mathcal{T}}-1) $ evolution steps, but Appendix B.4 uses only four representative evolution stages without specifying the layers. Thus, the claimed flexible many-to-many alignment is effectively a 4×4 matrix. Fig. 4 shows that DAT assigns about 90% of its mass to the last two teacher steps and 55–59% to the final step. The gains may therefore come largely from additional KL supervision on late-layer teacher distributions, overlapping with the existing (LKD(\mathcal{L}_{KD}) term. Key controls are missing, e.g., uniform or fixed-mapping (Γ(\Gamma) with the same sparse KL, last-stage-only KL, and downstream sensitivity to (λ(\lambda), (ε(\varepsilon), and (K(K).

  2. Novelty is insufficiently established. Many-to-many layer alignment using optimal transport for distillation already exists, but those methods are neither cited nor compared.

Questions for authors
  1. Which layers form the “four representative evolution stages” for each model? How are they chosen, and how does performance change when all L−1L-1 steps are used?

  2. How does EvaKD compare with (a) SFT+KD plus a uniform-weight sparse intermediate KL, and (b) SFT+KD plus KL only on the final teacher stage? Given the mass concentration in Fig. 4, this comparison is needed to attribute gains to PEOT/DAT.

  3. What is the end-to-end training time and memory overhead relative to KD and FDD? Fig. 6 reports only projection cost. In the heterogeneous setting, with different vocabularies, how are Ct\mathcal{C}_t and the cosine between teacher and student evolution vectors defined?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 8VJP (PART I)

Comment: We sincerely thank the reviewer for the positive assessment of EvaKD, particularly its effectiveness across different model families and domains, as well as its parameter-efficient prediction-space design. We appreciate the insightful concerns regarding the attribution of performance gains, the practical evolution-stage configuration, and the positioning of our method relative to existing OT-based approaches. We address each weakness and question separately below.

W1: Attribution of Performance Gains and the Effectiveness of PEOT and DAT

Response. We appreciate the reviewer for pointing out the potential ambiguity in attributing the performance improvements to different components. We clarify that PEOT and DAT address two fundamentally different aspects of intermediate knowledge transfer.

First, PEOT determines where knowledge should be transferred. Instead of directly aligning intermediate representations according to relative layer positions, PEOT characterizes each model's prediction evolution through the difference between consecutive-layer predictions. It then constructs an evolution-aware cost based on prediction-change similarity and relative depth, and solves an entropy-regularized OT problem to obtain a soft many-to-many correspondence. Thus, the transport plan is determined by the actual prediction dynamics rather than a predefined layer mapping.

Second, DAT determines which transferred knowledge is beneficial. Specifically, it measures the directional compatibility between teacher evolution and the student's remaining prediction deficiency. The resulting transport plan is reweighted by positive deficiency utility. This distinction is important because a teacher evolution step can be structurally similar to a student evolution step but still provide redundant or conflicting supervision.

Regarding the concentration of transport mass, Figure 4 shows that the last two teacher evolution steps account for approximately 90.1% and 89.9% of the normalized DAT mass for Llama3 and Qwen3, respectively, while the final step contributes 54.7% and 58.5%. This concentration is not manually imposed; it emerges from the deficiency-aware utility. Moreover, DAT is not equivalent to simply duplicating the final-layer KD objective. Specifically, the final teacher prediction can supervise multiple intermediate student states, whereas standard KD only matches the final teacher and student predictions. Therefore, only the final-teacher/final-student pair corresponds to endpoint-to-endpoint supervision.

To further isolate the contributions of the alignment and selection mechanisms, we provide the following comparisons. The fixed relative-depth results are directly reused from our response to Reviewer 33UR (Table B), while the PEOT-only results are available in the original Table 3. We additionally introduce uniform-weight and final-teacher-stage-only controls, as detailed in Q2.

Regarding hyperparameter sensitivity, we agree that the effects of the intermediate distillation weight, entropy regularization coefficient, and distillation temperature should be further investigated. We therefore propose the following controlled experiments, varying one hyperparameter at a time while keeping the others fixed at their default values.

Table A: Sensitivity analysis of EvaKD hyperparameters (Average ROUGE-L).

HyperparameterSettingLlama3.2-1BQwen3-0.6B
λ\lambda0.527.4432.42
1.0 (Default)29.7132.85
2.029.3032.56
ϵ\epsilon0.0129.5332.71
0.0529.3432.48
0.1 (Default)29.7132.85
0.229.2232.39
τ\tau0.529.4932.60
1.0 (Default)29.7132.85
2.029.4032.54

Here, λ\lambda, ϵ\epsilon, and τ\tau control the intermediate distillation weight, PEOT entropy regularization, and distillation temperature, respectively. The default settings consistently achieve the best performance on both model pairs, demonstrating the robustness of EvaKD to moderate hyperparameter variations. Together with the controlled comparisons in Q2, these results further distinguish the contributions of evolution-aware alignment and deficiency-aware selection from simply introducing additional intermediate KL supervision. We will clarify these findings and the corresponding implementation details in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer 8VJP (PART II)

W2: Novelty Relative to Existing OT-Based Intermediate-Layer Distillation

Response. We thank the reviewer for noting prior OT-based layer alignment. We acknowledge that many-to-many OT mapping has been explored, particularly in BERT-EMD [1], and will better discuss these connections.

Existing approaches differ in their transported objects and alignment granularity:

  • BERT-EMD [1] uses EMD for many-to-many teacher-student layer mapping. Its cost is based on MSE between intermediate attention maps and hidden representations, with adaptive cost attention.
  • Model Compression Using Optimal Transport [2] computes OT distances between teacher and student feature distributions at corresponding layers and aggregates them. Its OT matches feature samples within each layer, rather than learning cross-layer correspondences.
  • WCoRD [3] uses primal and dual Wasserstein objectives: the primal matches feature distributions within mini-batches, while the dual enables global contrastive transfer. It mainly distills penultimate-layer representations rather than aligning multiple intermediate layers.

EvaKD differs in three aspects:

  1. Alignment object: EvaKD performs OT over adjacent-layer prediction changes in a sparse prediction space, capturing how predictions evolve with model depth rather than matching static hidden representations.
  2. Evolution-aware cost: PEOT jointly considers prediction-change similarity and relative depth to establish flexible teacher-student evolution correspondences.
  3. Deficiency-aware selection: DAT further reweights and truncates the transport plan according to the compatibility between teacher evolution and the student's remaining prediction deficiency, filtering potentially redundant or conflicting supervision.

Table B: Comparison with existing OT-based intermediate-layer and representation distillation methods.

MethodTransported ObjectOT Alignment GranularityAlignment / Selection MechanismDistillation Objective
BERT-EMD [1]Intermediate attention maps and hidden statesMany-to-many teacher-student layer mappingEMD with adaptive cost attentionAttention and hidden-state MSE
Model Compression [2]Intermediate feature distributionsFeature-level OT within corresponding layersWasserstein-1 distanceIntermediate feature distribution matching
WCoRD [3]Penultimate-layer feature distributionsSample-level OT within mini-batchesPrimal and dual Wasserstein objectivesContrastive representation distillation
EvaKD (ours)Adjacent-layer prediction changesMany-to-many teacher-student evolution-step alignmentEvolution-aware OT (PEOT) + deficiency-aware truncation (DAT)Selective intermediate prediction KL

OT itself is not the sole novelty: PEOT compares depthwise prediction changes, while DAT evaluates each transported update against the student's remaining deficiency. Thus, even a well-aligned pair can be downweighted or removed if its teacher evolution does not address that deficiency. This sequential alignment-and-selection goes beyond static feature matching.

Thus, EvaKD's contribution is to align prediction-evolution trajectories and selectively transfer evolution signals according to student deficiency. Static-representation alignment does not explicitly capture prediction dynamics, while an OT correspondence alone does not ensure that transferred knowledge addresses the student's current deficiencies.

We will revise Related Work to discuss BERT-EMD and other OT-based representation-distillation methods, and clarify EvaKD's methodological distinctions.

[1] Li J, Liu X, Zhao H, et al. BERT-EMD: Many-to-many layer mapping for BERT compression with earth mover’s distance[C]//Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020.

[2] Lohit S, Jones M. Model compression using optimal transport[C]//2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). IEEE, 2022.

[3] Chen L, Wang D, Gan Z, et al. Wasserstein contrastive representation distillation[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.

AuthorsOpenReview ↗

Response to Reviewer 8VJP (PART III)

Q1: Selection of the Four Evolution Stages and the Effect of Stage Number

Response. We thank the reviewer for noting that the practical evolution-stage configuration needs clarification.

Sec. 3.4 defines prediction evolution through adjacent-layer prediction changes, with NT=LT−1N^{\mathcal T}=L^{\mathcal T}-1 teacher steps and NS=LS−1N^{\mathcal S}=L^{\mathcal S}-1 student steps. For efficiency, our implementation groups the prediction changes between Transformer layers into four representative evolution intervals per model, yielding a 4×44\times4 transport matrix.

Table C: Selected Transformer-layer intervals in the two main teacher-student pairs.

Model PairModelSelected Transformer-Layer Intervals (1-based)
Llama3.1-8B → Llama3.2-1BTeacher1→8, 8→16, 16→24, 24→32
Student1→4, 4→8, 8→12, 12→16
Qwen3-4B → Qwen3-0.6BTeacher1→9, 9→18, 18→27, 27→36
Student1→7, 7→14, 14→21, 21→28

Each interval represents the cumulative prediction change between its two endpoint Transformer layers and may span multiple adjacent-layer transitions. The selected nodes start from Transformer layer 1 and end at layer LXL^{\mathcal X}; the embedding output is excluded. Specifically, for model X∈{T,S}\mathcal X\in\{\mathcal T,\mathcal S\}, the boundaries follow m0X=1m_0^{\mathcal X}=1 and mrX=⌊1+r(LX−1)/4⌋m_r^{\mathcal X}=\lfloor 1+r(L^{\mathcal X}-1)/4\rfloor, r=1,…,4r=1,\ldots,4, yielding a uniform partition of the Transformer-layer depth. The partition is applied independently within each model and does not pair the rr-th teacher interval with the rr-th student interval. PEOT still optimizes the transport weights over all 16 cross-model interval pairs; therefore, the uniform partition defines the evolution units, not a fixed cross-model mapping.

To assess the number of evolution intervals, we compare different numbers of representative intervals while keeping other settings fixed. The all-stage setting uses every adjacent-layer prediction change between Transformer layers 1 and LXL^{\mathcal X}.

Table D: Sensitivity analysis of the number of evolution stages (Average ROUGE-L).

Number of Teacher / Student StagesLlama3.2-1BQwen3-0.6B
2 / 228.9631.87
4 / 4 (Default)29.7132.85
8 / 829.5332.79
All / All (NTN^{\mathcal T} / NSN^{\mathcal S})29.6432.76

Four stages achieve the best results on both pairs. Using 8 or all steps brings no further gain, while 2 stages performs worse. This suggests that four stages balance evolution coverage and computational efficiency. We will add the selected indices and sensitivity results to the revised appendix.

Q2: Comparison with Uniform-Weight and Final-Teacher-Stage-Only Intermediate Distillation

Response. We compare three strategies under the same SFT and final-layer KD objectives:

  1. Uniform-weight intermediate KL: all 16 teacher-student stage pairs receive equal weight 1/161/16, testing the effect of adding intermediate KL alone.
  2. Final-teacher-stage-only KL: only the final teacher stage supervises all four student stages with equal weights, testing whether gains mainly come from late-stage teacher predictions.
  3. EvaKD: PEOT learns evolution correspondences, and DAT reweights them according to student deficiency.

The first two controls use the same sparse prediction projection as EvaKD. We also reproduce the fixed relative-depth comparison reported in Reviewer 33UR's Table B.

Table E: Comparison between fixed relative-depth mapping and PEOT (ROUGE-L %).

Llama3.1-8B → Llama3.2-1BDollySelfInstS-NIUnNIVicunaAvg.
Fixed relative-depth (w/o PEOT)28.4621.3236.4336.5516.3627.82
PEOT (EvaKD)30.0223.2338.6138.1918.4929.71
Qwen3-4B → Qwen3-0.6BDollySelfInstS-NIUnNIVicunaAvg.
Fixed relative-depth (w/o PEOT)27.5719.7238.7941.5820.0529.54
PEOT (EvaKD)29.5624.7743.0745.6321.2332.85

PEOT improves average ROUGE-L by +1.89 and +3.31 over fixed relative-depth mapping on the two pairs, respectively.

AuthorsOpenReview ↗

Response to Reviewer 8VJP (PART IV)

We further report controlled comparisons to isolate the effect of intermediate supervision weighting.

Table F: Comparison of intermediate supervision strategies (Average ROUGE-L).

MethodTransport WeightLlama3.2-1BQwen3-0.6B
SFT + KDNo intermediate KL22.8731.55
Uniform-weight sparse KLΓrs=1/16\Gamma_{rs}=1/1627.8330.52
Final-teacher-stage-only KLΓrs=141[r=4]\Gamma_{rs}=\frac{1}{4}\mathbb{1}[r=4]28.0930.67
PEOT without DATΓPE\Gamma^{\mathrm{PE}}28.6631.34
EvaKDΓDAT\Gamma^{\mathrm{DAT}}29.7132.85

The uniform control assigns equal mass to all 16 pairs. The final-stage-only control retains only the last teacher stage (r=4r=4) and distributes its mass equally over four student stages. Both retain the same sparse projection and intermediate KL objective, changing only the supervision strategy. EvaKD outperforms uniform-weight KL by +1.88, +2.33 and final-stage-only KL by +1.62, +2.18 on both models. Thus, the gains are not explained by simply adding intermediate KL or emphasizing the final teacher stage.

Q3: End-to-End Training Overhead and Heterogeneous Vocabulary Alignment

Response. We agree that the sparse-projection microbenchmark alone does not fully characterize EvaKD's overhead. In addition to sparse projection, EvaKD solves a 4×44\times4 entropy-regularized OT problem with 100 Sinkhorn iterations and computes deficiency utilities. Importantly, the transport plan is estimated at the dataset level and reused during training, rather than recomputed for every training token, which limits the additional optimization overhead.

We reuse the end-to-end efficiency results from our response to Reviewer 33UR (Table D), comparing KD, TED, FDD, and EvaKD under identical hardware and training configurations.

Table G: End-to-end training efficiency comparison.

MethodTraining Time (h)Time per Step (s)Peak GPU Memory (GB)
KD3.221.62422.97
TED5.7642.90223.42
FDD5.9132.98225.12
EvaKD (ours)5.5772.80820.37

EvaKD takes 5.577 h, 3.2% less than TED and 5.7% less than FDD. Its peak memory is 20.37 GB, 13.0% and 18.9% lower than TED and FDD, respectively. EvaKD takes longer than KD because of intermediate projection and evolution/deficiency computation, but remains faster than TED and FDD while using less peak memory than KD.

For heterogeneous distillation, teacher and student logits cannot be directly compared when their vocabularies differ. In Appendix C.3, we adopt MultiLevelOT's rank-based alignment to construct a rank-aligned common support. Specifically, for each model X∈{T,S}\mathcal X\in\{\mathcal T,\mathcal S\}, we independently retain the top-KK logits and sort them in descending order:

z~t,mX=AX(zt,mX)∈RK,\widetilde{\mathbf z}^{\mathcal X}_{t,m}=\mathcal A_{\mathcal X}(\mathbf z^{\mathcal X}_{t,m}) \in \mathbb R^K,

where AX\mathcal A_{\mathcal X} denotes the model-specific rank-based mapping. The kk-th largest logit is assigned to the kk-th position in the shared rank support, so teacher and student predictions have the same dimensionality without requiring direct token-identity matching.

The teacher and student evolution vectors are then computed in this common rank space:

\Delta\widetilde{\mathbf z}^{\mathcal S}_{t,s} = \widetilde{\mathbf z}^{\mathcal S}_{t,m_s^{\mathcal S}} - \widetilde{\mathbf z}^{\mathcal S}_{t,m_{s-1}^{\mathcal S}}.$$ Their similarity is computed as: $$\operatorname{cos}\left( \Delta\widetilde{\mathbf z}^{\mathcal T}_{t,r}, \Delta\widetilde{\mathbf z}^{\mathcal S}_{t,s} \right).$$ **The same rank-based mappings are used consistently in DAT and intermediate distillation.** For DAT, we first map the teacher's final-layer prediction and the student's intermediate predictions to the rank-aligned common support, and then compute the deficiency utilities in this shared space. For the intermediate KL loss, we likewise map the corresponding teacher and student predictions before applying softmax and computing KL divergence over the $K$ rank positions.
Reviewer_33UROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

EvaKD targets two problems in intermediate-layer distillation for LLMs: (1) predefined depth-wise layer correspondences can mismatch the actual prediction-refinement stages of teacher and student, and (2) transferring all aligned intermediate signals can be redundant or even conflicting with what the student still needs. The method projects intermediate hidden states into a sparse prediction space, builds a prediction-evolution cost between consecutive teacher/student layers, uses entropy-regularized optimal transport (PEOT) to obtain a soft correspondence, and then truncates/reweights the transport plan by a “deficiency utility” that measures whether a teacher evolution direction aligns with the student’s remaining gap to the final teacher prediction (DAT). The final loss combines SFT, final-layer KD, and transport-weighted intermediate KL. Experiments cover Llama3 and Qwen3 teacher–student pairs, domain-agnostic and domain-specific instruction benchmarks, with additional reasoning, larger-teacher, and heterogeneous-family results in the appendix.

Critical strengths
  1. Figure 1 and the surrounding discussion make a convincing case that relative depth is not a reliable proxy for prediction-refinement stage, and that aligned teacher evolution can be useful or harmful depending on the student’s remaining deficiency. The paper connects these observations to concrete design choices rather than treating them as post-hoc intuition.

  2. Sparse prediction projection, PEOT, and DAT each address a distinct issue. The use of the native normalization and LM head avoids introducing a learned projector, and the sparse candidate set is a sensible way to keep intermediate projection cheap while preserving teacher-preferred, student-preferred, and gold tokens.

The propositions on exact conditionalization of sparse predictions, PEOT existence/uniqueness/Sinkhorn structure, stability under cost perturbations, and the KL-reweighting interpretation of DAT give the components formal grounding. The paper is also careful to note that the positive-utility result is a local logit-space descent statement, not a guarantee on the KL distillation objective.

The main experiments cover two model families, domain-agnostic instruction following, medical/legal domain-specific evaluation, and ten baselines spanning output-layer and intermediate-layer distillation. The appendix adds reasoning benchmarks, a larger teacher (Llama2-13B → TinyLlama-1.1B), and heterogeneous distillation (Llama3.1-8B → Qwen3-0.6B).

The ablation in Table 3 shows both PEOT and DAT matter, with PEOT contributing more on Qwen3. Figure 4 shows DAT concentrates mass on later/high-utility teacher steps, and Figure 5 reports strong correlation between U and sparse-KL reduction. The sparse-projection analysis in Figure 6 gives a practical accuracy–efficiency trade-off, and the depth-wise analysis in Figure 7 supports the claim that conflicting teacher evolution is concentrated earlier.

At K = 32, the candidate vocabulary retains 91.66%–99.16% of prediction mass while projection latency is 7.31%–23.59% of full projection and peak memory is 5.92%–27.55%. This is a concrete efficiency benefit that could be adopted beyond EvaKD.

Critical weaknesses
  1. The DAT fallback rule can retain negative-utility paths. Appendix A.3 states that when DAT falls back to \tau_{DAT} = \tau_{PE} under certain conditions. In that case all retained paths have non-positive utility, so the “remove conflicting supervision” claim no longer holds. The fallback may be practically necessary to avoid an empty plan, but the paper should discuss when this happens, how often, and whether a safer fallback (e.g., uniform over positive-utility pairs, or no intermediate loss) would be better.

  2. Table 3 shows removing PEOT drops average ROUGE-L by 1.89 and 3.31 points, but the main text does not say what replaces PEOT: fixed identical relative depth, proportional depth, last-layer-only, or uniform transport? Without this, readers cannot tell whether the gain comes from OT specifically or simply from using a soft/dynamic correspondence instead of a hard one. A “best fixed depth mapping” baseline would be especially useful.

  3. The cost combination C^{EVO} + C^{depth} has no weighting, but the terms have different scales. Eq. 9 adds a cosine-based evolution cost in [0,2] to a squared normalized-depth cost in [0,1]. The paper says this avoids an additional coefficient, but the relative scale is still a design choice. A sensitivity analysis on the depth-cost weight (or on removing it) would clarify whether PEOT is mainly driven by evolution similarity or by depth regularization.

  4. Training overhead is not fully reported. The sparse-projection microbenchmark is useful, but PEOT requires Sinkhorn iterations and DAT requires computing U over evolution pairs. If the plan is global, this overhead is small; if per-token/per-example, it may be substantial. The paper should report end-to-end training time/GPU memory relative to KD/TED/FDD, not only projection latency.

  5. The sparse candidate set is built from final-layer top-K predictions, which may bias early-layer supervision. Early intermediate layers may assign meaningful probability to tokens outside the final top-K sets, especially when the teacher and student evolve differently. The coverage analysis in Figure 6 is valuable, but it appears to measure final-layer coverage; the paper should report coverage/approximation error for intermediate layers as well, or at least discuss whether the same candidate set is appropriate across depth.

  6. The extracted Table 1/Table 2 contents are truncated, so I could not check the exact main numbers. More importantly, the paper reports averages over five seeds but does not give confidence intervals or paired tests for the main ROUGE-L/accuracy comparisons. The ablation drops (1.05–3.31 points) are meaningful, but variance matters. The abstract’s “up to 7.01% scores and 7.4% accuracy scores” is ambiguous: are these percentage points or relative improvements? Please clarify and report uncertainty.

  7. Novelty relative to existing OT-based and feature-dynamics distillation is not fully positioned. The paper cites MultiLevelOT (Cui et al., 2025) only for heterogeneous vocabulary alignment, and cites FDD (Gong et al., 2025) as a feature-dynamics baseline. But the core idea—using OT to align layer-wise evolution and selectively weighting transport mass—invites comparison to OT-based cross-layer/cross-tokenizer KD and to FDD’s first-order feature-dynamics alignment. A short conceptual table (what is transported, cost, selection rule, readout) would help readers assess the incremental contribution.

Questions for authors

Is the PEOT plan global or input-dependent? Eq. 7 averages the evolution cost over the dataset, which suggests a single \tau^PE for all examples. But Appendix B.5 says the plan is reconstructed per validation example and mentions token-level OT in chunks of 64. Please state explicitly whether training uses one global plan, a per-batch plan, or a per-token/per-example plan. If global, how does EvaKD handle the task-dependent evolution patterns shown in Appendix C.4? If per-example, what is the additional Sinkhorn cost and how noisy is the cost estimate?

What exactly replaces PEOT in the “w/o PEOT” ablation? Is it identical relative depth, proportional depth, a hard one-to-one mapping, or uniform transport? A “best fixed mapping” baseline would directly test whether OT’s soft correspondence is necessary beyond any dynamic/soft alignment.

How often does the DAT fallback occur? How frequently does this happen on Dolly/medical/legal data? Would a fallback that skips intermediate distillation or restricts to the least-negative paths be more consistent with the method’s motivation?

Why is there no weight on the depth cost in Eq. 9? How sensitive are the results to removing the depth term entirely? This would clarify whether PEOT’s gain comes from evolution matching or from preserving depth order.

Does the sparse candidate set preserve intermediate-layer coverage?

Are the main results statistically significant? Please report confidence intervals or paired significance tests for the main ROUGE-L and accuracy comparisons. Also clarify whether “7.01%” and “7.4%” in the abstract are percentage points or relative improvements.

How does EvaKD compare to OT-based intermediate distillation beyond MultiLevelOT? MultiLevelOT is cited for vocabulary alignment in the heterogeneous experiment, but not compared as a distillation baseline. Could you add a conceptual/experimental comparison to OT-based layer alignment or at least explain why MultiLevelOT is not an applicable baseline for homogeneous settings?

What is the end-to-end training overhead? Please report training wall-clock and peak memory for EvaKD vs. KD/TED/FDD under the same hardware, including Sinkhorn and DAT utility computation. The projection microbenchmark alone does not capture the full cost.

7 comments
AuthorsOpenReview ↗

Response to Reviewer 33UR (PART I)

Comment: We thank the reviewer for recognizing EvaKD's motivation, PEOT/DAT designs, theoretical analysis, and evaluation. We address the concerns below.

W1: DAT Fallback and Negative-Utility Transport Paths

Response. DAT reweights the PEOT plan using positive deficiency utility:

ΓrsDAT=Γrs{PE}[Urs]+Z,Z=∑r,sΓrs{PE}[Urs]+.\Gamma^{\mathrm{DAT}}_{rs}=\frac{\Gamma^{\mathrm\{PE\}}_{rs}[U_{rs}]_{+}}{Z}, \quad Z=\sum_{r,s}\Gamma^{\mathrm\{PE\}}_{rs}[U_{rs}]_+.

Here, [Urs]+=max⁡(Urs,0)[U_{rs}]_+=\max(U_{rs},0). When Z>0Z>0, DAT removes non-positive-utility paths and normalizes the remaining mass. When Z=0Z=0, normalization is undefined; our implementation therefore sets LDAT=0\mathcal L_{\mathrm{DAT}}=0, while retaining SFT and final-layer KD. This avoids transferring knowledge through paths with non-positive utility.

Theorem 4 (Appendix A.3) shows that DAT does not decrease transport-weighted deficiency utility when Z>0Z>0; this is an aggregate guarantee, not a claim that every retained path has positive utility. Figure 4 illustrates non-uniform utilities: 4 of 16 pairs have negative utility for Llama3, and Qwen3 assigns lower utility to early teacher stages.

We monitored six dataset–model-pair combinations (Llama3 and Qwen3 on Dolly, GenMedGPT, and AlpacaMyGavel). No Z=0Z=0 fallback occurred in 261,280 training microbatches (0%). As Table A shows, partial teacher-stage truncation nevertheless occurred in 37.99%–44.05% of Llama3 microbatches and 15.19%–19.91% of Qwen3 microbatches, confirming that DAT actively filters non-positive-utility stages whenever positive mass remains.

Table A: DAT fallback frequency and partial teacher-stage truncation statistics.

DatasetLlama3 Fallback (Count / Microbatches)Llama3 Partial Truncation (%)Qwen3 Fallback (Count / Microbatches)Qwen3 Partial Truncation (%)
Dolly0 / 57,175 (0%)37.990 / 57,175 (0%)19.09
Medical (GenMedGPT)0 / 27,260 (0%)41.330 / 27,260 (0%)15.19
Legal (AlpacaMyGavel)0 / 46,205 (0%)44.050 / 46,205 (0%)19.91
Total0 / 130,640 (0%)—0 / 130,640 (0%)—

The unit is a batch-size-1 training microbatch, including five-epoch repeated presentations, not a token or optimizer step. We will clarify the fallback condition, implementation, and theoretical implications in Appendix A.3.

W2: Contribution of PEOT and Comparison with Fixed Layer Mapping

Response. Our w/o PEOT ablation uses fixed relative-depth mapping, following TED [1]. Teacher and student stages are matched by minimizing their normalized-depth difference. With four representative stages in each model, this gives a fixed one-to-one, identical-index mapping:

Γrsfixed=14 1(r=s).\Gamma_{rs}^{\mathrm{fixed}} = \frac{1}{4}\,\mathbf{1}(r=s).

Unlike this fixed mapping, PEOT learns flexible many-to-many correspondences from prediction-evolution costs. The ablation retains intermediate prediction-space distillation and changes only the correspondence strategy, isolating PEOT's contribution.

Table B: Comparison between fixed relative-depth mapping and PEOT (ROUGE-L %).

Llama3.1-8B → Llama3.2-1BDollySelfInstS-NIUnNIVicunaAvg.
Fixed relative-depth (w/o PEOT)28.4621.3236.4336.5516.3627.82
PEOT (EvaKD)30.0223.2338.6138.1918.4929.71
Qwen3-4B → Qwen3-0.6BDollySelfInstS-NIUnNIVicunaAvg.
Fixed relative-depth (w/o PEOT)27.5719.7238.7941.5820.0529.54
PEOT (EvaKD)29.5624.7743.0745.6321.2332.85

As Table B shows, PEOT improves average ROUGE-L by +1.89 and +3.31 points, with gains across all five datasets. Per-dataset gains range from +1.56 to +2.18 for Llama3 and +1.18 to +5.05 for Qwen3. This supports the benefit of evolution-aware correspondence beyond intermediate supervision alone.

[1] Liang C, Zuo S, Zhang Q, et al. Less is more: Task-aware layer-wise distillation for language model compression[C] //International Conference on Machine Learning (ICML). PMLR, 2023.

AuthorsOpenReview ↗

Response to Reviewer 33UR (PART II)

W3: Relative Scaling of Evolution Cost and Depth Cost

Response. The evolution and depth costs capture complementary semantic and structural signals. We vary the depth coefficient:

Crsevo+βCrsdepth,C^{\mathrm{evo}}_{rs}+\beta C^{\mathrm{depth}}_{rs},

where β=1\beta=1 is the default and β=0\beta=0 removes depth regularization.

Table C: Sensitivity analysis of the depth-cost coefficient β\beta (ROUGE-L).

β\betaLlama3.2-1B Avg.Qwen3-0.6B Avg.
028.1431.47
0.128.8431.63
0.529.1232.18
1.0 (Default)29.7132.85
2.029.3731.91

Table C shows that β=1.0\beta=1.0 performs best (29.71/32.85). Setting β=0\beta=0 lowers scores by 1.57/1.38 points, while β=2.0\beta=2.0 lowers them by 0.34/0.94. Thus, depth cost is beneficial, but excessive weighting hurts evolution alignment.

W4: End-to-End Training Overhead

Response. EvaKD adds sparse prediction projection, a 4×44\times4 PEOT problem (100 Sinkhorn iterations), and DAT utility computation. The projection reuses existing normalization layers and LM heads, without introducing additional trainable projection modules. At K=32K=32, Figure 6 reports 91.66%–99.16% probability-mass coverage; projection latency and peak memory are 7.31%–23.59% and 5.92%–27.55% of full-vocabulary projection. We compare end-to-end efficiency under identical settings.

Table D: End-to-end training efficiency comparison.

MethodTraining Time (h)Time per Step (s)Peak GPU Memory (GB)
KD3.221.62422.97
TED5.7642.90223.42
FDD5.9132.98225.12
EvaKD (ours)5.5772.80820.37

Table D shows that EvaKD takes 5.577 hours (3.2%/5.7% less than TED/FDD) and uses 20.37 GB peak memory (13.0%/18.9% less). It is slower than KD (3.22 hours) but uses 11.3% less memory.

W5: Intermediate-Layer Coverage of the Sparse Candidate Vocabulary

Response. At position tt, EvaKD uses the union of teacher/student top-KK tokens and the ground-truth token:

\operatorname{TopK}(z^{\mathcal T}_t,K) \cup \operatorname{TopK}(z^{\mathcal S}_t,K) \cup\{y_t\}.$$ Figure 6 reports **91.66%–99.16%** final-layer coverage at $K=32$; Table E reports coverage across all four stages. > **Table E: Intermediate-layer probability coverage of the sparse candidate vocabulary ($K=32$).** | Model Pair | Model | Evolution Stage | Retained Mass (%) | TV Error | | ------------------------- | ------- | --------------: | ----------------: | -------: | | Llama3.1-8B → Llama3.2-1B | Teacher | 1 | 0.42 | 0.9958 | | | | 2 | 1.73 | 0.9827 | | | | 3 | 50.57 | 0.4943 | | | | 4 | 97.05 | 0.0295 | | | Student | 1 | 0.16 | 0.9984 | | | | 2 | 0.26 | 0.9974 | | | | 3 | 15.16 | 0.8484 | | | | 4 | 94.37 | 0.0563 | | Qwen3-4B → Qwen3-0.6B | Teacher | 1 | 1.98 | 0.9802 | | | | 2 | 5.66 | 0.9434 | | | | 3 | 66.22 | 0.3378 | | | | 4 | 99.16 | 0.0084 | | | Student | 1 | 0.55 | 0.9945 | | | | 2 | 1.69 | 0.9831 | | | | 3 | 35.38 | 0.6462 | | | | 4 | 91.66 | 0.0834 | Table E shows coverage rises with depth: **91.66%–99.16%** at stage four (TV **0.0084–0.0834**), **15.16%–66.22%** at stage three, and **0.16%–5.66%** at stages one and two. Thus, early-stage approximation is limited.
AuthorsOpenReview ↗

Response to Reviewer 33UR (PART III)

W6: Statistical Reliability and Interpretation of the Reported Improvements

Response. All results are averaged over five independent seeds (Appendix B.4). EvaKD reaches average ROUGE-L scores of 29.71 and 32.85, exceeding FDD by +1.55 and +0.58 points; on domain-specific benchmarks, it improves accuracy over the strongest intermediate-layer baselines by +0.024 and +0.030. We additionally report seed-level standard deviations and paired-bootstrap confidence intervals. The standard deviations reflect run-to-run variability, while the paired-bootstrap intervals quantify uncertainty in performance differences across evaluation examples.

Table F: Statistical analysis of the main experimental results.

Model PairMetricMethodMeanStd.95% CI of Difference
Llama3.1-8B → Llama3.2-1BROUGE-LTED28.160.073—
EvaKD29.710.061[1.526, 1.574]
Qwen3-4B → Qwen3-0.6BROUGE-LFDD32.270.084—
EvaKD32.850.067[0.563, 0.597]
Llama3.1-8B → Llama3.2-1BAccuracyTED0.3440.0017—
EvaKD0.3680.0015[0.0236, 0.0244]
Qwen3-4B → Qwen3-0.6BAccuracyFDD0.4540.0020—
EvaKD0.4840.0018[0.0294, 0.0306]

As Table F shows, standard deviations are 0.061–0.084 for ROUGE-L and 0.0015–0.0020 for accuracy. All four paired-bootstrap 95% CIs are positive: [1.526, 1.574], [0.563, 0.597], [0.0236, 0.0244], and [0.0294, 0.0306], supporting the reliability of the gains. Together, these statistics assess both run-to-run stability and evaluation-level uncertainty.

The abstract's maximum improvements, 7.01 ROUGE-L points and 7.4 accuracy percentage points, are absolute score differences, not relative improvements. We will clarify this wording and report their corresponding confidence intervals.

W7: Novelty Relative to Existing OT-Based and Feature-Dynamics Distillation Methods

Response. EvaKD's contribution is not simply applying OT to distillation; it aligns prediction-evolution steps and selects transferred knowledge according to student deficiency.

MultiLevelOT aligns token- and sequence-level logits for cross-tokenizer vocabulary mismatch; FDD aligns intermediate hidden features and their first-order changes. In contrast, EvaKD applies entropy-regularized OT to adjacent-layer prediction changes in sparse prediction space, then uses DAT to select knowledge addressing student deficiencies.

Table G: Conceptual comparison with related intermediate and OT-based distillation methods.

MethodTransported / Aligned ObjectAlignment SignalSelection MechanismMain Objective
MultiLevelOTToken- and sequence-level logit distributionsToken/sequence-level OT costsDistribution-level transportCross-tokenizer KD
FDDIntermediate features and feature changesFeature similarity and first-order dynamicsFeature-dynamics alignmentTransfer feature evolution
EvaKDAdjacent-layer prediction changesEvolution cosine cost + normalized depth costPEOT + deficiency-aware utility truncationFlexible and selective prediction-evolution transfer

EvaKD also outperforms FDD by +1.55 and +0.58 average ROUGE-L points on the two model pairs, as summarized in Table H:

Table H: Comparison of FDD and EvaKD on the two main model pairs (Average ROUGE-L).

Model PairFDD Avg. ROUGE-LEvaKD Avg. ROUGE-LImprovement
Llama3.1-8B → Llama3.2-1B28.1629.71+1.55
Qwen3-4B → Qwen3-0.6B32.2732.85+0.58

On domain-specific benchmarks, EvaKD further improves accuracy over the strongest intermediate-layer baselines by +0.024 and +0.030. We will clarify these distinctions in the related-work section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.