- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
Knowledge distillation (KD) is a common and important technique in LLM training; however, existing text embedding distillation methods primarily focus on matching the final embeddings of teacher and student models, while intermediate-layer representations remain less explored. The research team noticed this gap and argues that the teacher model's intermediate-layer structure is also important and should be learned by the student model. To do so, the authors proposed a new technique, layerwise structure transport (LST). LST is a method with three components: 1. the relational geometry encoding converts intermediate representations into dimension-independent sample-relation matrices, allowing direct comparison of teacher and student geometries without additional projection layers.; 2. the soft layer alignment, which is a soft optimal transport alignment between the teacher and student layers to enable the student layers to aggregate the information across multiple teacher layers; and 3. the emergence-weighted ranking, which proposes depth-dependent weights from the teacher-student transformation and tries to preserve its layerwise positive-negative preferences. To test LST's performance, the research team uses a 36-layer, 2560-dim Qwen3-Embedding-4B model as the teacher and distills a 28-layer, 1024-dim Qwen3-0.6B-Base model as the student. The researchers evaluate the model's performance on a 15-MTEB-task and showed that LST could generalized to different settings (the Jina-stry and LEAF-style) and retains good performance for most of the tasks.
Critical strengths
- The problem is well motivated. The research team noticed that transitional KD methods mainly align the teacher model's final layer, while ignoring its intermediate representations, which contain a lot of information. Also, the authors do not assume that the teacher model's intermediate layers are informative; instead, they visualize the query-document distance for the teacher and student models and observe that the two models differ.
- The method design is coherent. The logic of the three components is clear and flows smoothly between them. Also, beyond ideas being coherent, the components' outputs are connected; for example, the OT plan from the soft alignment is used not only for structure supervision but also in the last component for the ranking branch.
- The experimental results look promising. The results show consistent improvements across both KD objectives. This suggests the model generalizes well.
- There is a lot of solid mechanistic analysis of the model. For example, the authors show the Teacher–Student layer correspondence in Figure 4 and the Emergence-Weighted Ranking diagnostics in Figure 5.
Critical weaknesses
- The necessity of emergence-aware weighting is insufficiently demonstrated. The authors claim that the importance of different layers depends dynamically on the teacher’s retrieval transition at different layer depths. However, this is not well proven. From Appendix E.3, Table 10, we can only see: $ \text{structure + Ranking KL} > \text{structure only} $ by . But from my perspective, it is not showing $ \text{emergence-weighted ranking KL} > \text{uniform ranking KL}$. The current setup cannot clearly show the contribution of this individual component.
- The improvement from the soft-OT over simpler mappings is limited. The team proposed alignment: 's performance is 64.18; while the provides a performance of 63.98. The improvement is . This is a valid improvement, but if accounting for the increased complexity of the soft-alignment, the improvement is not that great.
- The student self-alignment lacks a clear justification. The authors use:
for student-teacher alignment, and use for student self-alignment. But it is unclear why the team uses this specific approach for self-alignment.
Questions for authors
Same as the weaknesses.
Besides the listed weaknesses, the authors only test LST on the Qwen3-4B model as the teacher. My question is: is LST still valid if the research team changes the Teacher model?
3 comments
Thank you for the detailed response and the additional experiments. I appreciate that the authors added direct ablations for both emergence weighting and student self-alignment. These results address two of my main concerns and provide better support for the proposed design choices, especially the use of student self-alignment.
The clarification on the computational overhead of soft OT is also helpful. I agree that the additional cost is relatively small compared with the Transformer forward passes, and that it does not affect inference efficiency. The additional experiments with two different teachers further strengthen the empirical evaluation.
My remaining concern is that some of the improvements are relatively small, particularly for emergence weighting and the Qwen3-8B teacher, and it is unclear how stable these results are across different random seeds. It would also be helpful to clarify the difference between the test mean of 63.940 reported in the new ablations and the main result reported in the original paper.
Overall, I think the rebuttal addresses most of my concerns, and I have a more positive assessment of the paper after considering the new evidence.
Thank you for your constructive follow-up. Due to the limited rebuttal time and resource, the additional comparisons use a single matched seed.
In text embedding, the 4B teacher produces a stronger 0.6B student than the 8B teacher even without LST (Output KD: 63.240 versus 63.160). These results support 4B→0.6B as our primary configuration, which is also used by Jina v5-text (Akram et al., 2026, Section 4). A larger teacher does not yield a better small student here. The larger teacher–student capacity gap offers a plausible explanation, consistent with prior distillation studies (Cho and Hariharan, 2019; Mirzadeh et al., 2020). The 8B experiment therefore provides additional teacher coverage beyond our primary setting.
References
- Cho, J. H., & Hariharan, B. (2019). On the Efficacy of Knowledge Distillation. ICCV.
- Mirzadeh, S. I., et al. (2020). Improved Knowledge Distillation via Teacher Assistant. AAAI.
- Akram, M. K., et al. (2026). jina-embeddings-v5-text: Task-Targeted Embedding Distillation. arXiv:2602.15547.
Response to Reviewer hrcD
Dear Reviewer hrcD,
Thank you for recognizing the motivation and the connections among the three modules. We address your concerns below.
1. Emergence weighting versus Uniform Rank-KL
We added a direct Uniform Rank-KL control, replacing only the layer weights while retaining the structural objective, ranking targets, and alignment:
Emergence weighting improves over uniform weighting by 0.113 points. It allocates supervision according to changes in the teacher’s positive–negative margins across depth, emphasizing layers where retrieval discrimination emerges. This direct control supports the effectiveness of this task-related allocation of ranking supervision.
2. Practical value of soft OT
The comparison with fixed floor, nearest-depth, and row-hard mappings is already reported in Appendix E.2 (Table 8).
The layer-level OT computation is lightweight and confined to training. In the main Qwen3-4B→0.6B setting, the cross-layer plan contains only 28×36=1,008 entries, with 28×28 and 36×36 self-plans and 20 Sinkhorn iterations. These computations reuse representations from existing teacher/student forwards without additional Transformer forwards. The teacher, training projector, transport, and auxiliary objectives are all removed at inference.
For embedding systems, the practical priority is to improve quality while reducing the cost and latency of online encoding. Our additional training control shows that simply extending Output KD by 1,028 updates (24,604 total) yields 63.167, compared with 63.240 before the extension. Additional training alone provides no gain in this control. This is consistent with an extensively trained baseline approaching its performance limit, making improvements to the quality attainable by a compact student particularly valuable.
Embedding research reports sub-point aggregate gains as valuable quality improvements: Echo (Springer et al., 2025, Table 5) improves MTEB by +0.43 in its compute-matched comparison; Jina v5-text (Akram et al., 2026, Table 5) reports a +0.29 retrieval improvement from auxiliary regularization; and Qwen3 (Zhang et al., 2025, Table 3) gains +0.62 on MTEB by scaling from 4B to 8B. LST adds +0.89 over an already distilled Output KD student while retaining small-model deployment cost, demonstrating its practical value for embedding distillation.
3. Student self-alignment and the identity control
A_ST aggregates teacher ranking probabilities across depths. A_SS applies corresponding geometric aggregation to student probabilities, so both inputs to KL are depth-aggregated. Using identity would compare aggregated teacher probabilities with unaggregated student probabilities.
Both plans are row-normalized and detached in the ranking branch. Teacher targets are detached, while student probabilities retain gradients. A_SS is computed directly from student relation geometry and depth costs.
We added a direct control replacing only A_SS with identity in ranking:
Student self-alignment improves performance by 0.309 points, supporting the corresponding student-side aggregation.
4. Downstream validation with additional teachers
We added downstream distillation experiments with two additional teachers:
Both comparisons use E5 data and seed 42. LST improves over Output KD with both teachers, extending the downstream evidence to a different backbone, Gemma2, and a larger Qwen3 teacher.
References