acceptodds
Under review as a conference paper at ICLR 2027

Transporting Layerwise Structure for Text Embedding Distillation

Abstract

Large text embedding models achieve strong performance but incur high deployment costs, motivating knowledge distillation (KD) into lightweight models. Existing text embedding distillation methods primarily focus on matching final embeddings, leaving the student's intermediate computation underconstrained. We first observe that strong text embedding teachers share a consistent layerwise trajectory: pairwise distances expand into a pronounced hump in the middle layers before contracting, while the positive–negative margin widens rapidly in the final third of the network. However, output-only KD does not transfer this intermediate trajectory, and existing layerwise objectives also fail to recover it when applied to text embeddings. These observations expose three challenges: (i) failure to preserve relations among input texts, (ii) inability to align teacher and student networks with different numbers of layers, and (iii) neglect of the late-layer separation trend between positive and negative samples. To address these challenges, we propose Layerwise Structure Transport (LST), which (i) encodes relations among input texts in a common space, (ii) learns a soft correspondence to align teacher and student networks of different depths, and (iii) transfers ranking preferences with greater weight at depths where positive–negative separation emerges. Experiments show that LST reliably transfers the teacher's layerwise trajectory and consistently improves student performance under all tested Output KD objectives. Our code can be found in the Reproducibility Statement.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Text Embeddings, Knowledge Distillation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_hrcDOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

Knowledge distillation (KD) is a common and important technique in LLM training; however, existing text embedding distillation methods primarily focus on matching the final embeddings of teacher and student models, while intermediate-layer representations remain less explored. The research team noticed this gap and argues that the teacher model's intermediate-layer structure is also important and should be learned by the student model. To do so, the authors proposed a new technique, layerwise structure transport (LST). LST is a method with three components: 1. the relational geometry encoding converts intermediate representations into dimension-independent sample-relation matrices, allowing direct comparison of teacher and student geometries without additional projection layers.; 2. the soft layer alignment, which is a soft optimal transport alignment between the teacher and student layers to enable the student layers to aggregate the information across multiple teacher layers; and 3. the emergence-weighted ranking, which proposes depth-dependent weights from the teacher-student transformation and tries to preserve its layerwise positive-negative preferences. To test LST's performance, the research team uses a 36-layer, 2560-dim Qwen3-Embedding-4B model as the teacher and distills a 28-layer, 1024-dim Qwen3-0.6B-Base model as the student. The researchers evaluate the model's performance on a 15-MTEB-task and showed that LST could generalized to different settings (the Jina-stry and LEAF-style) and retains good performance for most of the tasks.

Critical strengths
  1. The problem is well motivated. The research team noticed that transitional KD methods mainly align the teacher model's final layer, while ignoring its intermediate representations, which contain a lot of information. Also, the authors do not assume that the teacher model's intermediate layers are informative; instead, they visualize the query-document distance for the teacher and student models and observe that the two models differ.
  2. The method design is coherent. The logic of the three components is clear and flows smoothly between them. Also, beyond ideas being coherent, the components' outputs are connected; for example, the OT plan from the soft alignment is used not only for structure supervision but also in the last component for the ranking branch.
  3. The experimental results look promising. The results show consistent improvements across both KD objectives. This suggests the model generalizes well.
  4. There is a lot of solid mechanistic analysis of the model. For example, the authors show the Teacher–Student layer correspondence in Figure 4 and the Emergence-Weighted Ranking diagnostics in Figure 5.
Critical weaknesses
  1. The necessity of emergence-aware weighting is insufficiently demonstrated. The authors claim that the importance of different layers depends dynamically on the teacher’s retrieval transition at different layer depths. However, this is not well proven. From Appendix E.3, Table 10, we can only see: $ \text{structure + Ranking KL} > \text{structure only} $ by 63.84→64.1863.84 \rightarrow 64.18. But from my perspective, it is not showing $ \text{emergence-weighted ranking KL} > \text{uniform ranking KL}$. The current setup cannot clearly show the contribution of this individual component.
  2. The improvement from the soft-OT over simpler mappings is limited. The team proposed alignment: Relational Cost+Depth Regularization+Entropic OT\text{Relational Cost} + \text{Depth Regularization} + \text{Entropic OT}'s performance is 64.18; while the Fixed floor Map\text{Fixed floor Map} provides a performance of 63.98. The improvement is Δ=0.20\Delta = 0.20. This is a valid improvement, but if accounting for the increased complexity of the soft-alignment, the improvement is not that great.
  3. The student self-alignment lacks a clear justification. The authors use:

p~i,lT=∑mAlmSTpi,mT\tilde p_{i,l}^T=\sum_mA_{lm}^{ST}p_{i,m}^T for student-teacher alignment, and use p~i,lS=∑kAlkSSpi,kS\tilde p_{i,l}^S=\sum_kA_{lk}^{SS}p_{i,k}^S for student self-alignment. But it is unclear why the team uses this specific approach for self-alignment.

Questions for authors

Same as the weaknesses.

Besides the listed weaknesses, the authors only test LST on the Qwen3-4B model as the teacher. My question is: is LST still valid if the research team changes the Teacher model?

3 comments
AuthorsOpenReview ↗

Response to Reviewer hrcD

Dear Reviewer hrcD,

Thank you for recognizing the motivation and the connections among the three modules. We address your concerns below.

1. Emergence weighting versus Uniform Rank-KL

Your Concern (W1): A direct comparison between emergence weighting and Uniform Rank-KL is missing.

We added a direct Uniform Rank-KL control, replacing only the layer weights while retaining the structural objective, ranking targets, and alignment:

ConfigurationTest mean
Emergence-weighted LST63.940
Uniform Rank-KL63.827

Emergence weighting improves over uniform weighting by 0.113 points. It allocates supervision according to changes in the teacher’s positive–negative margins across depth, emphasizing layers where retrieval discrimination emerges. This direct control supports the effectiveness of this task-related allocation of ranking supervision.

2. Practical value of soft OT

Your Concern (W2): The gain over simpler mappings is limited; does it justify soft OT’s additional complexity?

The comparison with fixed floor, nearest-depth, and row-hard mappings is already reported in Appendix E.2 (Table 8).

The layer-level OT computation is lightweight and confined to training. In the main Qwen3-4B→0.6B setting, the cross-layer plan contains only 28×36=1,008 entries, with 28×28 and 36×36 self-plans and 20 Sinkhorn iterations. These computations reuse representations from existing teacher/student forwards without additional Transformer forwards. The teacher, training projector, transport, and auxiliary objectives are all removed at inference.

For embedding systems, the practical priority is to improve quality while reducing the cost and latency of online encoding. Our additional training control shows that simply extending Output KD by 1,028 updates (24,604 total) yields 63.167, compared with 63.240 before the extension. Additional training alone provides no gain in this control. This is consistent with an extensively trained baseline approaching its performance limit, making improvements to the quality attainable by a compact student particularly valuable.

Embedding research reports sub-point aggregate gains as valuable quality improvements: Echo (Springer et al., 2025, Table 5) improves MTEB by +0.43 in its compute-matched comparison; Jina v5-text (Akram et al., 2026, Table 5) reports a +0.29 retrieval improvement from auxiliary regularization; and Qwen3 (Zhang et al., 2025, Table 3) gains +0.62 on MTEB by scaling from 4B to 8B. LST adds +0.89 over an already distilled Output KD student while retaining small-model deployment cost, demonstrating its practical value for embedding distillation.

3. Student self-alignment and the identity control

Your Concern (W3): Why use student self-alignment A_SS rather than identity?

A_ST aggregates teacher ranking probabilities across depths. A_SS applies corresponding geometric aggregation to student probabilities, so both inputs to KL are depth-aggregated. Using identity would compare aggregated teacher probabilities with unaggregated student probabilities.

Both plans are row-normalized and detached in the ranking branch. Teacher targets are detached, while student probabilities retain gradients. A_SS is computed directly from student relation geometry and depth costs.

We added a direct control replacing only A_SS with identity in ranking:

ConfigurationTest mean
Student self-alignment A_SS63.940
Identity A_SS=I63.631

Student self-alignment improves performance by 0.309 points, supporting the corresponding student-side aggregation.

4. Downstream validation with additional teachers

Your Concern (additional teacher question): Does LST remain effective when the teacher changes?

We added downstream distillation experiments with two additional teachers:

Teacher → Qwen3-0.6B studentOutput KDLSTGain
BGE-multilingual-Gemma2-9B62.43463.028+0.595
Qwen3-Embedding-8B63.16063.329+0.169

Both comparisons use E5 data and seed 42. LST improves over Output KD with both teachers, extending the downstream evidence to a different backbone, Gemma2, and a larger Qwen3 teacher.

References

  • Springer, J. M., et al. (2025). Repetition Improves Language Model Embeddings. ICLR.
  • Akram, M. K., et al. (2026). jina-embeddings-v5-text: Compact and Robust Text Embedding Models using Task-Targeted Distillation. arXiv:2602.15547.
  • Zhang, Y., et al. (2025). Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models. arXiv:2506.05176.

Thank you for the detailed response and the additional experiments. I appreciate that the authors added direct ablations for both emergence weighting and student self-alignment. These results address two of my main concerns and provide better support for the proposed design choices, especially the use of student self-alignment.

The clarification on the computational overhead of soft OT is also helpful. I agree that the additional cost is relatively small compared with the Transformer forward passes, and that it does not affect inference efficiency. The additional experiments with two different teachers further strengthen the empirical evaluation.

My remaining concern is that some of the improvements are relatively small, particularly for emergence weighting and the Qwen3-8B teacher, and it is unclear how stable these results are across different random seeds. It would also be helpful to clarify the difference between the test mean of 63.940 reported in the new ablations and the main result reported in the original paper.

Overall, I think the rebuttal addresses most of my concerns, and I have a more positive assessment of the paper after considering the new evidence.

Thank you for your constructive follow-up. Due to the limited rebuttal time and resource, the additional comparisons use a single matched seed.

In text embedding, the 4B teacher produces a stronger 0.6B student than the 8B teacher even without LST (Output KD: 63.240 versus 63.160). These results support 4B→0.6B as our primary configuration, which is also used by Jina v5-text (Akram et al., 2026, Section 4). A larger teacher does not yield a better small student here. The larger teacher–student capacity gap offers a plausible explanation, consistent with prior distillation studies (Cho and Hariharan, 2019; Mirzadeh et al., 2020). The 8B experiment therefore provides additional teacher coverage beyond our primary setting.

References

Reviewer_hNnbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a training-only auxiliary framework for text embedding distillation called Layerwise Structure Transport (LST). It has three parts: (1) Relational Geometry Encoding, which captures relations between samples in the same batch using centered, trace-normalized Gram matrices; (2) Soft Layer Alignment, which learns correspondences between teacher and student layers via entropy-regularized optimal transport; and (3) Emergence-Weighted Ranking, which transfers positive vs. negative preferences with depth-dependent weights derived from changes in teacher margins.

The main experiments distill Qwen3-Embedding-4B into Qwen3-0.6B-Base under both Jina-style and LEAF-style output KD objectives. Over 15 MTEB tasks, the average score goes from 63.29 to 64.18 (Jina) and from 63.48 to 63.97 (LEAF). Appendix C.1 shows a further 0.92-point gain with Qwen3.5-0.8B as the student, and Appendix D.1 looks at similar teacher trajectories in a non-Qwen model family. The paper also includes geometry analysis, layer correspondence analysis, ranking margin analysis, and component ablations.

Critical strengths
  1. Well-motivated problem and coherent methodological framework.

The paper identifies a real gap in output-only KD: matching the final embeddings may not preserve the teacher's relational structure in the intermediate layers, and the observed teacher trajectories make a reasonable case for transferring more than the final output. LST tackles this from three angles (dimension-independent relational geometry, soft teacher-student layer alignment, depth-aware ranking supervision), the formulation makes sense, and it adds no inference cost, which is a big plus in practice. This matters for my assessment because the problem and the framework would be a useful contribution even if the downstream gains turn out smaller than reported.

  1. Extensive empirical investigation and consistent improvements over output-only distillation.

The experiments are broad: several intermediate-layer baselines, two output KD objectives, a second student (Qwen3.5-0.8B, +0.92 in Appendix C.1), and detailed analyses of geometry, layer mappings and rankings. LST improves both output objectives (+0.89/+0.49 Overall under Jina/LEAF), and the diagnostics show its intermediate geometry ends up closer to the teacher's, so it does more than match the final embeddings. This is good evidence that well-designed intermediate supervision can help text embedding distillation, though the size and source of the gains need more analysis (see Weaknesses).

Critical weaknesses
  1. The evidence that LST beats well-tuned output-only KD and alternatives is not yet convincing.

The central claim is a consistent gain over output-only KD, but the gains are modest (+0.89/+0.49), so the comparison has to be robust. Several intermediate-layer baselines fall far below Output KD (with Jina, Layerwise Hidden/Attention/TinyBERT drop from 63.29 to 57.70/50.04/46.45), and Appendix B.4 does not show that they were tuned (Q1), so it is hard to tell whether these drops come from the methods or from their settings. All distillation runs also use a single teacher (Qwen3-Embedding-4B) and a single E5-based training mixture, and the cross-family trajectory in Appendix D.1 does not show that the downstream gains carry over to other teachers.

Most importantly, I could not reproduce the main gain with the released code: Output KD matches the paper, but LST's gain over it averages −0.11 over three seeds instead of +0.89 (Q2). Since the case for the method rests on this gain, this is the main reason for my rating.

  1. Neither the individual components nor the transferred trajectory itself are shown to cause the gains.

The ablations do not show that each component is needed. Many differences are small and come without uncertainty estimates (0.01 pts for self-debiasing and 0.09-0.10 pts for centering or the depth prior in Table 7, 0.20-0.21 pts for soft transport over fixed mappings in Table 8). Emergence weighting is never tested on Rank-KL itself (uniform vs. emergence-weighted Rank-KL under otherwise identical settings), and there are no compute-matched Output KD or simpler relational distillation controls, so it is unclear whether the extra complexity pays off.

The diagnostics convincingly show that LST moves the student's intermediate geometry toward the teacher's, but not that this is what improves the final embeddings. The paper's own Table 2 suggests the two can diverge: Layerwise Hidden has a higher mean effective rank than Output KD (158.4 vs. 153.2) yet loses 5.6 pts Overall, while RAIL-KD has a much lower rank (117.8) and stays within 0.34 pts. In my reproduction the diagnostics match the paper closely while the MTEB gain disappears (Q2). A correlation analysis between these diagnostics and downstream scores (across configurations, seeds or checkpoints) would make this link convincing; without it, the mechanism claims are not supported.

Questions for authors
  1. Baseline hyperparameter fairness. How did you select the auxiliary loss coefficients and other key hyper parameters for Layerwise Hidden, Attention, TinyBERT, RAIL-KD, EMD? Are the baselines optimised independently under the same validation procedure? Do the authors have any sensitivity studies of auxiliary loss coefficient (especially for the configurations that perform much worse than Output KD)?

  2. Independent reproduction. I reran the main Jina setting with the released code on three seeds (global batch 64, one epoch of 23,576 updates, λ_str = λ_rank = 0.1, ε = 0.02, MTEB 1.14.12), adding an Output-KD-only path since the code does not include one. Evaluation and Output KD match the paper (teacher 72.53, Output KD 63.25 ± 0.36 vs. 63.29), but LST reaches only 63.14 ± 0.52 (vs. 64.18 ± 0.07): per-seed differences −0.54/+0.59/−0.38, mean −0.11 (95% CI [−0.35, +0.09]) vs. +0.89. One LEAF seed gives −0.02 (vs. +0.49). All geometry diagnostics we ran reproduce: mean effective rank 181.6 vs. 182.5, same-depth CKA 0.952 vs. 0.962, and 2.93 vs. 2.76 effective teacher layers at ε = 0.02. Could you release per-seed results and the exact code behind Table 1, and explain the much smaller seed variance you report?

  3. Paper vs. released code. (a) Eq. (B.3) uses the unsquared LEAF norm, but the code squares it. (b) The code cannot run Output KD alone, the five baselines, or the row-hard mapping. (c) The Q/K capture point for attention matching (before or after Qwen3's q_norm/k_norm) and the baseline loss coefficients are unspecified; with post-norm Q/K and coefficient 1, attention and TinyBERT drop only −2.6 and −6.9 (vs. −13.3 and −16.8). Which settings produced the reported numbers?

4 comments
AuthorsOpenReview ↗

Response to Reviewer hNnb [Part 1]

Dear Reviewer hNnb,

Thank you for the independent reproduction and detailed implementation checks. We have uploaded updated Supplementary Material containing the LEAF norm correction in the anonymized code and results for two additional seeds, as detailed below.

1. Baseline hyperparameter tuning

Your Concern (W1, Q1): Baseline tuning and auxiliary coefficient selection need clarification.

For Attention and TinyBERT, which suffered the largest drops, we expanded the coefficient search while retaining the original targets:

Auxiliary coefficientAttentionTinyBERT
1.0, original49.79446.113
0.158.22556.917
0.0161.49760.939

Even after expanded tuning, these layerwise baselines still fail to improve upon output-only distillation. Attention and TinyBERT reach 61.497 and 60.939, respectively, remaining below Output KD at 63.240, while LST reaches 63.940 under the same evaluation protocol.

2. Independent reproduction

Your Concern (W1, Q2): The reproduced results differ from those reported.

Our additional experiments show positive gains across both new seeds, further supporting LST's stability and effectiveness. We ran paired KD/LST training under the final Jina recipe:

Additional seedOutput KDLSTGain
763.17963.874+0.695
340763.16063.622+0.462

Both use 8×A100 PCIe 80GB, batch 8/GPU, accumulation 1, 23,576 updates, and LoRA 16/32/0.05. Both seeds show positive gains, averaging +0.579, supporting LST's stability and effectiveness. The updated Supplementary Material provides the paired results and reproducibility materials.

Two settings could explain the discrepancy in your reproduction:

  • Relation batch size. LST constructs sample relations within each forward pass; gradient accumulation does not combine samples across passes. Reducing the per-device batch size while increasing accumulation can therefore keep the effective global batch size at 64 but shrink the relation matrix from 192×192 to 96×96, changing the structural supervision.
  • Valid negatives. Ranking requires valid negative/neg triplets; missing or incorrectly parsed negatives can disable ranking supervision. Both additional LST runs begin with 64 triplets and ranking losses of approximately 0.364/0.249, confirming active ranking supervision.
AuthorsOpenReview ↗

Response to Reviewer hNnb [Part 2]

3. Downstream validation with additional teachers

Your Concern (W1): All distillation runs use a single teacher (Qwen3-Embedding-4B), and cross-family trajectory analysis does not establish that downstream gains carry over to other teachers.

We added downstream distillation experiments with two additional teachers:

Teacher → Qwen3-0.6B studentOutput KDLSTGain
BGE-multilingual-Gemma2-9B62.43463.028+0.595
Qwen3-Embedding-8B63.16063.329+0.169

Both comparisons use E5 data and seed 42. LST improves over Output KD with both teachers, extending the downstream evidence to a different backbone, Gemma2, and a larger Qwen3 teacher.

4. Component evidence, additional training, and geometric interpretation

Your Concern (W2): Direct weighting and compute controls are missing, and the link between geometry diagnostics and downstream gains is unclear.

4.1. Direct component controls

We added Uniform Rank-KL and identity student self-alignment controls:

SettingTest meanDefault LST minus control
Default LST63.940—
Uniform Rank-KL63.827+0.113
Identity student self-alignment63.631+0.309

Uniform replaces only the layer weights; identity replaces only A_SS in ranking, retaining the other core objectives. Default LST improves over these controls by 0.113 and 0.309, respectively, supporting emergence weighting and student self-alignment in ranking supervision. Soft alignment also improves over fixed floor/nearest mappings (63.681/63.677) by 0.258/0.263, supporting adaptive layer correspondence.

4.2. Additional training for Output KD

Additional training does not improve Output KD: 1,028 extra updates (24,604 total) yield 63.167, compared with 63.240 for the original run, while LST reaches 63.940. Raising the quality attainable by a compact student is crucial for real-time embedding systems, allowing them to meet accuracy requirements without deploying a larger, more expensive model.

4.3. Geometric diagnostics and downstream performance

Combining geometric diagnostics, component ablations, and downstream evaluation is an established practice in representation learning and distillation. Kornblith et al. (2019, Figure 3) jointly analyze inter-layer CKA and layerwise classification accuracy; RankMe (Garrido et al., 2023, Section 3.2) studies effective rank in relation to downstream representation quality; Similarity-Preserving KD (Tung and Mori, 2019) examines pairwise relation geometry and evaluates downstream classification and transfer performance.

LST follows this approach: effective rank characterizes the representation spectrum, CKA measures recovery of teacher–student sample relations, and the Uniform, Identity, and fixed-mapping controls test how the corresponding designs affect task performance. Geometric diagnostics characterize what is transferred; component controls evaluate whether the corresponding designs improve downstream quality.

5. LEAF objective and executable reproduction details

Your Concern (Q3): The LEAF norm disagrees with the equation; runnable entry points, attention Q/K capture, and coefficients need clarification.

Thank you for identifying this discrepancy in the anonymized code. Our original LEAF+LST experiments correctly used the mean unsquared L2 norm, consistent with the equation. The squared L2 term was introduced only when exporting the anonymized code and did not affect the reported experiments. The main Jina experiments use a separate cosine objective. The 11 existing tests and a zero-residual gradient check pass.

The original Attention/TinyBERT targets use pre-q_norm/k_norm, pre-RoPE q_proj/k_proj outputs, with masked raw-QK MSE, GQA expansion, head matching, and coefficient 1.0; the new scan retains that capture point. Baselines use 23,582 updates same to LST.

References

  • Kornblith, S., Norouzi, M., Lee, H., & Hinton, G. (2019). Similarity of Neural Network Representations Revisited. ICML.
  • Garrido, Q., Balestriero, R., Najman, L., & LeCun, Y. (2023). RankMe: Assessing the Downstream Performance of Pretrained Self-Supervised Representations by Their Rank. ICML.
  • Tung, F., & Mori, G. (2019). Similarity-Preserving Knowledge Distillation. ICCV.
Reviewer_hNnbOpenReview ↗

Official Review of Submission13727 by Reviewer hNnb

Thank you for the detailed response and the additional experiments.

However, I do not find W2 convincingly addressed yet. The new controls are single runs with small margins (relative gains of only 0.18% over Uniform Rank-KL, 0.49% over identity self-alignment, and 0.41% over fixed layer maps), which are within the 0.5% spread among your own LST seeds (63.62 to 63.94) and thus hard to separate from run-to-run variation. On the mechanism, the response cites prior practice instead of the requested correlation analysis, and the counterexamples remain: Layerwise Hidden has a higher effective rank than Output KD in Table 2 yet loses 5.6 points, and in my reproduction the diagnostics matched the paper while the MTEB gain did not appear.

In addition, changing only the auxiliary coefficient raised the attention and TinyBERT baselines by more than 11 and 14 points. This raises concerns about whether the baseline settings, including those of the untuned hidden, RAIL-KD and EMD baselines, have been fully optimized, and therefore about how much of LST's advantage over them comes from the method itself.

Therefore, I keep the orginial score.

Thank you for your feedback. We have re-uploaded our anonymized repository and provided our trained checkpoints to facilitate verification of LST's results.

Regarding baseline tuning, we would like to clarify that although the additional hyperparameter search substantially improves some baselines, this does not change our key observation: none of the layer-wise distillation baselines surpasses output-only KD. The LEAF paper also evaluates these layer-wise distillation methods and finds no improvement over output-only KD. This is consistent with current industrial practice, where output-only KD remains the dominant approach. In contrast, LST is the only evaluated layer-wise method that outperforms output-only KD. Therefore, the additional tuning does not change our main empirical conclusion.

We appreciate your time and thoughtful feedback.

Reference: LEAF: Knowledge Distillation of Text Embedding Models with Teacher-Aligned Representations. ACL 2026.

Reviewer_1EyPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LST, a training method for text-embedding knowledge distillation. It enables a lightweight student model not only to imitate the teacher’s final output, but also to learn how the teacher organizes text relationships across layers, how layers at different depths correspond, and how positive and negative samples become more separated in later layers. Experiments show that LST improves the student’s performance across multiple text tasks.

Critical strengths
  1. LST combines sample-relation encoding, soft alignment between teacher and student layers, and ranking supervision targeting the late-stage separation of positive and negative samples, making its overall design comprehensive.
  2. The paper compares LST with several layerwise-supervision baselines and evaluates it with two output-level KD objectives and an alternative student initialization, providing a broad validation.
  3. LST uses the teacher model and auxiliary losses only during training; it leaves the student’s inference architecture unchanged and adds no significant inference overhead.
Critical weaknesses
  1. In the main experiments, LST improves the 15-task average by only 0.49–0.89 points over Output KD, so the practical significance of the gains remains unclear.
  2. Since the distillation experiments use only Qwen3-Embedding-4B as the teacher, LST’s effectiveness on other architectures remains unverified. Although the authors analyze the layerwise trajectory of Harrier-27B, they do not report distillation experiments using it.
Questions for authors
  1. Overall, the paper is clearly written, and I have a positive impression of it. Its main limitation is the narrow scope of the experiments, which does not provide sufficient support for a perfect score.
3 comments
AuthorsOpenReview ↗

Response to Reviewer 1EyP

Dear Reviewer 1EyP,

Thank you for your careful review and for recognizing the motivation and unchanged student inference architecture. We discuss the practical significance of the gains, teacher selection, and additional validation below.

1. Practical significance of the gains

Your Concern (W1): The average gains are small; is their practical value sufficiently supported?

Embedding research reports sub-point aggregate gains as valuable quality improvements: Echo (Springer et al., 2025, Table 5) improves MTEB by +0.43 for its compute-matched comparison, Jina v5-text (Akram et al., 2026, Table 5) reports +0.29 retrieval improvement from auxiliary regularization, and Qwen3 (Zhang et al., 2025, Table 3) gains +0.62 on MTEB by scaling from 4B to 8B. LST adds +0.89 over an already distilled Output KD student while retaining small-model deployment cost, demonstrating its practical value for embedding distillation.

As additional evidence of stability, we added two seeds under the final Jina recipe, matching data, LoRA, optimizer, batch, and updates within each KD/LST pair:

Additional seedOutput KDLSTGain
763.17963.874+0.695
340763.16063.622+0.462

The mean gain is +0.579, with improvements on 14/15 and 12/15 tasks. The updated Supplementary Material provides per-task results, training configurations, complete numeric logs, and a verification script.

2. Teacher selection and validation beyond Qwen3-4B

Your Concern (W2, Q1): Distillation uses only Qwen3-4B; the experimental scope should be expanded to other teacher architectures.

The main Qwen3-4B→0.6B setting follows Jina v5-text (Akram et al., 2026, Section 4), enabling comparison with a representative strong teacher. To address the scope concern, we added downstream distillation experiments:

Teacher → Qwen3-0.6B studentOutput KDLSTGain
BGE-multilingual-Gemma2-9B62.43463.028+0.595
Qwen3-Embedding-8B63.16063.329+0.169

The +0.595 Gemma2 gain directly validates distillation with a different backbone, and the +0.169 Qwen3-8B gain validates a larger teacher. Both use E5 data and seed 42. Our available computational resources do not support distillation experiments with a teacher as large as Harrier-27B.

References

  • Springer, J. M., et al. (2025). Repetition Improves Language Model Embeddings. ICLR.
  • Akram, M. K., et al. (2026). jina-embeddings-v5-text: Compact and Robust Text Embedding Models using Task-Targeted Distillation. arXiv:2602.15547.
  • Zhang, Y., et al. (2025). Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models. arXiv:2506.05176.

Dear Reviewer, as the discussion period draws to a close, we would greatly appreciate hearing whether you have any remaining concerns after reading our rebuttal. We would be happy to address any further questions while the discussion is still open. Thank you again for your time and consideration.

Thank you for the authors' response. However, since Question W2 has not been adequately addressed, I have decided to maintain previous score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

Dear Reviewers, AC, SAC, and PC,

Thank you for the time and care you put into reviewing our manuscript. We appreciate the reviewers’ recognition of the motivation behind Layerwise Structure Transport (LST), the connections among its three components, and its ability to improve a compact student without adding inference cost. Your feedback helped us test the design more directly and broaden the evaluation. Below, we summarize the strengths highlighted in the reviews and the main updates from the discussion.


Key strengths noted by the reviewers

  • S1. A well-motivated problem. Reviewers highlighted the evidence that matching final embeddings alone leaves much of the teacher’s intermediate structure untransferred. (R2, R3)
  • S2. A coherent framework. Reviewers recognized how LST connects relational geometry, soft layer alignment, and ranking supervision, with the same layer correspondence informing both structural and ranking objectives. (R1, R2, R3)
  • S3. No added inference cost. LST improves a compact student while preserving its inference architecture, a practical advantage for embedding deployment. (R1, R2)
  • S4. Gains across distillation objectives. Reviewers noted that LST improves over Output KD under both Jina-style and LEAF-style objectives. (R1, R2, R3)
  • S5. Detailed analysis of what the student learns. Reviewers highlighted the analyses of relational geometry, teacher–student layer correspondence, and the emergence of retrieval margins. (R2, R3)

Key revisions in response to the comments

  • [R1, R2, R3] Broader teacher coverage. We added distillation experiments with BGE-multilingual-Gemma2-9B and Qwen3-Embedding-8B. With the same 0.6B student, LST improved over the respective Output KD baselines by +0.595 and +0.169 points, extending the evaluation to a different backbone and a larger Qwen3 teacher.
  • [R2] Expanded baseline hyperparameter search. We tested additional auxiliary loss coefficients for Attention and TinyBERT. Tuning substantially improved both baselines, and both still scored below Output KD. LST remained the only evaluated layerwise method to improve over Output KD.
  • [R1, R2] Consistent gains across additional seeds. In the main Qwen3-4B→0.6B setting, LST improved over Output KD on both new paired seeds, by +0.695 and +0.462 points, with gains on 14 and 12 of the 15 tasks, respectively.
  • [R2, R3] Direct ablations of ranking supervision. In matched single-seed comparisons, full LST scored 0.113 points above uniform Rank-KL and 0.309 points above identity student self-alignment. In the follow-up, R3 noted that these ablations addressed two of their main concerns, particularly the justification for student self-alignment.
  • [R2, R3] Additional training and deployment cost. Extending Output KD by 1,028 updates produced no gain in this control. We also clarified that soft alignment reuses representations from existing forward passes and adds no computation at inference.

R1 refers to Reviewer 1EyP, R2 to Reviewer hNnb, and R3 to Reviewer hrcD.


Thank you again for your careful evaluation and for the discussion. We appreciate the opportunity to clarify the method and share the additional evidence.

Sincerely,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.