acceptodds
Under review as a conference paper at ICLR 2027

Neighbor Co-Occurrence Alone Should Not Define Supervision in Embedding Distillation

Abstract

Relational knowledge distillation for text embeddings is typically performed within mini-batches, making batch composition determine which teacher-defined relationships the student can observe. Random batches may miss a seed text's close teacher neighbors. Grouping these neighbors makes the relevant comparisons available, but grouping alone does not tell the student which neighbor the teacher considers closer to the seed text, or by how much. We introduce Neighborhood Similarity Transfer Knowledge Distillation (NeST-KD), which turns each mini-batch into a shared teacher-neighborhood subgraph. NeST-KD samples seed texts, retrieves their nearest neighbors in the teacher embedding space, and then matches teacher and student conditional similarity distributions over the resulting subgraph. This transfers graded differences in similarity while allowing other eligible texts to provide additional supervision from the same encoded examples. An auxiliary objective also matches teacher and student similarity distributions from each seed text to the other examples in the shared subgraph. Extensive experiments on nine standard sentence-embedding datasets show that NeST-KD improves over the strongest baseline by and absolute points for the 66M and 22M MiniLM students, respectively. Controlled studies show that correctly assigning teacher similarities, beyond merely selecting neighbors, improves downstream performance and relational fidelity.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
embedding distillation, knowledge distillation, sentence embedding

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_4hduOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper explores knowledge distillation for text embedding models. The authors observation is that standard mini batch training exposes very few of an anchor's teacher defined nearest neighbors. The authors decompose the distortion of a sparse teacher target into two terms, a candidate selection term, and a probability assignment term, minimized by the conditional teacher probabilities on that support. They show that zero conditional KL requires preserving all pairwise similarity gaps within a row. Their proposed method NeST-KD samples b anchors, merges their teacher neighborhoods into a deduplicated shared pool, encodes each distinct pooled text once, and applies a neighborhood KL loss for every pooled text with at least two observed teacher neighbors, and an auxiliary KL loss comparing each sampled anchor against all other pooled texts at a corpus median temperature. Experiment spans three teacher student pairs and nine datasets, with Wiki1M corpora and zero shot BEIR retrieval.

Critical strengths
  • The hypergeometric derivation and the numbers are evident under the illustrative setting, roughly 92% of anchors see fewer than two teacher neighbors. This is though exists, like in contrastive distillation work have long noted that in-batch negatives are weak, but the paper quantifies it precisely and, more importantly, uses it to motivate a specific design. The decomposition in Proposition 3.1 is a new contribution. The finding that shuffling targets costs 1.26 points is a demonstration that assignment carries information.

  • The gains are consistent across settings and survive domain shift. The out of domain gains i.e. 2.46 and 1.22 points are particularly relevant, since distillation methods that only memorize in-domain structure would be expected to degrade there. The retrieval transfer in Table 8, 29.14 vs. 23.32 nDCG@10 for TALAS is a meaningful check, since the distillation objective is similarity based. The efficiency profile of 139 ms/step, 8.18 GB mean memory is competitive with DSKD and TALAS and far below EMO.

Critical weaknesses
  • The theoretical results do not establish what the method actually optimizes, such as Proposition 3.1 minimizes DKL(rj∥πj)D_{KL}(r_j \| \pi_j) over supports and targets. But the training objective in Eq. 17 is the reverse direction, DKL(PjT∥PjS)D_{KL}(P_j^T \| P_j^S) over the observed support Cj(B)=N(j)∩PBC_j(B) = \mathcal{N}(j) \cap \mathcal{P}_B. These are different objects, the proposition characterizes optimal target construction given a support. What it does not adequately address is that the actual support used at training time is a random subset of the top-k neighborhood determined by pool overlap. For non-anchor texts, the observed support can be a small, arbitrary subset of the top-k, the paper concedes this but lacks an analysis of how far from optimal it is. The claim that conditional targets remain optimal on their observed supports is technically true, however, superficial. Any conditional distribution is optimal on its own support under the assignment term. The interesting question, whether supervising many low quality supports helps or hurts relative to supervising fewer high quality ones is remained to be answered by the theory and is only partially probed by the anchor rows only ablation, which loses only 0.54 points.
  • The comparison with TALAS and DistillCSE is not budget matched, and the author's own controls suggest the margin may be partly attributable to encoding budget. NeST-KD encodes a variable pool PB\mathcal{P}_B per update, which for b=128 and k=100 can be substantially larger than 128 texts. The authors mentioned that in-batch, neighbor encodes 128 texts and outperforms TALAS, which is a fair point. However, the comparison in Table 1 is against TALAS at its shown budget, and the ablation showing scores below NeST-KD by 0.23 points is used to argue that more candidates do no help. Without a budget matched comparison to the strongest external baseline, the 2.64 point gap is unjustified to the method's design.
  • The temperature definition in Eq. 15 is inherited without justification. τj=(s(1)T−s(k)T)/log⁡k\tau_j = (s_{(1)}^T - s_{(k)}^T)/\log k is a reasonable heuristic, but the paper does not analyze its effect on the assignment term, and the sensitivity analysis only varies kk and λaux\lambda_{aux}. Since the whole notion turns on the graded nature of targets, and grading is controlled entirely by τj\tau_j, this is a gap between the theory and the method.
Questions for authors
  • What is the average realized ∣PB∣|\mathcal{P}_B| across training for the default configuration b=128, k=100 on the 13,553 text corpus, and how many distinct texts are encoded per optimizer update on average? The authors should show a comparison in which NeST-KD's per-update encoding budget is capped at the baseline's.

  • For non-anchor texts in RBR_B, what fraction of their top-k neighbors are typically observed in the pool, and how does the loss weighting interact with support quality? Also, the authors should quantify how much of the gain comes from non-anchor rows specifically.

  • The training loss uses DKL(PT∥PS)D_{KL}(P^T \| P^S) while the target construction analysis uses DKL(r∥π)D_{KL}(r \| \pi). Since these have different gradient properties, can you justify why the reverse direction is the right criterion for target construction but not for training?

  • For sampled anchors, the neighborhood loss and auxiliary loss both compare the anchor to its observed neighbors with different normalizers and temperatures. Is there a risk of conflicting gradients on these shared pairs, and can you experiment with excluding observed neighbors from the auxiliary candidate set?

  • All results assume a strong frozen teacher. Given the method's reliance on precomputed teacher neighborhoods and temperatures, how sensitive is it to teacher quality?

1 comment
Reviewer_4hduOpenReview ↗

Official Comment by Reviewer 4hdu

Thank you for providing such a detailed response to my questions. The analysis is reasonable and satisfactorily clarifies my concern. I am inclined to raise my overall rating to Weak Acceptance (3).

Reviewer_g1eSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes NeST-KD for distilling text embedding models. It groups texts with teacher-selected neighbors and teaches the student how similar those neighbors are, while reusing texts in the group for additional supervision. The experiments suggest that correctly transferring these relationships matters beyond simply putting neighbors together.

Critical strengths
  • The central idea is clear, and the same-pool ablations support it well.
  • The experiments cover several teacher–student pairs, with three-seed main results and encouraging retrieval gains.
Critical weaknesses
  • The expanded neighbor pools make training costs harder to compare. Total costs, including preprocessing, would be useful.
  • Gains on the Wikipedia sentence benchmarks are small, and the Wiki1M results use only one seed.
  • Some aggregate standard deviations in Table 2 appear inconsistent and need checking.
Questions for authors
  1. Does the advantage remain when methods receive similar training budgets?
  2. Could you add more seeds for Wiki1M and clarify the standard deviations in Table 2?
Reviewer_moXFOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • This paper studies knowledge distillation for embedding models
  • They argue that distillation should distill the relationships between text embeddings (this in particular is not a novel), and specifically that under this framework, the selection of the neighbors used for computing these relationship matter a lot
  • They provide some theoretical results (section 3.2), which essentially say, that in order to get maximum signal under the KL loss of equation 4, we should select batch samples that are close neighbors under the teacher. They further show that under random sampling, the probability of this happening is very low
  • This motivates their algorithm. NeST-KD, which essentially mines teacher nearest neighbors to use for their KL loss
  • They show that this results in improved performance over a large number of baselines on a number of tasks
Critical strengths
  • Clear presentation and motivation. The paper is clearly written and well motivated. Section 3 explains how random batching can limit access to informative teacher neighborhoods and distinguishes candidate selection from the supervision assigned to those candidates. The proposed method follows naturally from this analysis.

  • Strong empirical results. The method performs well against a broad range of baselines across multiple downstream tasks. The additional teacher–student pairs evaluated in the appendix strengthen the evidence that the improvements extend beyond the main experimental configuration.

  • Informative ablations. Table 2 examines many of the major design decisions, including neighborhood exposure, teacher-probability assignment, and the use of non-anchor texts as supervision sources. In particular, the same-pool controls help isolate the benefits of graded targets and additional supervised queries from changes in the encoded examples.

Critical weaknesses
  • Cost of teacher preprocessing and neighbor mining. The paper would benefit from reporting the time required to compute teacher embeddings and construct the nearest-neighbor graph, both of which are excluded from Figure 7. Exhaustive neighbor search requires O(N2d)O(N^2d) computation for NN embeddings of dimension dd, potentially becoming a substantial preprocessing cost on larger corpora. The authors should specify the search implementation, whether retrieval is exact or approximate, and its runtime and memory requirements, particularly for Wiki1M.

  • Comparisons under matched student computation. NeST-KD encodes and backpropagates through both anchors and retrieved neighbors, so matching the nominal anchor batch size does not match the number of student-processed texts. With 100 neighbors per anchor, the expanded pool can contain up to 101 times as many texts before accounting for overlap. The fixed-size “In-batch, neighbor” control provides a useful comparison: it achieves 74.51, versus 75.95 for full NeST-KD, while encoding only 128 texts per update. Although Figure 7 reports per-step timings, reporting average pool sizes and total student-processed tokens would make the computational differences clearer. Including this control in Table 1, or writing the per-step timing directly in Table 1 would help establish how much of the full method’s improvement comes from its supervision design versus increased example exposure.

  • Alternative objectives for transferring neighborhood structure. The ablations establish the value of correctly assigned teacher similarities, but do not determine whether conditional KL is preferable to other relational objectives on the same neighborhoods. Comparisons with similarity-matrix regression or centered kernel alignment, while holding the encoded pool, candidate relationships, and supervised queries fixed, would help distinguish the benefits of neighborhood construction from those of the particular distillation loss.

Questions for authors
  • Sampling versus top-KK selection. Have the authors considered sampling KK candidates from the teacher similarity distribution during preprocessing and caching these in place of the deterministic top-KK neighbors? Sampling-based sparse distillation can provide unbiased teacher-target and gradient estimates under an appropriate loss formulation, as discussed in [1]. Proposition 3.1 establishes optimality for a fixed sparse-target approximation, but does not establish that top-KK selection is preferable for stochastic training. Could the authors evaluate this simple offline sampling alternative and clarify whether their conditional normalization preserves or changes its unbiasedness properties?

[1] Anshumann et al. Sparse Logit Sampling: Accelerating Knowledge Distillation in LLMs. ACL, 2025. https://aclanthology.org/2025.acl-long.885/

Comments on the paper

1 comment
AuthorsOpenReview ↗

Reference tables

Reference tables

We provide the relevant comparison tables below for ease of reference.

Bold marks the highest score among the three methods. NeST-Full is labeled NeST-KD in the manuscript.

Per-task scores

Main corpus: Qwen3-0.6B → H384

MethodEmotionBanking77TweetWiCMRPCSciTailSTS-BSICKSTS12Avg-InAvg-OutAvg.
TALAS61.13 ± 0.5284.16 ± 0.1571.52 ± 0.1166.32 ± 0.0784.85 ± 0.1080.91 ± 0.2772.39 ± 0.3170.72 ± 0.1567.80 ± 0.0866.61 ± 0.2476.66 ± 0.0773.31 ± 0.10
NeST-Lite64.14 ± 0.6187.54 ± 0.2371.82 ± 0.2266.55 ± 0.0883.19 ± 0.0782.04 ± 0.1476.17 ± 0.1273.35 ± 0.0572.77 ± 0.0968.96 ± 0.1978.45 ± 0.0775.29 ± 0.11
NeST-Full64.34 ± 0.4088.45 ± 0.0771.58 ± 0.2465.78 ± 0.0683.91 ± 0.1282.76 ± 0.1278.67 ± 0.0474.09 ± 0.0673.95 ± 0.1269.60 ± 0.1279.12 ± 0.0875.95 ± 0.09

Main corpus: BGE-M3 → H768

MethodEmotionBanking77TweetWiCMRPCSciTailSTS-BSICKSTS12Avg-InAvg-OutAvg.
TALAS65.83 ± 0.5889.72 ± 0.1975.38 ± 0.1166.96 ± 0.0787.58 ± 0.0282.90 ± 0.1277.51 ± 0.1875.82 ± 0.1168.11 ± 0.2070.10 ± 0.1379.92 ± 0.0576.65 ± 0.04
NeST-Lite67.47 ± 0.4590.75 ± 0.0974.32 ± 0.0363.76 ± 0.0985.28 ± 0.1983.42 ± 0.2180.63 ± 0.0878.15 ± 0.1173.67 ± 0.2170.62 ± 0.1780.93 ± 0.0877.49 ± 0.08
NeST-Full68.47 ± 0.0990.96 ± 0.0873.40 ± 0.0862.99 ± 0.0885.34 ± 0.0183.91 ± 0.0682.22 ± 0.0578.50 ± 0.0474.71 ± 0.1471.23 ± 0.0481.14 ± 0.0377.83 ± 0.01

Main corpus: Qwen3-4B → BERT-base

MethodEmotionBanking77TweetWiCMRPCSciTailSTS-BSICKSTS12Avg-InAvg-OutAvg.
TALAS69.01 ± 0.5290.44 ± 0.1374.13 ± 0.1768.84 ± 0.1584.60 ± 0.1081.78 ± 0.2979.24 ± 0.1076.53 ± 0.1073.12 ± 0.3072.36 ± 0.1980.10 ± 0.0177.52 ± 0.07
NeST-Lite71.33 ± 0.6991.77 ± 0.0874.55 ± 0.2867.14 ± 0.1384.39 ± 0.1984.24 ± 0.1580.54 ± 0.0177.54 ± 0.1676.03 ± 0.1173.00 ± 0.1981.42 ± 0.0778.61 ± 0.09
NeST-Full69.46 ± 0.3492.00 ± 0.1274.21 ± 0.1766.85 ± 0.1084.62 ± 0.0485.55 ± 0.1982.60 ± 0.0377.53 ± 0.1075.08 ± 0.0772.97 ± 0.1381.50 ± 0.0578.66 ± 0.01

Expanded 100K: Qwen3-4B → BERT-base

MethodBanking77TweetEmotionMRPCSciTailWiCSICKSTS12STS-BAvg.
TALAS93.1274.6771.4286.3087.0668.0980.2976.3482.2679.95
NeST-Lite93.3273.6571.7784.5290.6266.1580.8277.1183.1780.12
NeST-Full93.4372.9871.4484.8689.8265.2280.5178.5584.4280.14

Expanded 100K: BGE-M3 → H768

MethodBanking77TweetEmotionMRPCSciTailWiCSICKSTS12STS-BAvg.
TALAS92.4475.5865.1387.0985.3964.3978.1472.5681.0377.97
NeST-Lite92.7574.5669.6485.8790.2863.0179.5276.7383.6579.56
NeST-Full92.9474.5469.4586.1189.7261.8378.5978.2884.1179.51

Expanded 100K: Qwen3-0.6B → H384

MethodBanking77TweetEmotionMRPCSciTailWiCSICKSTS12STS-BAvg.
TALAS89.7973.8761.7585.9982.0765.7975.6773.9178.3176.35
NeST-Lite91.3273.1066.0984.2285.6967.1077.5275.8679.9277.87
NeST-Full91.6673.3166.8184.8286.1265.7977.2677.8180.8878.27

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.