- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper explores knowledge distillation for text embedding models. The authors observation is that standard mini batch training exposes very few of an anchor's teacher defined nearest neighbors. The authors decompose the distortion of a sparse teacher target into two terms, a candidate selection term, and a probability assignment term, minimized by the conditional teacher probabilities on that support. They show that zero conditional KL requires preserving all pairwise similarity gaps within a row. Their proposed method NeST-KD samples b anchors, merges their teacher neighborhoods into a deduplicated shared pool, encodes each distinct pooled text once, and applies a neighborhood KL loss for every pooled text with at least two observed teacher neighbors, and an auxiliary KL loss comparing each sampled anchor against all other pooled texts at a corpus median temperature. Experiment spans three teacher student pairs and nine datasets, with Wiki1M corpora and zero shot BEIR retrieval.
Critical strengths
-
The hypergeometric derivation and the numbers are evident under the illustrative setting, roughly 92% of anchors see fewer than two teacher neighbors. This is though exists, like in contrastive distillation work have long noted that in-batch negatives are weak, but the paper quantifies it precisely and, more importantly, uses it to motivate a specific design. The decomposition in Proposition 3.1 is a new contribution. The finding that shuffling targets costs 1.26 points is a demonstration that assignment carries information.
-
The gains are consistent across settings and survive domain shift. The out of domain gains i.e. 2.46 and 1.22 points are particularly relevant, since distillation methods that only memorize in-domain structure would be expected to degrade there. The retrieval transfer in Table 8, 29.14 vs. 23.32 nDCG@10 for TALAS is a meaningful check, since the distillation objective is similarity based. The efficiency profile of 139 ms/step, 8.18 GB mean memory is competitive with DSKD and TALAS and far below EMO.
Critical weaknesses
- The theoretical results do not establish what the method actually optimizes, such as Proposition 3.1 minimizes over supports and targets. But the training objective in Eq. 17 is the reverse direction, over the observed support . These are different objects, the proposition characterizes optimal target construction given a support. What it does not adequately address is that the actual support used at training time is a random subset of the top-k neighborhood determined by pool overlap. For non-anchor texts, the observed support can be a small, arbitrary subset of the top-k, the paper concedes this but lacks an analysis of how far from optimal it is. The claim that conditional targets remain optimal on their observed supports is technically true, however, superficial. Any conditional distribution is optimal on its own support under the assignment term. The interesting question, whether supervising many low quality supports helps or hurts relative to supervising fewer high quality ones is remained to be answered by the theory and is only partially probed by the anchor rows only ablation, which loses only 0.54 points.
- The comparison with TALAS and DistillCSE is not budget matched, and the author's own controls suggest the margin may be partly attributable to encoding budget. NeST-KD encodes a variable pool per update, which for b=128 and k=100 can be substantially larger than 128 texts. The authors mentioned that in-batch, neighbor encodes 128 texts and outperforms TALAS, which is a fair point. However, the comparison in Table 1 is against TALAS at its shown budget, and the ablation showing scores below NeST-KD by 0.23 points is used to argue that more candidates do no help. Without a budget matched comparison to the strongest external baseline, the 2.64 point gap is unjustified to the method's design.
- The temperature definition in Eq. 15 is inherited without justification. is a reasonable heuristic, but the paper does not analyze its effect on the assignment term, and the sensitivity analysis only varies and . Since the whole notion turns on the graded nature of targets, and grading is controlled entirely by , this is a gap between the theory and the method.
Questions for authors
-
What is the average realized across training for the default configuration b=128, k=100 on the 13,553 text corpus, and how many distinct texts are encoded per optimizer update on average? The authors should show a comparison in which NeST-KD's per-update encoding budget is capped at the baseline's.
-
For non-anchor texts in , what fraction of their top-k neighbors are typically observed in the pool, and how does the loss weighting interact with support quality? Also, the authors should quantify how much of the gain comes from non-anchor rows specifically.
-
The training loss uses while the target construction analysis uses . Since these have different gradient properties, can you justify why the reverse direction is the right criterion for target construction but not for training?
-
For sampled anchors, the neighborhood loss and auxiliary loss both compare the anchor to its observed neighbors with different normalizers and temperatures. Is there a risk of conflicting gradients on these shared pairs, and can you experiment with excluding observed neighbors from the auxiliary candidate set?
-
All results assume a strong frozen teacher. Given the method's reliance on precomputed teacher neighborhoods and temperatures, how sensitive is it to teacher quality?
Official Comment by Reviewer 4hdu
Thank you for providing such a detailed response to my questions. The analysis is reasonable and satisfactorily clarifies my concern. I am inclined to raise my overall rating to Weak Acceptance (3).