- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies knowledge distillation of text-embedding models on an unlabeled corpus. The student is trained with a cosine alignment loss plus an InfoNCE term whose negatives are mined from the teacher’s embedding space. Nearest neighbours in that space are often paraphrases, so standard hard-negative mining introduces false negatives. The proposed method (Ballpark Mining) uses a frozen teacher model to determine a false-negative ellipsoid by passing anchor input through the frozen teacher K = 8 times with dropout perterbation. Among the top-128 cosine candidates, those inside the ellipsoid are excluded, and the 8 nearest remaining candidates become hard negatives.
Critical strengths
The authors propose a simple, label-free method for hard-negatives mining. Using the teacher’s own variation under perturbation as a scale to determine false negative is intuitive. It needs no auxiliary models or labels.
Critical weaknesses
As the author also acknoledge it: "cosine thresholds achieve similar duplicate filtering at matched exclusion rates, so the experiments do not establish that ellipsoidal geometry is necessary for the observed gains." Without this comparison, the evidence only shows that removing near-duplicates helps on a duplicate-rich corpus, which is expected. The following baselines are missing:
- a global cosine-similarity threshold, or a rank-skip (e.g. sample negatives from ranks 30–100), matched for exclusion rate;
- a positive-relative threshold in the style of NV-Retriever (exclude candidates whose similarity exceeds a fraction of the positive’s);
- debiased contrastive loss (Chuang et al., 2020);
Questions for authors
- How does Ballpark compare with a global cosine threshold or a rank-skip that excludes the same fraction of candidates, on both corpora? The conclusion suggests the duplicate filtering is similar. Does the downstream utility differ?
- What is the computational cost of K = 8 teacher passes compared with the simple alternatives, especially when teacher at 27B scale?
- The abstract says the evaluation uses MTEB, but the experiment covers only STS-B, SciFact and NFCorpus. Why these tasks, and are there results on other MTEB categories such as classification, clustering or reranking?
3 comments
Response to Reviewer KYRb (2/3)
-
(a) Global threshold and rank skip. On Quora, the threshold minus Ballpark is +0.010 (CI [+0.004, +0.015], 5/5) and the rank skip minus Ballpark is +0.008 (5/5). The rank skip is matched in mean selected rank, so it excludes 17.2% instead of 22.9%. On Wikipedia (Hard 0.387), Ballpark at the radius that the backfill rule picks on a finer ladder (; the coarser ladder of the submitted PDF gave ) reaches 0.387 and the threshold matched to it 0.391 (+0.004, 5/5). The Quora rank skip applied to Wikipedia reaches 0.402.
-
(b) NV-Retriever. We read the rule literally: in distillation the positive of an anchor is its own teacher vector, so the rule excludes candidates above 0.95 or 0.90 of that score. This removes few candidates, and duplicate contamination of the negatives stays at 15.4% and 9.1% (Hard 21.9%, Ballpark 2.3%). On Wikipedia NV-95 reaches 0.385.
-
(c) Debiased loss, on Hard's negatives. is the measured contamination (which uses labels); is a label-free setting. Both stay at or below Hard, with lower duplicate Recall@10 (0.913, 0.909) and lower pair AUC (0.693, 0.701 against 0.718). Reweighting removes no negative, so the paraphrases stay in the list.
On "expected". In numbers, the NV-Retriever threshold recovers less than half of the gain (+0.015 of +0.034), and the debiased loss recovers none. These tests do not cover other tunings of either method.
Q2. Cost of the teacher passes. Measured on the 200k corpus rows; the passes cost about eight times the deterministic pass (wall-time ratio 8.0 for both decoder teachers, 9.33 for BGE).
| Teacher | GPU time of the passes | Cost at Modal list prices |
|---|---|---|
| BGE-base, 110M | 1.4 L4-hours | about 1.1 USD |
| Qwen3-Embedding, 4B | 1.7 A100-hours | about 3.6 USD |
| 27B decoder | 3.3 H100-hours | about 13 USD |
For comparison, one 30k-step training run of the 11M student takes 0.41 L4-hours (about 0.33 USD), so the 27B passes cost about as much as forty student runs. The cosine threshold, the rank skip, the NV threshold and the debiased loss need only the deterministic pass and the shared top-128 retrieval. The passes are a one-time cost per corpus and teacher, before training, and add nothing at student inference. trains as well as (0.397 against 0.395) and halves the cost.
Response to Reviewer KYRb (3/3)
Q3. "MTEB" in the abstract, the choice of tasks, and other categories.
Choice of tasks. We agree with the comment on the wording of our abstract. The composite uses three tasks (STS-B, SciFact, NFCorpus), and the appendix adds four BEIR datasets (ArguAna, FiQA, SciDocs, TREC-COVID). In the revised PDF, the abstract will replace "MTEB" with "the similarity-retrieval composite", and the experimental setup will name the three tasks. We chose them before the first run because they cover similarity and retrieval at small model size and their evaluation fits in the training loop. The recipe and the final-checkpoint rule were also fixed before the first run, and we do no checkpoint selection. One choice used the composite: on Quora was read from the composite of the ladder. In the revised PDF, the experimental setup will state this order of decisions.
Other categories. We scored the frozen Quora checkpoints on 18 further MTEB tasks in five categories, with five paired seeds, used for no selection.
| Method (Quora) | Class. (3) | Clust. (2) | Rerank. (3) | Pair cl. (3) | STS (7) | All (18) | vs. Hard |
|---|---|---|---|---|---|---|---|
| Align | 0.493 | 0.267 | 0.515 | 0.713 | 0.654 | 0.571 | -0.001, 1/5 |
| Hard | 0.478 | 0.202 | 0.518 | 0.761 | 0.659 | 0.572 | - |
| KD-Gauss | 0.489 | 0.245 | 0.522 | 0.739 | 0.666 | 0.578 | +0.006, 5/5 |
| Ballpark, | 0.496 | 0.219 | 0.532 | 0.769 | 0.680 | 0.589 | +0.017 [+0.012, +0.022], 5/5 |
| Ballpark, layer 10, | 0.506 | 0.225 | 0.538 | 0.770 | 0.692 | 0.597 | +0.025, 5/5 |
| Teacher (BGE-base) | 0.668 | 0.430 | 0.681 | 0.866 | 0.836 | 0.742 | - |
The 18 tasks rank the methods in the same order as the three-task composite, and Ballpark is ahead of Hard in every category and every seed. The categories do not move together: relative to Align, the mined-negative methods lose clustering (Hard 0.202 against 0.267) and gain pair classification (+0.048 to +0.057). Ballpark recovers part of the clustering loss (0.219 and 0.225). On the Wikipedia checkpoints, Ballpark at is level with Hard (-0.002), as on the composite. On BEIR-6, which the appendix of the submitted PDF reports, the order is also the same: Hard 0.138, Ballpark 0.161.
Tasks: Banking77, Emotion, TweetSentimentExtraction; TwentyNewsgroups, StackExchangeP2P; SciDocsRR, AskUbuntuDupQuestions, StackOverflowDupQuestions; SprintDuplicateQuestions, TwitterSemEval2015, TwitterURLCorpus; STS12 to STS16, SICK-R, BIOSSES.
We thank the reviewer again for the precise list of baselines. We trained all of them on both corpora, measured the cost of the passes, and broadened the evaluation. If any concern remains, please let us know, and we will answer it before the discussion ends.
Response to Reviewer KYRb (1/3)
We thank the reviewer for the precise review and the clear list of missing baselines. We are glad that the reviewer found our method "simple" and "label-free", and the teacher's own variation as a scale "intuitive". We trained all three baselines on both corpora, under the same teacher, candidate lists, eight-negative budget, recipe and paired seeds. We will add them to the revised PDF.
TL;DR
(a) Global cosine threshold and rank skip (W1, Q1). On Quora, the threshold matched for exclusion reaches 0.404 and the rank skip 0.402, against 0.395 for Ballpark and 0.361 for Hard. On Wikipedia, the matched threshold reaches 0.391 against 0.387 for Ballpark (Hard 0.387). This confirms that Ballpark's gain over Hard (+0.034, 5/5) comes from excluding the probable false negatives before training, as our conclusion anticipated; at the same exclusion rate, a cosine threshold reproduces it. In the revised PDF, we will state this in the abstract.
(b) NV-Retriever threshold (W1). It excludes only 0.4% and 1.3% of candidates and reaches 0.368 and 0.376. Duplicates still fill 15.4% and 9.1% of the negatives (Hard 21.9%).
(c) Debiased contrastive loss (W1). It reaches 0.354 and 0.359, at or below Hard.
"Removing near-duplicates helps, which is expected" (W1). We agree that removing near-duplicates helps. Our claim is about false negatives: many hard negatives that the teacher mines are duplicates of the anchor, so the student learns to push texts with the same meaning apart. Ballpark repairs this. For Quora anchors in labelled duplicate clusters, it lowers the share of duplicates among the negatives from 21.9% (Hard) to 2.3%. It also raises duplicate Recall@10 from 0.919 to 0.973. The two established remedies, (b) and (c), do not give this repair at their standard settings. The matched threshold in (a) gives it, but it copies its exclusion rate (22.9%) from Ballpark. Thus the open question is how much to exclude, and our backfill rule answers it without labels. If an exclusion leaves fewer than eight negatives, the miner backfills the empty slots with excluded candidates. The rule picks the largest radius whose mean backfill is below one slot. On a new corpus (CQADupStack), we hashed the rule's picks before any training, with no labels and no scores. Ballpark at the pick beat Hard in 5/5 seeds (+0.011), and the cosine cutoff that the same rule picked reached 0.367 against 0.351 for Hard. The same count also detects a wrong dropout rate and corrects it without labels.
Cost (Q2). The K=8 passes over 200k rows take 1.4 L4-hours for the 110M teacher and 3.3 H100-hours for the 27B teacher (about 1 to 13 USD), once per corpus and teacher. The simple alternatives need none. K=4 halves the cost with no measured loss.
MTEB (Q3). We agree with the comment on the wording of our abstract. In the revised PDF, the abstract will name the composite instead of MTEB. On 18 further MTEB tasks in five categories, Ballpark is +0.017 over Hard, ahead in every category and every seed.
The full predeclared test and the limits of the rule (Wikipedia, 27B teacher) are in our reply to Reviewer Bf8W (Q3).
Setup. Unless stated otherwise, a number is the utility composite (STS-B Spearman, SciFact and NFCorpus nDCG@10) at 30k steps, BGE-base (110M) to the 11M student on Quora, with the submitted top-128 lists, eight negatives, recipe and five paired seeds. A paired difference is given as mean, 95% CI and the number of seeds with that sign.
W1, Q1. The three baselines.