acceptodds
Under review as a conference paper at ICLR 2027

Ballpark Mining: Region-Based Contrastive Distillation for Embeddings

22 views

Abstract

When compressing large embedding models through knowledge distillation, aligning student and teacher representations alone can leave substantial gaps in downstream performance. Contrastive objectives over hard negatives mined from the teacher’s embedding space are therefore commonly added to supply additional supervision. However, hard-negative mining suffers from a well-known limitation: the presence of false negatives. In particular, in duplicate-rich corpora, the nearest candidates to an anchor are often its paraphrases, so the contrastive loss pushes apart texts with the same meaning. Existing remedies can require similarity labels or auxiliary models. To address this, we propose Ballpark Mining, a label-free contrastive distillation method that uses teacher-derived regions to select negatives. Using the variability of a frozen teacher’s embeddings under stochastic perturbation, we define for each anchor a region of representations that the teacher cannot reliably distinguish from it. Candidates within this region are treated as probable false negatives, and the nearest candidates outside it are selected as negatives. We evaluate our proposed method with teachers ranging from 110M to 27B parameters on a deduplicated Wikipedia corpus and a duplicate-rich Quora corpus, using semantic similarity and retrieval tasks from MTEB and BEIR. On Quora, distilling a 110M-parameter teacher into an 11M-parameter student with Ballpark Mining reduces the fraction of duplicate negatives from 21.9% to 2.3% and increases duplicate recall@10 from 0.919 to 0.973. The gains extend to a 27B teacher, where Ballpark Mining improves the average score across semantic similarity and retrieval tasks from 0.211 with standard hard-negative mining to 0.285 when distilled on Quora. On Wikipedia, where duplicates are rare, performance remains comparable to standard hard-negative mining. In the 110M-to-11M setting, estimating regions at an intermediate teacher layer further improves the average score by approximately 3% on Quora and 6% on Wikipedia relative to final-layer regions. Together, our studies show that the perturbation behavior of the teacher provides a simple, label-free criterion for negative selection in embedding distillation, while identifying a corpus-transferable choice of region size as a key challenge.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
knowledge distillation, small models, embedding models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_KYRbOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies knowledge distillation of text-embedding models on an unlabeled corpus. The student is trained with a cosine alignment loss plus an InfoNCE term whose negatives are mined from the teacher’s embedding space. Nearest neighbours in that space are often paraphrases, so standard hard-negative mining introduces false negatives. The proposed method (Ballpark Mining) uses a frozen teacher model to determine a false-negative ellipsoid by passing anchor input through the frozen teacher K = 8 times with dropout perterbation. Among the top-128 cosine candidates, those inside the ellipsoid are excluded, and the 8 nearest remaining candidates become hard negatives.

Critical strengths

The authors propose a simple, label-free method for hard-negatives mining. Using the teacher’s own variation under perturbation as a scale to determine false negative is intuitive. It needs no auxiliary models or labels.

Critical weaknesses

As the author also acknoledge it: "cosine thresholds achieve similar duplicate filtering at matched exclusion rates, so the experiments do not establish that ellipsoidal geometry is necessary for the observed gains." Without this comparison, the evidence only shows that removing near-duplicates helps on a duplicate-rich corpus, which is expected. The following baselines are missing:

  • a global cosine-similarity threshold, or a rank-skip (e.g. sample negatives from ranks 30–100), matched for exclusion rate;
  • a positive-relative threshold in the style of NV-Retriever (exclude candidates whose similarity exceeds a fraction of the positive’s);
  • debiased contrastive loss (Chuang et al., 2020);
Questions for authors
  1. How does Ballpark compare with a global cosine threshold or a rank-skip that excludes the same fraction of candidates, on both corpora? The conclusion suggests the duplicate filtering is similar. Does the downstream utility differ?
  2. What is the computational cost of K = 8 teacher passes compared with the simple alternatives, especially when teacher at 27B scale?
  3. The abstract says the evaluation uses MTEB, but the experiment covers only STS-B, SciFact and NFCorpus. Why these tasks, and are there results on other MTEB categories such as classification, clustering or reranking?
3 comments
AuthorsOpenReview ↗

Response to Reviewer KYRb (1/3)

We thank the reviewer for the precise review and the clear list of missing baselines. We are glad that the reviewer found our method "simple" and "label-free", and the teacher's own variation as a scale "intuitive". We trained all three baselines on both corpora, under the same teacher, candidate lists, eight-negative budget, recipe and paired seeds. We will add them to the revised PDF.

TL;DR

  1. (a) Global cosine threshold and rank skip (W1, Q1). On Quora, the threshold matched for exclusion reaches 0.404 and the rank skip 0.402, against 0.395 for Ballpark and 0.361 for Hard. On Wikipedia, the matched threshold reaches 0.391 against 0.387 for Ballpark (Hard 0.387). This confirms that Ballpark's gain over Hard (+0.034, 5/5) comes from excluding the probable false negatives before training, as our conclusion anticipated; at the same exclusion rate, a cosine threshold reproduces it. In the revised PDF, we will state this in the abstract.

  2. (b) NV-Retriever threshold (W1). It excludes only 0.4% and 1.3% of candidates and reaches 0.368 and 0.376. Duplicates still fill 15.4% and 9.1% of the negatives (Hard 21.9%).

  3. (c) Debiased contrastive loss (W1). It reaches 0.354 and 0.359, at or below Hard.

  4. "Removing near-duplicates helps, which is expected" (W1). We agree that removing near-duplicates helps. Our claim is about false negatives: many hard negatives that the teacher mines are duplicates of the anchor, so the student learns to push texts with the same meaning apart. Ballpark repairs this. For Quora anchors in labelled duplicate clusters, it lowers the share of duplicates among the negatives from 21.9% (Hard) to 2.3%. It also raises duplicate Recall@10 from 0.919 to 0.973. The two established remedies, (b) and (c), do not give this repair at their standard settings. The matched threshold in (a) gives it, but it copies its exclusion rate (22.9%) from Ballpark. Thus the open question is how much to exclude, and our backfill rule answers it without labels. If an exclusion leaves fewer than eight negatives, the miner backfills the empty slots with excluded candidates. The rule picks the largest radius whose mean backfill is below one slot. On a new corpus (CQADupStack), we hashed the rule's picks before any training, with no labels and no scores. Ballpark at the pick beat Hard in 5/5 seeds (+0.011), and the cosine cutoff that the same rule picked reached 0.367 against 0.351 for Hard. The same count also detects a wrong dropout rate and corrects it without labels.

  5. Cost (Q2). The K=8K = 8 passes over 200k rows take 1.4 L4-hours for the 110M teacher and 3.3 H100-hours for the 27B teacher (about 1 to 13 USD), once per corpus and teacher. The simple alternatives need none. K=4K = 4 halves the cost with no measured loss.

  6. MTEB (Q3). We agree with the comment on the wording of our abstract. In the revised PDF, the abstract will name the composite instead of MTEB. On 18 further MTEB tasks in five categories, Ballpark is +0.017 over Hard, ahead in every category and every seed.

The full predeclared test and the limits of the rule (Wikipedia, 27B teacher) are in our reply to Reviewer Bf8W (Q3).

Setup. Unless stated otherwise, a number is the utility composite (STS-B Spearman, SciFact and NFCorpus nDCG@10) at 30k steps, BGE-base (110M) to the 11M student on Quora, with the submitted top-128 lists, eight negatives, recipe and five paired seeds. A paired difference is given as mean, 95% CI and the number of seeds with that sign.


W1, Q1. The three baselines.

Method (Quora)Excluded slotsCompositevs. Hard
Hard (eight nearest by cosine)00.361-
(c) Debiased contrastive loss (Chuang et al., 2020), τ+=0.22\tau^+ = 0.22 / 0.100.100 (reweights the loss)0.354 / 0.359-0.007 / -0.002
(b) NV-Retriever positive-relative threshold, 0.95 / 0.900.4% / 1.3%0.368 / 0.376+0.007 / +0.015, 5/5
Ballpark, α=3\alpha = 322.9%0.395+0.034, 5/5
(a) Rank skip, matched to Ballpark in mean selected rank17.2%0.402+0.041
(a) Global cosine threshold matched for exclusion (0.750)22.9%0.404+0.043
AuthorsOpenReview ↗

Response to Reviewer KYRb (2/3)

  • (a) Global threshold and rank skip. On Quora, the threshold minus Ballpark is +0.010 (CI [+0.004, +0.015], 5/5) and the rank skip minus Ballpark is +0.008 (5/5). The rank skip is matched in mean selected rank, so it excludes 17.2% instead of 22.9%. On Wikipedia (Hard 0.387), Ballpark at the radius that the backfill rule picks on a finer α\alpha ladder (α=2.5\alpha = 2.5; the coarser ladder of the submitted PDF gave α=2\alpha = 2) reaches 0.387 and the threshold matched to it 0.391 (+0.004, 5/5). The Quora rank skip applied to Wikipedia reaches 0.402.

  • (b) NV-Retriever. We read the rule literally: in distillation the positive of an anchor is its own teacher vector, so the rule excludes candidates above 0.95 or 0.90 of that score. This removes few candidates, and duplicate contamination of the negatives stays at 15.4% and 9.1% (Hard 21.9%, Ballpark 2.3%). On Wikipedia NV-95 reaches 0.385.

  • (c) Debiased loss, on Hard's negatives. τ+=0.22\tau^+ = 0.22 is the measured contamination (which uses labels); τ+=0.10\tau^+ = 0.10 is a label-free setting. Both stay at or below Hard, with lower duplicate Recall@10 (0.913, 0.909) and lower pair AUC (0.693, 0.701 against 0.718). Reweighting removes no negative, so the paraphrases stay in the list.

On "expected". In numbers, the NV-Retriever threshold recovers less than half of the gain (+0.015 of +0.034), and the debiased loss recovers none. These tests do not cover other tunings of either method.


Q2. Cost of the K=8K = 8 teacher passes. Measured on the 200k corpus rows; the K=8K = 8 passes cost about eight times the deterministic pass (wall-time ratio 8.0 for both decoder teachers, 9.33 for BGE).

TeacherGPU time of the K=8K = 8 passesCost at Modal list prices
BGE-base, 110M1.4 L4-hoursabout 1.1 USD
Qwen3-Embedding, 4B1.7 A100-hoursabout 3.6 USD
27B decoder3.3 H100-hoursabout 13 USD

For comparison, one 30k-step training run of the 11M student takes 0.41 L4-hours (about 0.33 USD), so the 27B passes cost about as much as forty student runs. The cosine threshold, the rank skip, the NV threshold and the debiased loss need only the deterministic pass and the shared top-128 retrieval. The passes are a one-time cost per corpus and teacher, before training, and add nothing at student inference. K=4K = 4 trains as well as K=8K = 8 (0.397 against 0.395) and halves the cost.

AuthorsOpenReview ↗

Response to Reviewer KYRb (3/3)

Q3. "MTEB" in the abstract, the choice of tasks, and other categories.

Choice of tasks. We agree with the comment on the wording of our abstract. The composite uses three tasks (STS-B, SciFact, NFCorpus), and the appendix adds four BEIR datasets (ArguAna, FiQA, SciDocs, TREC-COVID). In the revised PDF, the abstract will replace "MTEB" with "the similarity-retrieval composite", and the experimental setup will name the three tasks. We chose them before the first run because they cover similarity and retrieval at small model size and their evaluation fits in the training loop. The recipe and the final-checkpoint rule were also fixed before the first run, and we do no checkpoint selection. One choice used the composite: α=3\alpha = 3 on Quora was read from the composite of the α\alpha ladder. In the revised PDF, the experimental setup will state this order of decisions.

Other categories. We scored the frozen Quora checkpoints on 18 further MTEB tasks in five categories, with five paired seeds, used for no selection.

Method (Quora)Class. (3)Clust. (2)Rerank. (3)Pair cl. (3)STS (7)All (18)vs. Hard
Align0.4930.2670.5150.7130.6540.571-0.001, 1/5
Hard0.4780.2020.5180.7610.6590.572-
KD-Gauss0.4890.2450.5220.7390.6660.578+0.006, 5/5
Ballpark, α=3\alpha = 30.4960.2190.5320.7690.6800.589+0.017 [+0.012, +0.022], 5/5
Ballpark, layer 10, α=3\alpha = 30.5060.2250.5380.7700.6920.597+0.025, 5/5
Teacher (BGE-base)0.6680.4300.6810.8660.8360.742-

The 18 tasks rank the methods in the same order as the three-task composite, and Ballpark is ahead of Hard in every category and every seed. The categories do not move together: relative to Align, the mined-negative methods lose clustering (Hard 0.202 against 0.267) and gain pair classification (+0.048 to +0.057). Ballpark recovers part of the clustering loss (0.219 and 0.225). On the Wikipedia checkpoints, Ballpark at α=1\alpha = 1 is level with Hard (-0.002), as on the composite. On BEIR-6, which the appendix of the submitted PDF reports, the order is also the same: Hard 0.138, Ballpark 0.161.

Tasks: Banking77, Emotion, TweetSentimentExtraction; TwentyNewsgroups, StackExchangeP2P; SciDocsRR, AskUbuntuDupQuestions, StackOverflowDupQuestions; SprintDuplicateQuestions, TwitterSemEval2015, TwitterURLCorpus; STS12 to STS16, SICK-R, BIOSSES.


We thank the reviewer again for the precise list of baselines. We trained all of them on both corpora, measured the cost of the passes, and broadened the evaluation. If any concern remains, please let us know, and we will answer it before the discussion ends.

Reviewer_Bf8WOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies false negatives in contrastive distillation of text embeddings. Its proposed method, Ballpark Mining, uses repeated teacher forward passes with dropout to estimate an ellipsoidal region around each anchor. Candidates inside this region are excluded as probable false negatives, and nearby candidates outside it provide contrastive supervision alongside representation alignment. Experiments use duplicate-rich Quora questions and deduplicated Wikipedia sentences, with teachers ranging from 110M to 27B parameters. On Quora, the method substantially reduces duplicate-negative contamination and improves duplicate retrieval and downstream performance over standard hard-negative mining. Final-layer regions provide little additional benefit on Wikipedia, while intermediate-layer regions improve results on both corpora. The paper also derives an exact geometric bound for preserving linear score order under ellipsoidal perturbations.

Critical strengths

The paper directly measures the targeted failure. On Quora, final-layer Ballpark Mining reduces duplicate-negative contamination from 21.9% to 2.3%, improves duplicate Recall@10 from 0.919 to 0.973, and raises the utility composite from 0.361 to 0.395. Linking contamination measurements to training outcomes strengthens the empirical argument.

The evaluation transparently reports limitations. These include limited gains on deduplicated Wikipedia, lower Pair AUC than alignment-only training, and no improvement from larger teachers under the fixed recipe. Five paired seeds for the main comparisons and additional BEIR evaluations further strengthen the evidence.

Critical weaknesses

The experiments do not isolate the contribution of the proposed regions. Section 5 acknowledges comparable duplicate filtering from cosine thresholds at matched exclusion rates, but provides no downstream training comparison. Moreover, HARD ranks negatives by cosine similarity, whereas Ballpark uses ellipsoid distance, confounding filtering with ranking. This particularly complicates interpreting intermediate-layer gains on Wikipedia, where exclusions are uncommon. Matched cosine-filtering and ellipsoid-ranking-without-filtering controls are needed to establish the distinctive value of perturbation-derived regions.

The broader significance and semantic justification remain unclear. Effective embedding distillation can use direct alignment [1] or relational/optimal-transport alignment [2], so InfoNCE-based negative mining is not intrinsic to distillation. Although the paper includes alignment and regression baselines, its fixed-recipe comparisons leave the advantage over well-tuned alternatives unclear. Furthermore, the theory characterizes separation within a prescribed ellipsoid without establishing why teacher perturbations define meaningful semantic boundaries. The necessity of the proposed machinery and the semantic interpretation underpinning it therefore remain insufficiently supported.

[1]: LEAF: Knowledge Distillation of Text Embedding Models with Teacher-Aligned Representations (ACL 2026)

[2]: EMO: Embedding Model Distillation via Intra-Model Relation and Optimal Transport Alignments (EMNLP 2025)

Questions for authors

How does Ballpark compare with cosine-threshold filtering at matched exclusion rates, using the same candidate pool and negative count? How does ellipsoid-distance ranking perform without exclusion?

What empirical coverage does α=1 achieve on fresh teacher perturbations? How sensitive are coverage and downstream performance to the number of passes and dropout settings?

Can the backfill-based radius rule be fixed in advance and evaluated on a new corpus without consulting duplicate labels or downstream evaluation scores?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Bf8W (1/3)

We thank the reviewer for the careful reading and for asking for the experiments that decide what the paper contributes. We are glad that the reviewer found that the paper "directly measures the targeted failure". We ran every experiment that the review asked for. We will add them to the revised PDF.

TL;DR

  1. Matched cosine filtering (W1, Q1). At Ballpark's exclusion rate, a cosine cutoff reaches 0.404 against 0.395 for Ballpark on Quora (5/5 seeds), and 0.391 against 0.387 on Wikipedia.

  2. Region ranking without filtering (W1, Q1). Ranking by region distance without exclusion reaches 0.365 on Quora (Hard 0.361) and 0.383 on Wikipedia (Hard 0.387). The intermediate-layer gains on Wikipedia come from the layer-10 ordering: layer-10 cosine order without exclusion matches the layer-10 region (0.402).

  3. Distinctive value of the regions (W1). Ballpark's gain over Hard (+0.034, 5/5) comes from excluding the probable false negatives before training, as our conclusion anticipated. The cutoff above repeats the gain only at Ballpark's exclusion rate, and the label-free backfill rule sets that rate (Q3). In the revised PDF, we will state this in the abstract.

  4. Well-tuned alternatives (W2). Re-tuning lifts every method and keeps the order: Align 0.384, KD-Gauss 0.409, Ballpark 0.422. In the revised PDF, we will cite LEAF and EMO.

  5. Semantic justification (W2). Empirically, the candidates inside the region are mostly duplicates of the anchor (judged paraphrases: 76.0% on Quora, 97.5% on Wikipedia; duplicate negatives fall from 21.9% to 2.3%). Theorem 1 and its corollary formalise the geometric side of this observation: no linear score can order a candidate inside the region against the anchor's perturbed views.

  6. Coverage and sensitivity (Q2). At α=1\alpha = 1 the region covers 0.88 to 0.93 of fresh views (nominal 0.95); at α≥1.5\alpha \ge 1.5 at least 0.9998. KK changes utility by at most 0.003. A wrong dropout rate costs up to 0.021, and the backfill rule detects and corrects it without labels.

  7. Prospective test of the radius rule (Q3). Yes. We hashed the rule's picks and the hypothesis before any training on a new corpus (CQADupStack), without duplicate labels or evaluation scores. The rule selected α=3\alpha = 3; Ballpark beat Hard in 5/5 seeds (+0.011) and came within 0.01 of the rule's cosine cutoff.

Setup. Unless stated otherwise, a number is the utility composite (STS-B Spearman, SciFact and NFCorpus nDCG@10) at 30k steps, BGE-base (110M) to the 11M student on Quora, with the submitted top-128 lists, eight negatives, recipe and five paired seeds. A paired difference is given as mean, 95% CI and the number of seeds with that sign.


W1, Q1. Isolating the regions. Same candidate pool and eight negatives throughout.

MethodQuoraWikipedia
Hard (eight nearest by cosine)0.3610.387
Region-order: region distance ranks the list, no exclusion0.3650.383
Ballpark: region exclusion and region ranking0.395 (α=3\alpha = 3)0.387 (α=2.5\alpha = 2.5)
Cosine cutoff at the region's exclusion rate0.4040.391
  • Ranking without exclusion. Region-order shares 92% of Hard's negatives. Its difference from Hard is +0.004 (CI [-0.000, +0.009]) on Quora and -0.004 (CI [-0.008, -0.000]) on Wikipedia.

  • Exclusion. The matched cosine cutoff is +0.010 above Ballpark on Quora (CI [+0.004, +0.015], 5/5) and +0.004 on Wikipedia (5/5). On Wikipedia, Ballpark uses the radius that the backfill rule picks on a finer α\alpha ladder (α=2.5\alpha = 2.5; the coarser ladder of the submitted PDF gave α=2\alpha = 2).

  • Intermediate layers on Wikipedia. The layer-10 gains come from the layer-10 ordering. Layer-10 cosine order with no exclusion reaches 0.402 (+0.015 over Hard, 5/5), equal to the layer-10 region (0.402). Layer-10 region distance order with no exclusion reaches 0.399.

AuthorsOpenReview ↗

Response to Reviewer Bf8W (2/3)

W2. Well-tuned alternatives and significance.

Answer. Re-tuning does not change the order. With the probable false negatives excluded, mined negatives beat the re-tuned alignment and regression losses.

Quora, three paired seeds. The first column re-runs the submitted recipe on the same three seeds.

MethodSubmitted recipe (lr 3e-4, 30k steps)lr 1e-360k steps
Align (cosine alignment only)0.3520.3840.383
KD-Gauss (regression)0.3900.4090.413
Hard0.3580.393-
Ballpark0.3910.422-
  • Order. At lr 1e-3, Ballpark minus KD-Gauss is +0.013 (CI [+0.010, +0.017], 3/3), and Hard minus KD-Gauss is -0.017 (0/3). Among the contrastive methods, only Hard falls below re-tuned KD-Gauss.

  • Temperature and loss weight. At τ=0.1\tau = 0.1, τ=0.02\tau = 0.02 and InfoNCE weight 0.4, each method moves by at most 0.010 from its submitted-recipe score on the same three seeds (Hard 0.358 to 0.358 / 0.364 / 0.351; Ballpark 0.391 to 0.382 / 0.390 / 0.396). The worst Ballpark setting stays above the best Hard setting.

  • Why mined negatives matter. We agree that InfoNCE with mined negatives is not intrinsic to distillation. Our claim is narrower: when a recipe adds mined negatives, false negatives damage the student on a duplicate-rich corpus, and excluding them repairs this. Without exclusion, re-tuned Hard barely beats re-tuned Align (0.393 against 0.384), and under the 27B teacher it falls below Align (0.211 against 0.262, 11M student, two seeds). With exclusion, Ballpark reaches 0.422.

  • LEAF and EMO. In the revised PDF, we will cite LEAF as direct alignment and EMO as relational and optimal-transport alignment. We will also state that our Align and KD baselines are losses implemented under one shared recipe, not reproductions of those systems.


W2. Semantic justification of the theory.

Answer. Theorem 1 and its corollary are geometric statements, motivated by an empirical observation.

Empirically, the candidates inside the region are mostly duplicates of the anchor. An LLM judge labels 76.0% of sampled Quora exclusions and 97.5% of Wikipedia exclusions as paraphrases, and on Quora its agreement with the QQP labels is 0.82 (Cohen's kappa 0.51). For anchors in labelled duplicate clusters, the share of negatives that are duplicates falls from 21.9% (Hard) to 2.3% (Ballpark), and duplicate Recall@10 rises from 0.919 to 0.973.

Theorem 1 and its corollary formalise the geometric side of this observation: a candidate inside the region cannot be ordered against the anchor's perturbed views by any linear score. They do not establish that teacher perturbations follow semantic boundaries, and in the revised PDF we will state this limit explicitly.

AuthorsOpenReview ↗

Response to Reviewer Bf8W (3/3)

Q2. Coverage and sensitivity.

Answer. Coverage at α=1\alpha = 1 is slightly below the nominal 95%, and in the revised PDF we will state this. KK matters little. The dropout rate sets the size of the region, and the backfill rule corrects a wrong rate without labels.

Coverage. On 4,000 Quora anchors we drew eight new dropout views per anchor, which the region did not see, and fitted the region from the first KK of 32 views:

Coverage at α=1\alpha = 1K=4K = 4K=8K = 8K=16K = 16K=32K = 32
p=0.05p = 0.050.8770.8870.9140.930
p=0.1p = 0.1 (paper)0.8880.8890.9120.929
p=0.2p = 0.20.8900.8900.9110.928

Coverage rises with KK and does not depend on pp. At α≥1.5\alpha \ge 1.5 it is at least 0.9998, so the operating radius α=3\alpha = 3 contains essentially every fresh view.

Downstream sensitivity (α=3\alpha = 3; reference K=8K = 8, p=0.1p = 0.1: 0.395):

Region cacheExcludedBackfilled slotsCompositevs. reference
K=4K = 433.6%1.340.397+0.003, 3/5
K=16K = 1619.2%0.490.396+0.001, 3/5
p=0.05p = 0.052.0%0.000.378-0.016, 0/5
p=0.2p = 0.296.0%7.370.374-0.021, 0/5
p=0.05p = 0.05, rule's pick α=4\alpha = 414.4%0.320.395-0.000
p=0.2p = 0.2, rule's pick α=2\alpha = 215.3%0.250.393-0.002

K=4K = 4 trains as well as K=8K = 8 and halves the number of perturbed passes. At a fixed α\alpha, a lower dropout rate makes the region exclude almost nothing and a higher one almost everything. The backfill count flags both without labels (0.00 and 7.37, against 0.68 at the reference). We fixed the rule's new picks before training, and at those picks both students return to the reference level.


Q3. Prospective test of the radius rule.

Answer. Yes. The rule takes the largest α\alpha on the ladder whose mean backfill is below one slot; we applied the same rule to a cosine ladder. The corpus is 200k CQADupStack titles from twelve StackExchange forums, not used in the paper (18% of rows in a labelled duplicate cluster, against 75% on Quora). We hashed the rule's picks (α=3\alpha = 3, 30% cosine cutoff) and the hypothesis before building the training tables. The hypothesis: Ballpark beats Hard in 5/5 seeds, and its mean is within 0.01 of the cutoff's. No duplicate label and no evaluation score entered the picks; the labels only built the corpus before the freeze and measured contamination after training.

CQADupStack (five paired seeds)CompositeDuplicate slotsRecall@10
Align0.291-0.410
Hard0.35111.3%0.365
Ballpark, rule's α=3\alpha = 30.3624.5%0.404
Cosine cutoff, rule's 30%0.3672.3%0.422

Both parts held. Ballpark beat Hard in 5/5 seeds (+0.011, CI [+0.006, +0.016]), and its mean was within 0.01 of the cutoff (-0.005, CI [-0.012, +0.002]). The rule selected the same α\alpha as on Quora. It has limits, which we report: on Wikipedia its region pick is +0.000 over Hard, and under the 27B teacher its pick did not train best.


We thank the reviewer again. If any concern remains, please let us know, and we will answer it before the discussion ends.

Reviewer_fwKMOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies false negatives in contrastive distillation of text embedding models from unlabeled corpora. Ballpark Mining estimates an anchor-specific diagonal ellipsoid from repeated stochastic teacher outputs, excludes nearby candidates inside the region, and selects eight surviving negatives by the region distance. Student training then uses the same cosine-alignment and InfoNCE losses as the hard-negative baseline; regions are used only during preprocessing.

The authors conduct experiments on a deduplicated Wikipedia corpus and a duplicate-rich Quora corpus. From results, on Quora, final-layer Ballpark Mining at alpha = 3 reduces labeled duplicate-negative contamination from 21.9% to 2.3%, raises duplicate Recall@10 from 0.919 to 0.973, and improves the three-task composite from 0.361 to 0.395 relative to ordinary hard mining. Final-layer filtering provides little benefit on Wikipedia. Intermediate-layer regions improve the composite further, and additional teacher/student configurations test the pattern at larger compression ratios.

Critical strengths
  1. A consequential failure mode is connected to observable training behavior. On Quora, hard mining increases duplicate-negative contamination and harms duplicate retrieval, while the proposed selection recovers Recall@10 and improves downstream utility. The paired five-seed main study, training-horizon curves, contamination measurements, and additional BEIR evaluation provide a reasonably coherent empirical case for filtering overly close candidates in duplicate-rich distillation corpora.
  2. The paper investigates useful boundary conditions and reports unfavorable results. Final-layer filtering brings little benefit on Wikipedia, Pair AUC remains below alignment-only distillation, KD-Gauss is competitive, and larger teachers do not improve the fixed small student. The method adds no student inference cost, and the intermediate-layer and model-scale studies broaden its practical interest.
Critical weaknesses
  1. Additional details on the teacher and perturbation process would improve reproducibility. It would be helpful to specify the exact 27B teacher model/checkpoint, together with the dropout locations and probabilities used for each teacher. These details are particularly relevant because the perturbation statistics determine the estimated variances and exclusion regions. Providing these configurations would help readers better understand and reproduce the negative-selection procedure.
  2. Some architecture and initialization descriptions would benefit from clarification. Appendix C.1 describes all students as randomly initialized (p. 17, lines 913–917), while Table 2 also includes students initialized from pretrained checkpoints. Clarifying which configurations the appendix description applies to would improve consistency. Similarly, Appendix A.3 mentions two independently trained output maps (p. 12, lines 636–639), but their roles and connection to the main method and Algorithm 1 could be explained more explicitly. A brief clarification would help readers connect the architectural discussion with the reported experiments.
  3. It is recommended to provide more experiments to support the proof of contributions. Only the main methodological contribution is not isolated by the training comparisons. HARD selects by cosine similarity, whereas BALLPARK both excludes candidates and reranks them by a variance-weighted distance; the intermediate-layer variant changes the selection space as well. On Wikipedia, only about 7% of anchors have exclusions, yet the reported median overlap with HARD is only two of eight negatives, making reranking a plausible contributor. The conclusion explicitly acknowledges that cosine thresholds achieve similar duplicate filtering at matched exclusion rates, but does not report downstream student training for that control. Adding this control experiment can strengthen the verification of the contribution.
  4. The interpretation and calibration of the regions are insufficiently established. Equation (4) uses the empirical 95th percentile of only eight leave-one-out distances; this construction does not by itself imply 95% coverage of a fresh stochastic view under the final fitted metric. Theorem 1 and Corollary 1 are valid geometric statements, but neither establishes that membership implies paraphrase status nor that an outside candidate is safe for the actual task score or learned student. The alpha = 3 operating point was checked using duplicate labels, while the label-free backfill rule is explicitly post hoc and tested on only two corpora. These issues weaken the claim that the full selection procedure offers a calibrated, portable criterion without labeled calibration.
  5. There is an unresolved cross-reference, “Appendix ??,” on p. 16, line 854, which could be corrected during revision.
Questions for authors
  • Under the same teacher, candidate pools, eight-negative budget, and paired seeds, how does downstream utility compare with matched-exclusion cosine thresholds, variance-distance reranking without exclusion, and cosine-ranked survivors after the same Ballpark exclusions? Can these controls also be run for the layer-10 variant?
  • Is the 95% statement intended as a nominal target or an empirical coverage result? Please provide fresh-view coverage measurements or a justification for Equation (4), including how the leave-one-out scores relate to the final K-view metric.
  • Are duplicate-diagnostic anchors, candidate texts, and entire duplicate clusters disjoint from distillation training, or is the evaluation transductive? Please specify the Recall@10 denominator and negative-pair sampling, and provide the judge modelsettings and human checks behind the 76.0% and 97.5% paraphrase figures.
5 comments
AuthorsOpenReview ↗

Response to Reviewer fwKM (1/4)

We thank the reviewer for the careful reading and the concrete suggestions. We are glad that the reviewer found that "a consequential failure mode is connected to observable training behavior" and that the paper "reports unfavorable results". We ran every experiment that the review asked for. We will add them to the revised PDF.

TL;DR

  1. Isolation of the methodological contribution (W3, Q1). We added new training runs to answer this. Ranking by region distance adds only +0.004 over Hard, so Ballpark's gain (+0.034, 5/5) comes from excluding the probable false negatives before training, as our conclusion anticipated; at the same exclusion rate, a cosine cutoff reproduces it (0.404 against 0.395). In the revised PDF, we will state this in the abstract.

  2. Transfer of the region size (W4). Our conclusion named a corpus-transferable radius as the next step. We froze the picks of the label-free backfill rule before any training on a new corpus (CQADupStack). It selected α=3\alpha = 3, as on Quora, and Ballpark beat Hard in 5/5 seeds.

  3. The central finding holds. Hard negatives mined from the teacher contain probable false negatives, and excluding them before training repairs duplicate retrieval and utility.

  4. Calibration (W4, Q2). The 95% in Eq. 4 is a nominal target. Measured coverage of fresh views at α=1\alpha = 1 is 0.88 to 0.93; at the operating radius α=3\alpha = 3 it is above 0.999.

  5. Details (W1, W2, Q3, W5). In the revised PDF, we will add or correct the teacher settings, the student initialisation, Appendix A.3, the evaluation protocol, the judge details and the broken reference.

Setup. Unless stated otherwise, a number is the utility composite (STS-B Spearman, SciFact and NFCorpus nDCG@10) at 30k steps, BGE-base (110M) to the 11M student on Quora, with the submitted top-128 lists, eight negatives, recipe and five paired seeds. A paired difference is given as mean, 95% CI and the number of seeds with that sign.


W1. Teacher checkpoints and perturbation settings. In the revised PDF, we will add them to the appendix, with the precision and GPU time of each pass.

TeacherCheckpointPoolingDropout in the KK perturbed passes
BGE-base, 110MBAAI/bge-base-en-v1.5meanall dropout sites of the shipped config (hidden, attention), p=0.1p = 0.1
Qwen3-Embedding, 4BQwen/Qwen3-Embedding-4Blast tokenattention dropout only, p=0.1p = 0.1
27B decodermicrosoft/harrier-oss-v1-27blast tokenattention dropout only, p=0.1p = 0.1

Maximum input length is 128 tokens for all teachers. The anchor embedding is the deterministic pass with dropout off. We store the dropout seed of each perturbed pass per shard of 25,000 rows, so the regions can be rebuilt exactly.


W2. Initialisation and the two output maps.

  • Initialisation. The statement in Appendix C.1 holds for every cold-start student. B2W and C2W are the only exceptions; they start from the pretrained Qwen2.5-0.5B and Qwen2.5-3B checkpoints. In the revised PDF, the appendix will state this and give the warm-start recipe.

  • Two output maps. The proposition in Appendix A.3 comes from an earlier student design with a learned region map next to the sentence output map. No reported experiment trains that design, and Algorithm 1 does not use it: the regions come from the teacher and are used only to select negatives. The A.3 sentence about an implemented architecture with two maps described that earlier design. In the revised PDF, we will remove it together with the proposition.

AuthorsOpenReview ↗

Response to Reviewer fwKM (2/4)

W3, Q1. Isolating the contribution.

Answer. Excluding the near neighbourhood produces the gain. Ranking by region distance does not, and at a matched exclusion rate a cosine cutoff is slightly better than the region. The cutoff still needs a rate, and the backfill rule sets that rate without labels (W4).

We trained the three requested comparisons at the final layer and at layer 10, plus a label oracle and a per-anchor cosine cap, with the same teacher, candidate lists, eight negatives and paired seeds.

Method (Quora)Excluded slotsCompositevs. Ballpark
Hard (eight nearest by cosine)00.361-0.034, 0/5
Region-order: region distance ranks the list, no exclusion00.365-0.030, 0/5
Oracle: removes every labelled duplicate (uses labels)all labelled duplicates0.378-0.017, 0/5
Ballpark, α=3\alpha = 322.9%0.395-
Ballpark exclusions, survivors ranked by cosine22.9%0.395+0.001, 2/5
Per-anchor cosine cap with Ballpark's exclusion counts22.9%0.395+0.000, 3/5
Global cosine cutoff at Ballpark's exclusion rate (0.750)22.9%0.404+0.010 [+0.004, +0.015], 5/5
  • Ranking. Region-order shares 92% of Hard's negatives and adds +0.004 over Hard. Ranking Ballpark's survivors by cosine gives the same 0.395.

  • Exclusion. The global cutoff at the same exclusion rate is +0.010 above Ballpark. The label oracle (0.378) is below Ballpark because QQP duplicate labels are sparse: it still takes the nearest unlabelled candidates, and many of them are paraphrases.

Wikipedia (Hard 0.387). Region-order reaches 0.383 (-0.004, CI [-0.008, -0.000]). Ballpark at the radius that the backfill rule picks on a finer α\alpha ladder (α=2.5\alpha = 2.5; the coarser ladder of the submitted PDF gave α=2\alpha = 2) reaches 0.387, and the cosine cutoff matched to it 0.391 (+0.004, 5/5). The low overlap with Hard that you noted concerns the intermediate-layer regions. There, the layer-10 ordering explains the gain: layer-10 cosine order with no exclusion reaches 0.402 (+0.015 over Hard, 5/5), equal to the layer-10 region (0.402).

Layer 10. Exclusion and ordering crossed, Quora:

ExclusionOrderComposite
nonefinal-layer cosine (Hard)0.361
nonelayer-10 cosine0.399
nonelayer-10 region distance0.400
layer-10 regionlayer-10 cosine0.403
layer-10 regionlayer-10 region distance (submitted)0.408
layer-10 cosine cutoff, same ratelayer-10 cosine0.413

The layer-10 order draws negatives from a median final-layer rank of about 8 to 9 instead of 3.5, and this accounts for most of the gain. The exclusion adds about +0.008, and a matched layer-10 cosine cutoff reproduces it.

Consequence for the manuscript. On fixed candidate lists, the shape, per-anchor radius and ranking of the region add no measurable utility at a matched exclusion rate, and in the revised PDF we will state this. The region remains a label-free criterion for which candidates to exclude; W4 tests whether its radius transfers to a new corpus.

AuthorsOpenReview ↗

Response to Reviewer fwKM (3/4)

W4, Q2. Calibration of the regions.

Answer. The 95% is a nominal target, and we will state this concisely in the revised PDF. Coverage at α=1\alpha = 1 falls slightly below it for a reason we can state. The operating radius α=3\alpha = 3 covers essentially every fresh view.

Fresh-view coverage. On 4,000 Quora anchors we drew eight new dropout views per anchor, which the region did not see, and fitted the region from the first KK of 32 views:

Coverage at α=1\alpha = 1K=4K = 4K=8K = 8K=16K = 16K=32K = 32
p=0.05p = 0.050.8770.8870.9140.930
p=0.1p = 0.1 (paper)0.8880.8890.9120.929
p=0.2p = 0.20.8900.8900.9110.928

Coverage rises with KK and does not depend on pp. At α≥1.5\alpha \ge 1.5 it is at least 0.9998.

Relation of the leave-one-out scores to the KK-view metric. If the KK leave-one-out scores and the score of a fresh view were exchangeable, the fresh view would fall below the jj-th smallest score with probability j/(K+1)j/(K+1), the rank argument behind split conformal prediction (Vovk et al., 2005; Barber et al., 2021). The region meets this condition only approximately: each leave-one-out score uses the other K−1K-1 views, while a fresh view is scored against all KK. Thus K/(K+1)K/(K+1) is not a strict cap (at K=4K = 4, coverage is 0.877 to 0.890, above 4/54/5). The shortfall comes from the small number of scores: the 95th percentile of KK scores is a noisy estimate, and coverage rises as KK grows.

Dropout rate. The dropout rate sets the scale of the region. A wrong rate shows up in the backfill count without labels: 0.00 backfilled slots at p=0.05p = 0.05 and 7.37 at p=0.2p = 0.2, against 0.68 at p=0.1p = 0.1. At the radius that the rule then picks (α=4\alpha = 4 at p=0.05p = 0.05, α=2\alpha = 2 at p=0.2p = 0.2), both students return to the reference level (0.395 and 0.393, against 0.395).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.