acceptodds
Under review as a conference paper at ICLR 2027

DriftRetrieval: Learning from Many Negatives with Less Backpropagation

Abstract

Large contrastive batches give retrieval models many documents to learn from, but training through every query and document is expensive. We introduce DriftRetrieval, which separates comparing embeddings from updating them. For each query, it constructs a temporary target that moves toward the matching document and away from high-scoring alternatives. The model learns to reach this target with a squared-error loss. Successive steps alternate between updating queries and their matching documents; the remaining embeddings help determine the targets but receive no gradient on that step. Processing small groups of examples at a time keeps GPU working memory bounded as the batch grows. DriftRetrieval reaches 41.66 retrieval on the full 56-task MTEB benchmark, versus 38.81 for a strong InfoNCE baseline that also computes fewer encoder gradients. Across three training seeds on a 14-task subset, the retrieval gain is points (mean and sample standard deviation), while overall scores are similar. In controlled synthetic tests at 32K and 128K pairs, optimized DriftRetrieval achieves about 15% higher throughput than optimized sampled-gradient InfoNCE.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Machine Learning, Retrieval, LLM

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_GyC6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a training method that uses candidate documents to construct temporary target embeddings, then backpropagates through only the query or its paired document on alternating steps. It shows retrieval gains on Qwen2.5-0.5B against InfoNCE.

Critical strengths
  1. The method is clearly specified.
  2. The ablation studies are extensive.
Critical weaknesses
  1. The evidence of generalization to other models is limited: The performance gains on Mistral and Gemma are marginal. Experiments with larger modern LLM models would further strengthen the evidence.
  2. The blockwise similarity computation and encoder replay build on established techniques and can also be applied to exact InfoNCE. It is valuable for an efficient implementation, but should not be presented as the core contribution.
Questions for authors

See weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer GyC6

We thank the reviewer for their careful review and for recognizing the clear specification and extensive ablations. We appreciate the suggestions about larger models and the distinction between the learning method and its efficient implementation.

We agree that encoder replay and blockwise similarity computation are established techniques that also apply to InfoNCE. Our full-gradient InfoNCE control uses streamed scoring and gradient-cache replay for this reason. Both sampled InfoNCE and Drift compare against the full candidate set. We study a particular allocation of gradients, detached-target updates alternating between queries and paired documents. Replay and tiling make this practical.

The relation to InfoNCE is also explicit in Section 3 and Appendix A, under matching conditions, the query-side update gives the query-side InfoNCE gradient up to a positive scalar. We have now added a control that replaces the proposed reverse rule with the exact-key update on the positive-pair matrix. Its full-suite retrieval scores are 41.48 and 40.62, compared with 41.49 and 40.85 for the proposed rule. Although individual tasks respond differently, these close averages do not establish that the proposed reverse rule uniquely explains the retrieval gain.

Due to compute constraints, we haven't yet completed the model size comparisons that the reviewer suggested but we plan to add those experiments in the camera ready version. The additional experiments, however, do strengthen the evidence for the smaller-model setting. With matched temperature, two fresh Qwen2.5-0.5B seeds improve full-suite retrieval by 2.98 and 1.54 points, remaining 2.43 and 0.79 ahead without ArguAna. The overall 56-task average improves by 0.83 and 0.62, and the two-seed retrieval difference remains positive at all three tested learning rates.

Reviewer_GyC6OpenReview ↗

Response to authors

Thanks for your reply. However, my concerns are still unresolved. I will keep my score.

Reviewer_oPfJOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

Fine-tuning the embedding models is a standard process in retrieval problems. The paper attempts to improve on the efficiency so that more performance can be extracted out with same memory budget. In particular, it proposes DriftRetrieval which trains dense retrievers without differentiating every encoding in the batch. For each query, a temporary target is built from a (weighted) average of candidate documents (the drift field). A mean squared error loss is used with the query and the stop-gradient target. Steps alternate between query updates and paired-document updates. A Gradient-Cache-style replay with online-softmax tiling bounds the GPU memory even as the batch size grows.

The primary results are reported on MEDI2 with the Qwen2.5-0.5B embedding model. DriftRetrieval achieves 41.66 MTEB retrieval vs. 38.81 (sampled-gradient InfoNCE) and 33.77 (full-gradient InfoNCE), with similar overall scores. Finally, the proposed method achives ~15% higher (synthetic) throughput as compared to an optimized sampled-InfoNCE baseline at 32K/131K pairs.

Critical strengths
  1. The DriftRetrieval demonstrates benefits on retrieval tasks where it achieves 41.66 MTEB retrieval vs. 38.81 (sampled-gradient InfoNCE) and 33.77 (full-gradient InfoNCE). This may be come in handy to practitioners and researchers in the retrieval community.
  2. Table 2 demonstrates that DriftRetrieval improves throughput by ~14.7% as compared to sampled InforNCE with large batch sizes or 32K and 131K.
  3. The writing including the core algorithm and experiment sections are easy to follow.
Critical weaknesses
  1. The gains do not generalize across various model families or sizes. Table 3 (D.1) demonstrates that there is a clear gain in overall (55.34 vs 52.85) and retrieval performance (40.35 vs 33.67) with Qwen2.5-0.5B as compared to InfoNCE no-grad. However, the gap is marginal with Mistral-7B (59.09 vs 58.83) and Gemma-3-1B-IT (52.88 vs 52.63) models.
  2. Table 2 demonstrates that the proposal improves on throughput, however, there has been no discussion on accuracy benefits with large batch sizes.
  3. Increasing the batch size is one factor, however, using more than one negative have been demonstrated to be useful in several retrieval applications [1, 2, 3]. It is worth investigating considering "Learning from Many Negatives" part in the title.
  4. The paper is thin on baselines: The literature review discusses Memory efficient contrastive learning (Gao et al.,2021; Cheng et al., 2025; Chen et al., 2023) and detached gradients (Wang & Liu, 2021; Yeh et al., 2022; Tao et al., 2022). However, only two variants of InfoNCE are considered for the experiments section.
  5. In Fig. 2a, classification drops 2.64 points and STS drops 0.61. The "overall scores are similar" may not convey the full picture. The paper should seriously discuss the possible reasons or focus the paper entirely on retrieval tasks.
  6. The primary results of 41.66 vs. 38.81 are with a single seed on the full suite. The seed replication is on a 14-task subset where retrieval score is the mean of four datasets (ArguAna, NFCorpus, SciFact, SCIDOCS). ArguAna is the largest contributor (Table 8), is there any property (symmetric nature) of the dataset the method is able to exploit?
  7. All main runs use a single learning rate (2e-5). T = 0.02 is used for InfoNCE and T = 0.05 for Drift. Was there any hyper-parameter tuning done?
  8. There is no time-to-quality comparison.

[1] https://dl.acm.org/doi/10.5555/3157096.3157304

[2] https://arxiv.org/pdf/2407.15831

[3] https://dl.acm.org/doi/10.1145/3539597.3570392

Questions for authors
  1. Are the gains due to the reverse field, the temperature, or alternation? A drift variant without the reverse update, and a query-only baseline, would separate these.
  2. Can you show a real-data experiment at larger batch sizes (e.g., 8K -- 32K) with downstream quality, to support the "many negatives" framing? You can additional consider experimenting with multiple hard negatives in future versions (optional for current review cycle).
  3. Are the gains achieved at a specific size or a specific family? It would be helpful to see the trend on different sizes from the Qwen family. Is it possible that gains are more substantial with small models?
  4. It will be good to analyze the losses for the STS and classification tasks.
  5. Why does sampled InfoNCE degrade with more steps (Table 5)? Does the same instability affect Drift?
3 comments
AuthorsOpenReview ↗

Response to Reviewer oPfJ

We thank the reviewer for their detailed review and concrete suggestions for strengthening the retrieval experiments.

Regarding replication and temperature, we trained two fresh Qwen2.5-0.5B seeds at T = 0.05 for both methods (G = 2) and evaluated all 56 MTEB tasks:

MethodSeed 17011Seed 17012
Sampled-gradient InfoNCE38.5139.31
DriftRetrieval41.4940.85
Difference+2.98+1.54

Drift wins on 12/15 and 10/15 retrieval datasets. Excluding ArguAna leaves gains of 2.43 and 0.79 points; the overall 56-task average improves by 0.83 and 0.62.

We also compared both methods at learning rates 1e-5, 2e-5, and 4e-5. The two-seed mean retrieval gains are +0.68, +2.26, and +2.76 points. Five of six pairs favor Drift, one seed at 1e-5 loses 0.22. We report the full grid without selecting each method's best test result. Temperature matters: Drift loses to matched InfoNCE at T = 0.02 in both seeds. These checks support the T = 0.05 result, without establishing a generally optimal setting.

For real-data large batches, we trained at batch sizes 2,048 and 8,192 with 10,485,760 query presentations each: 5,120 and 1,280 updates, respectively:

BatchSeedSampled InfoNCEDriftSampled InfoNCE minutesDrift minutes
2,0481701136.6741.64207.5172.7
2,0481701236.5741.70233.8211.0
8,1921701138.1541.15225.6215.2
8,1921701238.7342.36258.8238.5

For the reverse-update question, we ran the requested query-only control and additional update comparisons at T = 0.05:

Drift variantSeed 17011Seed 17012
Query only34.4334.44
Both sides each step41.2240.57
Alternating, exact-key reverse41.4840.62
Alternating, proposed reverse41.4940.85

Query-only training is substantially weaker here, while the other three variants have close aggregate retrieval scores despite differing task profiles. This supports training the paired-document side in this recipe, without establishing a unique advantage for the proposed reverse rule. The exact-key control uses the positive-pair reverse matrix, not the full hard-negative InfoNCE gradient. As derived in Section 3 and Appendix A, the query update is proportional to stopped-document InfoNCE. The learning-rate sweep does not resolve gradient-scale, Adam-epsilon, or clipping effects.

Regarding other baselines, the submitted full-gradient control uses streamed scoring/gradient-cache replay. We added directional and symmetric InfoNCE controls, the G = 1 symmetric control reaches 42.68 and 42.62, exceeding Drift in that study. We plan to add the missing comparisons to the other cited detached-gradient objectives in the camera ready version. In the main G = 2 comparison, classification falls by 0.73 and 0.66 points, STS changes by -0.54 and +0.49, and reranking falls slightly. We do not yet have a causal explanation for these category differences.

Due to compute constraints, we haven't yet completed the model size comparison experiments, but we plan to add that in the camera ready version as well.

Reviewer_oPfJOpenReview ↗

Thanks for your responses

Thanks for the responses. Please look more into stability of results ~48% dip in the benefits with a different seed is a red flag. Could it be related to convergence? Please consider reporting variance etc. Moreover, please consider reporting the best numbers for the proposed method against the best numbers of the baseline -- the full grid can be the support.

We thank the reviewer for their follow-up and for pointing out that the variation in the gain needs to be quantified. We agree that two positive runs alone do not establish stability. At the original setting (G = 2, T = 0.05, learning rate 2e-5, 1,253 updates), the results are:

QuantityRetrieval mean ± sample SD
Sampled-gradient InfoNCE38.91 ± 0.56
DriftRetrieval41.17 ± 0.46
Paired improvement+2.26 ± 1.02

These statistics use the two training seeds, 17011 and 17012. The individual improvements are +2.98 and +1.54 points, so the reviewer's approximately 48% reduction in the measured benefit is correct. We report uncertainty in the paired difference, as well as in each method's score. With n = 2, these SDs are descriptive, they cannot reliably characterize seed variability or establish statistical significance.

Following the reviewer's suggestion, we also compare each method's best observed two-seed mean within the same learning-rate grid, {1e-5, 2e-5, 4e-5}, keeping G = 2, T = 0.05, batch size 2,048, and 1,253 updates fixed. We select one learning rate per method by its two-seed mean and retain both seeds at that rate:

MethodSelected learning rateRetrieval mean ± sample SD
Sampled-gradient InfoNCE1e-539.35 ± 0.98
DriftRetrieval4e-542.07 ± 0.76

This gives a +2.72-point difference between the best observed means in this G = 2 grid. The full grid is provided below. It reuses the same two seeds, rather than providing six independent seed replications.

Regarding convergence, the separate five-checkpoint training curves show that the training horizon can substantially change the comparison. Applying the same selection principle, one common checkpoint per method, chosen by its two-seed mean gives:

BatchSampled InfoNCE best observed mean ± SD (update)Drift best observed mean ± SD (update)
2,04840.72 ± 0.24 (1,280)43.04 ± 0.78 (1,280)
8,19242.44 ± 0.01 (320)41.75 ± 0.86 (1,280)

In particular, sampled InfoNCE's earlier 8K checkpoint is better than Drift's best observed checkpoint, despite Drift's higher final-horizon score. These are maxima over five evaluated checkpoints, not validation-selected early-stopping results or proof of the best attainable score. The trajectory runs also use a separate fixed-memory execution and data-stream configuration, they are not intermediate checkpoints from the questioned 1,253-update runs. They demonstrate training-horizon sensitivity but do not identify convergence as the cause of the original seed discrepancy. The available runs favor Drift, but additional paired seeds and retrieval evaluations throughout the exact main recipe are needed to determine why it varies.

The supporting learning-rate results are below. All entries are retrieval mean ± sample SD across the same two seeds, the final column uses paired differences.

Learning rateSampled InfoNCEDriftPaired improvement
1e-539.35 ± 0.9840.02 ± 0.28+0.68 ± 1.27
2e-538.91 ± 0.5641.17 ± 0.46+2.26 ± 1.02
4e-539.32 ± 0.0642.07 ± 0.76+2.76 ± 0.71
Reviewer_2jpDOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Thee authors propose drift retrieval, a method to reduce the cost of contrastive learning by only backpropagating through a fraction of query/document embeddings instead of the all query-document pairs each optimization step. The key steps/intuitions here are that: (i) a drift field is constructed such that it points towards positive embeddings (and away from negative ones) regardless of whether the positive embedding dominates the score, (ii) a synthetic target is defined as the query embedding moved in the direction of the drift field, and (iii) because the synthetic target is not backpropagated through, the overall method requires less gradient computations and is cheaper than the full InfoNCE matrix. This process is performed for documents every other step (i.e. alternating updates of query embeddings and documents embeddings. The authors evaluate their method against full and sampled InfoNCE. Their approach cannot beat sampled InfoNCE speed-wise and classification-wise, but is better on retrieval and other tasks.

Critical strengths
  1. The overall method seems sound and represents a valid way to carry out performance-cost tradeoff, by trading full gradient computation and (presumably) stronger downstream performance for faster wall clock time. It can be useful for applications where full InfoNCE can be prohibitively expensive and such tradeoffs become desirable. To demonstrate this point, the experiments show a considerably higher throughput at scale.

  2. A minor strength is that the paper is generally well-written and easy to follow, with the key intuitions being clearly presented, especially in the method sections (S2-S4).

Critical weaknesses
  1. The key issue here is the relative weakness of the experimental findings, especially Table 1 and Figure 2, which shows that sampled InfoNCE is not only significantly faster (67% of DriftRetrieval) but also outperforms on certain tasks. Although DriftRetrieval ends up having better throughput at scale, it does somewhat muddy the narrative that drifting retrieval is necessarily the right approach to doing a performance-cost tradeoff.
Questions for authors
  1. I am somewhat confused by why both sampled InfoNCE and drifting retrieval ends up having a stronger retrieval score than the full InfoNCE computation, which seems to be counterintuitive unless I am misunderstanding something here.

  2. Is there a reason why you only chose to report 4 MTEB tasks in Figure 3 (scaling up the data/document count)? Is there a analogous table for the full MTEB, i.e. comparable to Table 1?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 2jpD

We thank the reviewer for their thoughtful review and for recognizing the method's clear formulation and potential as a performance-cost tradeoff. We agree with the central point that the measured application result and the large-batch throughput result need to be discussed separately.

At the submitted MEDI2 setting, sampled InfoNCE takes 30.8 minutes and Drift takes 47.5 minutes. Drift therefore offers higher retrieval at additional training time there. The roughly 15% throughput advantage comes from the separate large-batch synthetic experiments. We have now added real-data 2K/8K training runs, each processing 10,485,760 query presentations. Their final retrieval scores and total training times are:

BatchSeedSampled InfoNCEDriftSampled InfoNCE minutesDrift minutes
2,0481701136.6741.64207.5172.7
2,0481701236.5741.70233.8211.0
8,1921701138.1541.15225.6215.2
8,1921701238.7342.36258.8238.5

Drift takes 4.6–16.8% less total training time and has higher retrieval at the common final horizon. However, sampled InfoNCE reaches strong scores earlier, so these runs do not show a consistent time-to-quality advantage.

Regarding Figure 3, we agree that four tasks were insufficient to show how the candidate-coverage result extends to the full benchmark. We have now evaluated the retained models on all 56 MTEB tasks, including all 12 CQA subsets. Below, InfoNCE denotes the sampled-gradient baseline, “overall” is the mean over the 56 tasks, and “retrieval” is the 15-dataset retrieval category.

Candidate poolSeedInfoNCE overallDrift overallInfoNCE retrievalDrift retrieval
2564255.4755.3841.9440.60
2,0484255.2555.6741.8941.69
2564355.5655.5942.1840.24
2,0484355.1355.8342.1541.57

The full suite preserves a narrower positive finding, increasing the candidate pool raises Drift's retrieval by 1.10 and 1.33 points in the two seeds. However, temperature-matched sampled InfoNCE has higher absolute retrieval in all four pool/seed comparisons.

The fresh G = 2 (num_hard_neg = 1) experiment gives a different result, drift improves the complete retrieval category by 2.98 and 1.54 points, remaining 2.43 and 0.79 points ahead without ArguAna, while classification still declines. We also tested a stronger symmetric InfoNCE baseline at G = 1 (num_hard_neg = 0). It has higher retrieval than Drift in that study, whereas Drift has higher overall scores and all six nonretrieval category averages. These comparisons reinforce the reviewer's point that the practical choice depends on which task and cost matter.

The reviewer's question about why sampled methods can outperform full InfoNCE highlights the distinction between optimizing a training objective and obtaining the best downstream embedding. Full InfoNCE gives the exact gradient of its objective, but changing which encodings receive gradients changes the training trajectory under a finite optimizer and schedule. Regularization, false negatives, and conflicting gradients are possible explanations for the ordering, our experiments do not distinguish them. The new learning curves also show why the training horizon matters, at 8K, sampled InfoNCE reaches higher best-observed retrieval than Drift in both seeds, but falls further by the final evaluation. We therefore describe the decline as an observed training effect, rather than established numerical instability.

One small clarification to the summary, the positive is included in the softmax center. As its probability tends to one, the drift field tends to zero. The method therefore does not retain a nonzero attraction regardless of positive dominance, its query-side relation to InfoNCE is derived in Section 3 and Appendix A.

Reviewer_2jpDOpenReview ↗

Response to authors

I thank the authors for their reply. I guess my only critique here is that the authors are suggesting that learning dynamics/inductive biases plays a significant role here, hence why sampling and the proposed drift approach can outperform even full InfoNCE computation. But this also decreases my overall confidence in the empirical evaluation (beyond the obvious computational speedup), because it is not clear how sensitive the scores are to variance. In any case, I retain my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.