DriftRetrieval: Learning from Many Negatives with Less Backpropagation

Large contrastive batches give retrieval models many documents to learn from, but training through every query and document is expensive. We introduce DriftRetrieval, which separates comparing embeddings from updating them. For each query, it constructs a temporary target that moves toward the matching document and away from high-scoring alternatives. The model learns to reach this target with a squared-error loss. Successive steps alternate between updating queries and their matching documents; the remaining embeddings help determine the targets but receive no gradient on that step. Processing small groups of examples at a time keeps GPU working memory bounded as the batch grows. DriftRetrieval reaches 41.66 retrieval on the full 56-task MTEB benchmark, versus 38.81 for a strong InfoNCE baseline that also computes fewer encoder gradients. Across three training seeds on a 14-task subset, the retrieval gain is $3.44\pm 0.78$ points (mean and sample standard deviation), while overall scores are similar. In controlled synthetic tests at 32K and 128K pairs, optimized DriftRetrieval achieves about 15% higher throughput than optimized sampled-gradient InfoNCE.