acceptodds
Under review as a conference paper at ICLR 2027

xPRT: Cross-Passage Retrieval Pretraining

Abstract

Dense retrievers are a foundational component underlying modern RAG and search systems. Training these models typically requires a curated collection of annotated query-document pairs, which are prohibitively expensive to obtain outside a handful of well-resourced domains. This has motivated a line of work that leverages large-scale unsupervised retrieval pretraining from unlabelled text alone. Most such methods learn by optimizing a language modelling or contrastive learning objective over *short spans within a document*. This raises the question: *how can we extract richer learning signals from a single document for retrieval pretraining?* To address it, we introduce **xPRT** (Cross-Passage Retrieval Pretraining), an unsupervised framework that learns from the key information distributed across an entire document and the connections between its parts. We evaluate xPRT against five state-of-the-art objectives on MTEB English v2, continuing pretraining from a public unsupervised checkpoint while holding the corpus, schedule, and fine-tuning procedure fixed. Across diverse tasks spanning all task types, we observe that xPRT consistently outperforms the strongest baseline by **3.2%** on an average. Notably, in this continual pretraining setting, many baseline methods incur substantial performance regressions, whereas our method delivers the largest complementary gains over the initial pretrained checkpoint. Controlled ablations confirm that xPRT's performance gains stem from its proposed cross-passage signals. xPRT demonstrates that a single unlabelled document is a much richer source of retrieval signal than prior work assumed, and offers **complementary gains** to what current retrievers already capture.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Information Retrieval, Unsupervised Learning, Self-Supervised Learning, Masked Language Modelling, Retrieval, Pretraining

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_sHaYOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes xPRT, a retrieval-oriented pretraining method that selects related passages from the same document and uses cross-passage contrastive alignment and prediction to train a dense encoder. It evaluates continued pretraining from E5 and pretraining from BERT on MTEB English v2 benchmark. xPRT improves the E5 and BERT starting point and substantially outperforms the compared pretraining objectives.

Critical strengths
  1. The method design is thoughtful. Selecting semantically related passages within a document gives the model a more targeted training signal than pairing arbitrary passages. The combination of cross-passage alignment and prediction is well motivated, and the ablations in Table 3 help identify which components matter.
  2. The paper explains the motivation, training objectives, and experimental settings clearly. The figures and tables make the method and its results easy to follow.
Critical weaknesses
  1. The BERT comparison stops before the downstream training stage that would test the value of a pretraining method. A BERT-initialized model trained only with a retrieval pretraining objective is usually an initialization for later contrastive retrieval training, rather than the final retriever. Table 2 evaluates the resulting embeddings directly, without applying the same subsequent contrastive training to xPRT and the baselines. It therefore cannot show whether xPRT is a better starting point for a trained dense retriever.

  2. Direct evaluation may favor xPRT over bottleneck-based baselines for reasons unrelated to initialization quality. RetroMAE and related methods aim to strengthen the representation bottleneck before later retrieval training, whereas xPRT already includes a cross-passage contrastive loss that directly trains embedding similarity. Its advantage in the direct evaluation in Table 2 is thus plausible by design, but the paper does not show whether that advantage survives a common contrastive training stage.

Based on the above two points, starting all BERT-based models from their respective pretrained checkpoints and training them with the same contrastive objective and data would make the comparison more informative.

  1. More pretraining data does not yield a clear improvement. In Table 1, the average performance of xPRT did not improve as the training data increased; in fact, retrieval performance even declined. The paper suggests that the one-pass schedule and low learning rate may explain the plateau, but the current results do not establish that the method benefits from additional data. This prompted me to wonder what the effect might be of conducting an additional round of contrastive learning training.
Questions for authors

See above weakness

2 comments
AuthorsOpenReview ↗

Rebuttal Response to Reviewer sHaY [1/2]

We thank the reviewer for finding the method well motivated and the paper clear. The central question is whether xPRT's advantage survives the contrastive training that normally follows pre-training, and that is the right test. We ran it for every BERT-initialized checkpoint in Table 2 and for the contrastive controls requested by the other reviewers.

W1 and W2. Identical contrastive fine-tuning of every initialization. Every checkpoint is fine-tuned on MSMARCO with the same objective, data and schedule.

Initialization (then fine-tuned)MTEB AvgRetrieval nDCG@10xPRT's MTEB Avg leadRetrieval tasks xPRT wins
xPRT51.1538.84——
RetroMAE50.2737.37+0.88/10
BoW-DPR50.1436.86+1.0010/10
SimLM50.0236.98+1.127/10
DupMAE49.8736.68+1.289/10
CoT-MAE49.7136.57+1.449/10
InfoNCE, curated pairs49.6435.59+1.5110/10
InfoNCE, adjacent passage49.1834.56+1.9710/10
ReContriever49.1634.56+1.9810/10
Raw BERT (no pre-training)49.1235.36+2.039/10
ReContriever, uniform weights49.0634.45+2.0910/10
InfoNCE, random crop48.9934.45+2.1610/10
InfoNCE, random same-doc passage48.7234.51+2.4310/10
Revela47.6833.21+3.4710/10

Key Takeaways

  1. xPRT is the best starting point. It leads all 13 other initializations by +0.87 to +3.47 MTEB Avg and by +1.47 to +5.63 Retrieval nDCG@10.
  2. W2: the advantage does not come from having a contrastive loss. If a contrastive loss alone produced a better starting point, the contrastive-only initializations would lead. Instead, they end within about half a point of raw BERT (48.72–49.64 vs. 49.12). The same holds without fine-tuning: from BERT, InfoNCE with established positives scores 40.7–43.6 MTEB Avg, at the level of the bottleneck baselines (see our response to EBNi). The direct-evaluation advantage in Table 2 (in our manuscript) therefore does not come from a contrastive loss alone.
  3. The reviewer's intuition about bottleneck methods holds, and xPRT goes further. As designed, the bottleneck methods are better starting points than raw BERT (+0.59 to +1.15). xPRT improves on raw BERT by +2.03.

xPRT on top of existing pre-training baselines

The reviewer notes that bottleneck methods are meant to prepare the encoder for later contrastive training, so to make a fair comparison we run xPRT as a next stage in pretraining over botteneck as well as contrastive pre-trained checkpoints. This baseline in the BERT setting and with 100k data scale.

Starting checkpointBeforeAfter + xPRTGain (MTEB Avg / Retrieval nDCG@10)
RetroMAE41.96 / 19.6546.52 / 24.78+4.56 / +5.13
DupMAE40.80 / 18.8946.56 / 24.92+5.76 / +6.02
SimLM42.73 / 20.6846.12 / 23.59+3.39 / +2.90
BoW-DPR42.57 / 19.0446.68 / 24.91+4.12 / +5.88
Revela42.24 / 17.4546.26 / 25.99+4.02 / +8.54
CoT-MAE40.94 / 16.1945.49 / 23.30+4.55 / +7.11
InfoNCE, random same-doc passage41.08 / 18.5445.03 / 22.59+3.95 / +4.05
InfoNCE, adjacent40.67 / 18.8545.73 / 23.72+5.06 / +4.87

The key point to note here is XPRT as a continual pretraining objective offers significant over and above every above baseline, be it bottleneck or contrastive. This directly addresses the reviewer's concern even InfoNCE baselines move beyond their baseline number and this proves the gains are due to XPRT's full objective and not just because of it being a contrastive style loss.

AuthorsOpenReview ↗

Rebuttal Response to Reviewer sHaY [2/2]

W3. More pretraining data does not yield a clear improvement. In Table 1, the average performance of xPRT did not improve as the training data increased; in fact, retrieval performance even declined. The paper suggests that the one-pass schedule and low learning rate may explain the plateau, but the current results do not establish that the method benefits from additional data. This prompted me to wonder what the effect might be of conducting an additional round of contrastive learning training.

We agree that our results do not show gains beyond 100K documents under this protocol. What the results do show:

  • The gain is reached early and stays stable. xPRT improves E5 by +1.80, +1.87 and +1.83 at 100K, 500K and 1M documents. The flat curve is shared by every objective under this protocol: BoW-DPR changes by +0.08 / +0.38 / −0.04 and SimLM by −2.05 / +0.26 / −1.70. It is not specific to xPRT, and xPRT is the only method that is both stable and positive.
  • Most task types keep improving. Clustering and STS continue to rise (+0.40 and +0.28 from 100K to 1M). The small drop in retrieval is concentrated in TREC-COVID, Touché and FiQA (Table 9 in the manuscript).
  • The advantage holds after an additional round of contrastive training, which is what the reviewer asked about. Please check our previous comment that answers this directly.

We thank the reviewer for proposing this experiment. We believe the fine-tuning results answer their concerns directly, and we would be grateful if the reviewer would reconsider the score.

Reviewer_EBNiOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes xPRT, an unsupervised retrieval pretraining method that uses relations between passages within the same document as its training signal. Using a frozen E5 encoder, it selects anchor passages close to their document’s centroid and pairs each anchor with its four most related same-document siblings under fused E5 and TF-IDF similarity. The main objective is an InfoNCE loss between each anchor and its top sibling. Auxiliary terms include a one-layer decoder that predicts the siblings’ masked tokens from the anchor’s embedding (L_cross), plus self-distillation, MLM, and uniformity regularizers. The authors continue pretraining E5-base-unsupervised on FineWeb, holding data, schedule, and a retention regularizer fixed across methods, and evaluate on 41 MTEB(eng, v2) tasks without fine-tuning. Under this protocol, xPRT improves E5, while five baselines (RetroMAE, DupMAE, SimLM, BoW-DPR, Revela) change it less. When pretraining from BERT, xPRT outperforms the strongest baseline. Ablations attribute the gains mainly to using curated siblings rather than random same-document passages as positives.

Critical strengths

The experimental comparison is carefully controlled. All methods share the same initialization, corpus, schedule, retention regularizer and evaluation protocol, so the objective is the only variable. Under this setup, xPRT consistently improves E5 on MTEB(eng, v2) across all three corpus scales and all task types. The finding that established objectives fail to improve a strong contrastive checkpoint is also informative in its own right.

The ablations and analysis are also informative. Replacing curated siblings with random same-document passages will trigger degradation, while giving within-passage objectives more text or the training steps does not close the gap, which rules out a pure data-volume explanation. The ArguAna analysis, stratified by lexical overlap, shows that xPRT improves even on low-overlap queries where the lexical baseline degrades, suggesting the learned signal goes beyond term matching.

Critical weaknesses
  1. Baselines confound the claimed contribution with the choice of objective.

The paper attributes its gains to replacing random or positional positives with curated same-document passages. Yet all five baselines are non-contrastive, while xPRT is the only method with an InfoNCE loss. Two factors favor xPRT regardless of how it selects positives: the E5 starting checkpoint is itself contrastively trained, and evaluation without fine-tuning likely favors contrastive objectives over auto-encoding ones. The paper’s own ablations in the BERT regime agree: removing L_xp causes a large drop in retrieval performance, while removing L_cross has only a marginal effect. The “random same-document passage” ablation partially addresses this concern, but it is run only in the BERT regime and retains xPRT’s auxiliary terms. The missing controls are contrastive baselines that differ from xPRT only in how positives are constructed, run under the same continual protocol: random crops [1] and recurring spans [2]. CoT-MAE [3], which reconstructs a passage from an adjacent span, would additionally isolate the value of L_cross. Without these controls, the gains cannot be attributed to cross-passage positive selection.

  1. Novelty is overstated, and the closest prior work is neither cited nor compared.

The claim that existing objectives leave intra-document passage relations unused is overstated. ReContriever [5] uses the model under training to estimate the relevance of same-document positive pairs and reweights the contrastive loss accordingly. Beyond these works including [4], xPRT’s distinct components are the anchor filter, the score fusion, and L_cross. Yet the ablations show that L_cross contributes little and that TF-IDF-only sibling ranking performs on par with the fused ranking. The paper should position itself against these works and compare to at least one of them under the same protocol.

[1] Unsupervised Dense Information Retrieval with Contrastive Learning. TMLR 2022.

[2] Learning to Retrieve Passages without Supervision. NAACL 2022.

[3] ConTextual Masked Auto-Encoder for Dense Passage Retrieval. AAAI 2023.

[4] A Mutually Reinforced Framework for Pretrained Sentence Embeddings. arXiv:2202.13802, 2022.

[5] Unsupervised Dense Retrieval with Relevance-Aware Contrastive Pre-Training. Findings of ACL 2023.

Questions for authors
  • Was any learning-rate search performed for the baselines? From my understanding, all methods share a peak learning rate of 1e-6 in the continual setting, whereas several baselines were originally tuned for much larger rates (around 1e-4). For example, can the authors show that Revela’s LoRA-adapted Qwen2.5-0.5B is effectively trained at this rate, for example by reporting its LM loss curve?

  • How do the baselines compare to the E5 starting point when trained under their native settings (learning rate, schedule, and training budget)? This would clarify whether their failure to improve E5 reflects a limitation of the objectives themselves or the shared hyperparameters imposed by the protocol.

7 comments
AuthorsOpenReview ↗

Rebuttal Response to Reviewer EBNi [1/3]

We thank the reviewer for their insightful comments. We ran every control that was requested:

  • contrastive arms that differ from xPRT only in the positive (random crop, random passage, adjacent passage, recurring span), in both regimes;
  • CoT-MAE;
  • ReContriever;
  • Revela at both learning rates, with LM loss curves;
  • baselines under their native settings.

The results confirm the central claim from scratch and after fine-tuning. They also correct one claim in the E5 regime, which we revise.

W1. Baselines confound the claimed contribution with the choice of objective.

(a) Pretraining from BERT

Every baseline below uses the InfoNCE loss of Eq. 2 alone, and only the positive selection strategy changes.

BaselinePre-trainingFine-tuning
MTEB AvgRetrieval nDCG@10MTEB AvgRetrieval nDCG@10
InfoNCE, random same-doc passage (Contriever)41.0018.1750.6738.59
InfoNCE, adjacent passage (CoT-MAE pairing)40.6718.8551.3138.90
InfoNCE, recurring span (Spider)43.6121.9551.3040.04
InfoNCE, curated sibling (xPRT's positive)44.5024.8151.6140.19
SimLM (best non-contrastive baseline)42.7320.6852.2841.34
xPRT47.1229.2152.6842.25
  1. The contrastive loss is not the only factor that puts xPRT ahead. Despite using established positive selection methodologies, InfoNCE baselines score at the same level as the non-contrastive baselines (40.67–43.61 vs. 42.73 for SimLM).On the other hand, xPRT is +3.5 MTEB Avg / +7.3 Retrieval nDCG@10 above the best of all the above baseline. Furthermore, holding the InfoNCE loss fixed and just swapping in xPRT's curated sibling selection lifts InfoNCE to 44.50 / 24.81, above every established positive selection method (+0.89 / +2.86 over recurring spans). With an identical loss, we observe the following trends on retrieval performance positive: random passage < adjacent passage < recurring span < curated sibling, which would be a key takeaway from our work.
  2. Changing only the positive inside xPRT's objective costs 4.45 MTEB Avg/ 8.59 Retrieval nDCG@10 (Table 3, with L_cross off in both variants in the manuscript). This is the isolation the reviewer asked for, with the auxiliary terms held fixed rather than removed.
  3. The result survives fine-tuning, which addresses the concern that zero-shot evaluation favors contrastive objectives. After identical contrastive fine-tuning, xPRT beats recurring-span pre-training by +1.38 MTEB Avg.
  4. xPRT adds more signal to contrastive pre-training rather than duplicating it. We continued a BERT checkpoint pre-trained with InfoNCE on adjacent pairs using xPRT. It improves from 40.67 / 18.85 to 45.73 / 23.72, and the gain remains after fine-tuning.
AuthorsOpenReview ↗

Rebuttal Response to Reviewer EBNi [2/3]

(b) Continual from E5, under the protocol the reviewer requested.

Positive (InfoNCE only)MTEB AvgRetrieval nDCG@10
E5 (initial)53.0643.26
Contriever (Random crop)54.9743.88
Contriever (Random same-doc passage)54.6944.49
Contriever (Adjacent passage)55.0045.10
Recurring span (Spider)54.6145.43
Curated sibling (xPRT selection)54.9545.64
For reference: CoT-MAE (official)53.0142.74
For reference: full xPRT54.8645.78

Continuing E5 with any contrastive objective raises the MTEB average by 1.5–1.9 points. Our comparison was against five non-contrastive baselines and it was cruicial to include contrastive baselines to better contextualize the merits of the method, and we thank the reviewer for their suggestion.

The positive determines retrieval. With an identical loss, the curated sibling is the best positive: +1.76 Retrieval nDCG@10 over Contriever crops, +1.15 over random passages, +0.54 over adjacent passages and +0.21 over recurring spans. This is the same ordering as from BERT, where it spans 11 Retrieval nDCG@10 points.

The gap is smaller in the E5 regime for an identifiable reason. E5 was already contrastively trained on about 270M consistency-filtered web pairs. In addition, our protocol is designed to keep the model close to E5.

L_cross and CoT-MAE. We agree that L_cross has a small average effect: removing it lowers BERT’s scores by 0.45 / 0.75 and makes no measurable difference for E5. But from BERT, the gain is broad and consistent. L_cross is ahead on 34 of 41 tasks. Gains are from the Table 3 (from our manuscript) BERT checkpoints:

TypeTasks (gain from L_cross)
L_cross wins all 9 comparisons
RetrievalTREC-COVID (+1.89), HotpotQA (+1.39), FiQA (+0.77), ArguAna (+0.76), ClimateFEVER (+0.76), CQADupstack Unix (+0.75), CQADupstack Gaming (+0.68), SCIDOCS (+0.49)
ClusteringTwentyNewsgroups (+1.87), StackExchange (+0.46)
ClassificationMTOPDomain (+1.00)
SummarizationSummEval (+0.98)
STSSTS17 (+0.93), STS15 (+0.88)
Pair classificationSprintDuplicateQuestions (+0.60)
L_cross loses on average (comparisons lost)
ClusteringArXiv Hierarchical S2S (−0.65; 8/9), ArXiv Hierarchical P2P (−0.55; 8/9), Medrxiv P2P (−0.25; 8/9), Medrxiv S2S (−0.02; 3/9)
ClassificationIMDB (−0.26; 7/9)
RetrievalFEVER (−0.14; 5/9)
RerankingMindSmall (−0.11; 8/9)

L_cross makes the embedding keep what a text is specifically about. That helps on 34 of 41 tasks, and it costs a little mainly where the label is a broad property such as field or sentiment.

The queries it fixes show what it does: without L_cross, the model gets the topic right but the specifics wrong. Here are three examples, each holding on all three seeds:

  • “How do I unlock Disarm and Destroy?” (CQADupstack Gaming). Without L_cross, every seed ranks “How do I disarm enemies in Assassins Creed Revelations?” first. With it, the actual Disarm and Destroy thread is first on every seed.
  • “Are both Dictyosperma, and Huernia described as a genus?” (HotpotQA). Without L_cross, every seed ranks Dictyospermum first. That’s a different genus with an almost identical name, and the Dictyosperma page drops out of the top 10. With it, that page is back at rank 3–8.
  • A question about which “close friend” of a former player the Timberwolves signed (HotpotQA). Without L_cross, pages for other Timberwolves seasons come first. The page with the answer (2000–01, which names Chauncey Billups as Kevin Garnett’s close friend) drops out of the top 10. With it, that page is at rank 2–4.

The other content losses matter more when starting from BERT: removing L_self costs 2.72 MTEB Avg and removing L_MLM costs 3.94. None of them matter when continuing from a strong contrastive model like E5. CoT-MAE, which rebuilds a passage with help from its adjacent span, can’t replace them. It lowers E5’s scores (to 53.01 / 42.74) and reaches only 40.94 / 16.19 from BERT.

AuthorsOpenReview ↗

Rebuttal Response to Reviewer EBNi [3/3]

W2. Novelty is overstated, and the closest prior work is neither cited nor compared..

We agree that the sentence "existing objectives leave intra-document relations unused" is too strong. The revision says instead: existing retrieval pre-training objectives either fix the positive by position or by chance, or reweight such pairs online using the model being trained; xPRT selects the positive offline, based on content.

ReContriever (official implementation, same protocol):

MTEB AvgRetrieval nDCG@10
ReContriever (relevance-weighted crops)51.4033.39
Same pipeline, uniform weights51.4433.43
xPRT54.8645.78

Relevance weighting has no effect relative to ReContriever's own uniform-weight control. Both suggested baselines underperform and regress below E5. On the remaining components: we agree that neither the score fusion nor L_cross is the core contribution. The core is the selection principle. It is robust to the choice of scorer. Please refer to our response to Reviewer wF4n where we compare scorers.

Q1. Was any learning-rate search performed for the baselines? From my understanding, all methods share a peak learning rate of 1e-6 in the continual setting, whereas several baselines were originally tuned for much larger rates (around 1e-4). For example, can the authors show that Revela’s LoRA-adapted Qwen2.5-0.5B is effectively trained at this rate, for example by reporting its LM loss curve?

In the submission, all methods shared xPRT's peak learning rate of 1e-6, and the baselines were not tuned separately. We selected 1e-6 using contrastive perplexity on the MS MARCO dev set, which is disjoint from MTEB Avg.

The reviewer should please note that the original Revela paper was trained from scratch while ours is a continual pretraining setup. Thus we have a different hyperparameter setup which is optimal and tuned for our setting and across all baselines it is favorable than using their native hyperparameters. In subsequent points, we show this with Revela as well as BowDPR as examples.

Revela LRMTEB AvgRetrieval nDCG@10Final LM loss, with context / without context
1e-6 (shared)52.9642.955.67 / 3.59 (context hurts)
1e-4 (native)52.8440.733.20 / 3.99 (context helps)

At 1e-4, the LM trains properly and learns to use the retrieved context, but the retriever degrades further (−2.2 Retrieval nDCG@10). Training Revela effectively moves the retriever away from E5, so the shared rate was, if anything, favorable to Revela.

Q2.How do the baselines compare to the E5 starting point when trained under their native settings (learning rate, schedule, and training budget)? This would clarify whether their failure to improve E5 reflects a limitation of the objectives themselves or the shared hyperparameters imposed by the protocol.

ModelSettingMTEB AvgRetrieval nDCG@10
E5 (initial)—53.0643.26
BoW-DPRshared protocol53.1443.12
BoW-DPRnative (LR 3e-4, 140K steps, no retention)41.4518.04
Revelashared protocol52.9642.95
Revelanative (LR 1e-4, 10K steps, no retention)47.7728.61
xPRTshared protocol54.8645.78

Under their native settings, both baselines fall far below E5: BoW-DPR by 25.2 Retrieval nDCG@10 and Revela by 14.7. The shared protocol is therefore the setting most favorable to the baselines. Their failure to improve E5 reflects their objectives, not our hyperparameters.

We hope that the reviewer that the reviewer now understands the rationality behind our design choice and we would be grateful if the reviewer would reconsider the score in light of these clarifications and the additional baselines presented.

Reviewer_wF4nOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper builds on work in retrieval-based pretraining, proposing xPRT as a new way to isolate richer information from a document. For a pairwise anchor-sibling loss, this work proposes filtering for siblings that are relevant with another model, rather than picking random siblings. In doing so, the approach exceeds baselines, with ablations directly showing the improvement coming from the filtering.

Critical strengths
  1. The approach is quite simple and easy to understand
  2. It clearly works: the evaluation is pretty good! Fixing the training setup with E5 clearly shows improvement over all settings and over all baselines, with key ablations directly showing how good the contributions (the pairing strategy) really are
Critical weaknesses
  1. Novelty: the loss components are selected from other works (CoT-MAE and Contriever), and the big contribution here seems to be just claiming that filtering is good (which, to be fair, is well-supported)
  2. The selection rule for siblings seems to not offer much over plain TF-IDF? It is better, but not by a significant amount...
Questions for authors
  1. For the ablations, I'm curious to see how xPRT compares with Contriever using randomly selected siblings and CoT-MAE (selecting adjacent passages).
  2. I'm also curious how this paper's findings relate to https://arxiv.org/pdf/2212.09598? This paper also seems to find that intentional selection of siblings is important, although their methodology seems to be different (using queries as context rather than improving sibling selection).
3 comments
AuthorsOpenReview ↗

Rebuttal Response to Reviewer wF4n [1/3]

Thank you for noting the method's simplicity and appreciating the evaluation we presented. We ran all the comparisons you asked for and present our conclusions below.

Q1. ''For the ablations, I'm curious to see how xPRT compares with Contriever using randomly selected siblings and CoT-MAE (selecting adjacent passages).''

All experiments use the same initialization, corpus, number of steps, batch size and schedule, plus the retention regularizer in the E5 regime. The "Contriever-style" experiments use xPRT's own InfoNCE loss (Eq. 2) with a random positive from the same document (Row 2) and uses CoT-MAE's pairing (Row 3). "CoT-MAE" uses its official implementation. Scores are MTEB average (41 tasks) / Retrieval average (10 tasks).

MethodContinual from E5From BERTFrom BERT, then contrastive FT
Initial model53.06 / 43.2637.92 / 9.4349.12 / 35.36
Contriever-style (random crop)54.97 / 43.8840.87 / 15.4248.99 / 34.45
Contriever-style (adjacent passage)55.00 / 45.1040.67 / 18.8549.18 / 34.56
CoT-MAE (official)53.01 / 42.7440.94 / 16.1949.71 / 36.57
xPRT54.86 / 45.7847.12 / 29.2151.15 / 38.84
  • From BERT, xPRT leads by a wide margin: +6.1 to +6.5 MTEB Avg and +10.4 to +13.0 Retrieval nDCG@10 over every alternative. After identical fine-tuning it still leads each one by +1.4 to +2.4 MTEB Avg, in all three seeds.
  • From E5, xPRT has the best retrieval (+0.68 to +3.04 Retrieval nDCG@10). All contrastive controls raise the MTEB Avg average by a similar amount (54.69–55.00). Continuing E5 with any contrastive loss raises the average, and the choice of positive determines retrieval. We will make this explicit in the revision.

Q2. I'm also curious how this paper's findings relate to https://arxiv.org/pdf/2212.09598? This paper also seems to find that intentional selection of siblings is important, although their methodology seems to be different (using queries as context rather than improving sibling selection).

Thank you for the pointer. QaC [1] supports our premise: its human annotation finds that only 35.5% of the same-document passage pairs used by coCondenser are highly correlated. The two methods fix this in different ways:

  • QaC pairs each passage with a query from a doc2query T5 model, which was itself trained on labelled retrieval data (MS MARCO, NQ, Quora, among others). Every passage has to go through the generator, so the approach is costly to scale, and the pairs are only as good as the generator. [2] find that such generated queries often overlap heavily, and that their gains plateau as more are generated.
  • xPRT selects anchor-sibling pairs from the same document such that the sibling passage compliments or restates the anchor’s facts in different ways (see the W1 example). With reciprocal rank fusion between TF-IDF and E5 scores, our method needs no labels and the supply of pairs grows with the corpus.

Both works show that the sibling has to be chosen. xPRT chooses it from real text, cheaply and without labels, which we see as the more scalable route. We will add this comparison to the related work.

[1] https://arxiv.org/pdf/2212.09598

[2] https://aclanthology.org/2025.emnlp-main.888/

AuthorsOpenReview ↗

Rebuttal Response to Reviewer wF4n [2/3]

W1: Novelty: the loss components are selected from other works (CoT-MAE and Contriever), and the big contribution here seems to be just claiming that filtering is good (which, to be fair, is well-supported)

Thank you, this is a fair point. We agree that pair selection (the filtering step) is the largest factor in our gains, and that the paper should lead with it. While we already open our paper's method with cross-passage selection, we will highlight this aspect more as the key enabling factor of our method. But the finding is more than “filtering is good”: what matters is which pair is selected, and simpler ways of choosing or weighting pairs fall short. The loss terms do build on prior work, as the reviewer notes, but we designed the objective deliberately, and it adds to what selection gives.


Anchor: “…was CFO from 1998–2005 and president of TransAlta Power LP from 1998–2006; before that, CFO at Canada Post after 20 years at General Electric…”

PositiveTextWhat the encoder learns from it
Random crop of the anchor (Contriever)"…president of TransAlta Power LP from 1998–2006; he had been CFO at Canada Post…"to match identical words
Random passage, same page"April 13 2012: SNC-Lavalin to pay exiting CEO close to $5 million in severance…"nothing (an unrelated list of news links)
Adjacent passage (CoT-MAE pairing)"…CE Generation LLC – CPP Investment Board – Salton Sea Funding Corp. – …"names, without the content
xPRT's selected sibling (4 passages away)"1998–2006: former president of TransAlta Power Ltd… 2003–present: non-executive chairman, member of audit committee…"the same facts, stated differently

Here, only the sibling gives the encoder what retrieval needs: the same facts in different words. The Contriever crop can be matched on identical words. CoT-MAE’s adjacent passage shares names but not content, and the random passage shares nothing. The numbers are in line with our hypothesis:

  • The positive drives retrieval. With the loss fixed, changing only the positive moves Retrieval nDCG@10 from 15.4 to 24.8. Inside xPRT’s objective, replacing the selected positive costs 4.45 MTEB Avg / 8.59 Retrieval nDCG@10 (Table 3 in the manuscript). That is more than double the 1.93-point spread between five established objectives (Table 2: 40.80–42.73 in the manuscript).
  • Not every way of picking pairs works. Reweighting pairs instead of selecting them, as ReContriever does, does not help: its relevance weighting scores 51.40 MTEB Avg vs. 51.44 for its own uniform-weight control. Ranking siblings by dense E5 similarity alone, instead of our selection, costs 2.3 MTEB Avg / 3.3 Retrieval nDCG@10 (Table 3 in the manuscript).

Selection is the largest factor, but it is not the whole story. The objective decides how much of a good pair’s signal the model actually uses, and the two are complementary:

  • Same pairs, better objective. On identical selected pairs, xPRT’s objective adds +2.6 MTEB Avg / +4.4 Retrieval nDCG@10 over plain InfoNCE, so a plain contrastive loss leaves signal on the table. The contrastive term, which learns which passages are relevant to each other, does most of the work. The content terms, including the cross-passage term L_cross, add to it.
  • Not a re-run of existing pre-training. Applied on top of RetroMAE, DupMAE, SimLM, BoW-DPR, Revela, CoT-MAE or contrastive checkpoints, xPRT still improves each one by +3.4 to +5.8 MTEB Avg (please see our response to sHaY). So what xPRT provides is over and above what is learnt through any of the baselines (including bottleneck methods and contrastive methods).

So our contribution is a label-free recipe with two complementary parts. Selecting a sibling from the same document decides what a pair can teach, and the objective decides how much of it the model learns. We provide detailed ablations and analysis to this end which goes beyond what was previous established in the literature.


AuthorsOpenReview ↗

Rebuttal Response to Reviewer wF4n [3/3]

W2. The selection rule for siblings seems to not offer much over plain TF-IDF? It is better, but not by a significant amount...

One of our most critical findings is that sibling selection has a profound impact on the downstream performance.

A random sibling instead of a selected one costs 4.45 MTEB Avg / 8.59 Retrieval nDCG@10 (Table 3 in the manuscript). TF-IDF alone matches the fused ranking from BERT (47.10 / 29.89 vs. 47.12 / 29.21), from E5 (54.94 / 45.97 vs. 54.86 / 45.78) and after fine-tuning (57.87 vs. 57.90 MTEB Avg). Although we see similar average performance, we observe that the two scorers still behave differently. They pick different siblings, and the resulting models get different queries right.

TF-IDF pairs teach lexical properties like siblings with shared words are related; fused pairs offer semantic signals that select siblings with the same meaning.

  • Anchor: “Poll shows teens are hungry to learn how to start a business…” Fused: “51% of teens want to start their own business someday” (same survey, different words) TF-IDF: “Business leaders agree entrepreneurship adds value…” (more shared words, but a separate 2008 Gallup study)
  • Anchor: “Chicago teachers’ strike vote: polls open 6:30am, ballots checked…” Fused: “Reform groups ran ads urging parents to petition against the strike vote” (same event) TF-IDF: “Teachers want 30% pay raise… teachers begin voting on a strike” (repeats teachers / voting / strike)

We present some examples from the data to articulate our point:

AnchorFusedE5 onlyTF-IDF only
Artist bio: “Palm Springs Art Museum 2008 juried exhibition… honorable mention… 2004 UCLA Fowler Museum”“2008 Palm Springs Art Museum… 2008 Palm Springs City Hall (honorable mention)… 2004 Fowler Museum, UCLA” (same facts, as a list)“1990 Highways, Santa Monica… 1988 Olympic Art Festival, Seoul…” (same list, older shows)“Her screenplay wins first place at the Palm Springs Writers’ Guild contest” (shares “Palm Springs”, other award)
“Excess iron can encourage hair algae… it usually comes in on a purchased live plant or snail shells”“It can also come in as fragments with newly purchased fish… on snail shells” (same point)“Otocinclus catfish or bristlenose plecos may keep this algae in check” (a remedy instead)“This article covers brown diatom algae, thread/hair algae, black beard algae…” (article overview)
Story: “I was staring at her forever… her voice was blank and emotionless… ‘I got a bit distracted’”“‘…distracted… by how beautiful you are right now,’ I replied… her face isn’t expressing any emotions” (same moment)“Our food arrived. We enjoyed the food in silence…” (later scene)“‘Have you ever noticed how she talks so blankly?’” (earlier scene, shared words)
“Au pairs learn respect for other cultures… tolerance and mutual understanding”“Go Au Pair recruits au pairs from over 60 countries: Brazilian, Chinese, Colombian…” (same discussion)“‘Au pair’ is a French term meaning ‘on par’…” (generic definition)“Tags: adoption with au pair help, au pairs Bucks County…” (tag list)

Across all tasks, these differences get averaged and mostly cancel out. Some tasks are lexical in nature while some are semantic. As the fused captures the best of both worlds we recommend fused as the default choice, however practitioners may choose to tweak this based on their use cases.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Request for Review of Rebuttal Responses

Dear reviewers,

We hope our rebuttal addresses all the clarification questions, and that the newly provided results comparing additional baselines (including InfoNCE-based methods) against XpRT, as well as XpRT as a stage after every other pre-training baseline, provide a strong signal of complementary gains from our method. If the reviewers need any further clarification, we would be more than happy to provide more details and engage in discussion. As the discussion period ends soon on November 18, 2026 (AoE), we sincerely urge the reviewers to engage in discussion.

Thank you very much for your time and consideration.

AuthorsOpenReview ↗

Reminder 1: Engage in Discussion

Dear Reviewers,

As a gentle reminder, the discussion period closes on November 18, 2026 (AoE). We would greatly appreciate it if you could take a look at our rebuttal and let us know whether it addresses your concerns. If any questions remain, we are happy to clarify further while there is still time.

Thank you very much for your time.

AuthorsOpenReview ↗

Reminder: Engage in Discussion

Dear Reviewers,

As the discussion period closes very soon (November 18, 2026, AoE), we wanted to follow up one final time. We would be very grateful if you could take a look at our rebuttal and engage in discussion with us; we are happy to address any remaining questions or concerns.

If you feel our responses have addressed your concerns, we would sincerely appreciate it if you would consider revisiting your score. Thank you again for your time and effort in reviewing our work.

Best regards,

The Authors

AuthorsOpenReview ↗

Final Reminder: Engage in Discussion

Dear Reviewers,

As the discussion period is coming to a close in the next few hours, we wanted to reach out one last time. We would be truly grateful if you could take a look at our rebuttal and share your thoughts. If our responses have addressed your concerns, we would sincerely appreciate it if you would consider revisiting your score.

Thank you very much for your time and consideration.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.