- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes xPRT, a retrieval-oriented pretraining method that selects related passages from the same document and uses cross-passage contrastive alignment and prediction to train a dense encoder. It evaluates continued pretraining from E5 and pretraining from BERT on MTEB English v2 benchmark. xPRT improves the E5 and BERT starting point and substantially outperforms the compared pretraining objectives.
Critical strengths
- The method design is thoughtful. Selecting semantically related passages within a document gives the model a more targeted training signal than pairing arbitrary passages. The combination of cross-passage alignment and prediction is well motivated, and the ablations in Table 3 help identify which components matter.
- The paper explains the motivation, training objectives, and experimental settings clearly. The figures and tables make the method and its results easy to follow.
Critical weaknesses
-
The BERT comparison stops before the downstream training stage that would test the value of a pretraining method. A BERT-initialized model trained only with a retrieval pretraining objective is usually an initialization for later contrastive retrieval training, rather than the final retriever. Table 2 evaluates the resulting embeddings directly, without applying the same subsequent contrastive training to xPRT and the baselines. It therefore cannot show whether xPRT is a better starting point for a trained dense retriever.
-
Direct evaluation may favor xPRT over bottleneck-based baselines for reasons unrelated to initialization quality. RetroMAE and related methods aim to strengthen the representation bottleneck before later retrieval training, whereas xPRT already includes a cross-passage contrastive loss that directly trains embedding similarity. Its advantage in the direct evaluation in Table 2 is thus plausible by design, but the paper does not show whether that advantage survives a common contrastive training stage.
Based on the above two points, starting all BERT-based models from their respective pretrained checkpoints and training them with the same contrastive objective and data would make the comparison more informative.
- More pretraining data does not yield a clear improvement. In Table 1, the average performance of xPRT did not improve as the training data increased; in fact, retrieval performance even declined. The paper suggests that the one-pass schedule and low learning rate may explain the plateau, but the current results do not establish that the method benefits from additional data. This prompted me to wonder what the effect might be of conducting an additional round of contrastive learning training.
Questions for authors
See above weakness
2 comments
Rebuttal Response to Reviewer sHaY [2/2]
W3. More pretraining data does not yield a clear improvement. In Table 1, the average performance of xPRT did not improve as the training data increased; in fact, retrieval performance even declined. The paper suggests that the one-pass schedule and low learning rate may explain the plateau, but the current results do not establish that the method benefits from additional data. This prompted me to wonder what the effect might be of conducting an additional round of contrastive learning training.
We agree that our results do not show gains beyond 100K documents under this protocol. What the results do show:
- The gain is reached early and stays stable. xPRT improves E5 by +1.80, +1.87 and +1.83 at 100K, 500K and 1M documents. The flat curve is shared by every objective under this protocol: BoW-DPR changes by +0.08 / +0.38 / −0.04 and SimLM by −2.05 / +0.26 / −1.70. It is not specific to xPRT, and xPRT is the only method that is both stable and positive.
- Most task types keep improving. Clustering and STS continue to rise (+0.40 and +0.28 from 100K to 1M). The small drop in retrieval is concentrated in TREC-COVID, Touché and FiQA (Table 9 in the manuscript).
- The advantage holds after an additional round of contrastive training, which is what the reviewer asked about. Please check our previous comment that answers this directly.
We thank the reviewer for proposing this experiment. We believe the fine-tuning results answer their concerns directly, and we would be grateful if the reviewer would reconsider the score.
Rebuttal Response to Reviewer sHaY [1/2]
We thank the reviewer for finding the method well motivated and the paper clear. The central question is whether xPRT's advantage survives the contrastive training that normally follows pre-training, and that is the right test. We ran it for every BERT-initialized checkpoint in Table 2 and for the contrastive controls requested by the other reviewers.
W1 and W2. Identical contrastive fine-tuning of every initialization. Every checkpoint is fine-tuned on MSMARCO with the same objective, data and schedule.
Key Takeaways
xPRT on top of existing pre-training baselines
The reviewer notes that bottleneck methods are meant to prepare the encoder for later contrastive training, so to make a fair comparison we run xPRT as a next stage in pretraining over botteneck as well as contrastive pre-trained checkpoints. This baseline in the BERT setting and with 100k data scale.
The key point to note here is XPRT as a continual pretraining objective offers significant over and above every above baseline, be it bottleneck or contrastive. This directly addresses the reviewer's concern even InfoNCE baselines move beyond their baseline number and this proves the gains are due to XPRT's full objective and not just because of it being a contrastive style loss.