- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a training method that uses candidate documents to construct temporary target embeddings, then backpropagates through only the query or its paired document on alternating steps. It shows retrieval gains on Qwen2.5-0.5B against InfoNCE.
Critical strengths
- The method is clearly specified.
- The ablation studies are extensive.
Critical weaknesses
- The evidence of generalization to other models is limited: The performance gains on Mistral and Gemma are marginal. Experiments with larger modern LLM models would further strengthen the evidence.
- The blockwise similarity computation and encoder replay build on established techniques and can also be applied to exact InfoNCE. It is valuable for an efficient implementation, but should not be presented as the core contribution.
Questions for authors
See weaknesses.
2 comments
Response to authors
Thanks for your reply. However, my concerns are still unresolved. I will keep my score.
Response to Reviewer GyC6
We thank the reviewer for their careful review and for recognizing the clear specification and extensive ablations. We appreciate the suggestions about larger models and the distinction between the learning method and its efficient implementation.
We agree that encoder replay and blockwise similarity computation are established techniques that also apply to InfoNCE. Our full-gradient InfoNCE control uses streamed scoring and gradient-cache replay for this reason. Both sampled InfoNCE and Drift compare against the full candidate set. We study a particular allocation of gradients, detached-target updates alternating between queries and paired documents. Replay and tiling make this practical.
The relation to InfoNCE is also explicit in Section 3 and Appendix A, under matching conditions, the query-side update gives the query-side InfoNCE gradient up to a positive scalar. We have now added a control that replaces the proposed reverse rule with the exact-key update on the positive-pair matrix. Its full-suite retrieval scores are 41.48 and 40.62, compared with 41.49 and 40.85 for the proposed rule. Although individual tasks respond differently, these close averages do not establish that the proposed reverse rule uniquely explains the retrieval gain.
Due to compute constraints, we haven't yet completed the model size comparisons that the reviewer suggested but we plan to add those experiments in the camera ready version. The additional experiments, however, do strengthen the evidence for the smaller-model setting. With matched temperature, two fresh Qwen2.5-0.5B seeds improve full-suite retrieval by 2.98 and 1.54 points, remaining 2.43 and 0.79 ahead without ArguAna. The overall 56-task average improves by 0.83 and 0.62, and the two-seed retrieval difference remains positive at all three tested learning rates.