- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies when dense retrieval should use shared query–document projections versus separate projections. The authors develop a bias–variance theory showing that dual projections can better capture directional differences between queries and documents, but require more data to estimate reliably. They derive a local Gaussian decision boundary and propose CARS, a cross-fitted method for choosing between the two geometries.
Critical strengths
-
The paper provides a useful theoretical perspective on the tradeoff between shared and dual projections, with an explicit boundary relating directional signal to the estimation cost of additional degrees of freedom.
-
The experimental evaluation covers multiple embedding models, retrieval datasets, ranks, and training sizes.
Critical weaknesses
-
The paper was hard to follow; particularly, although the mathematical derivation is detailed, it lacked the intuition, and the practical implications of several key results are not sufficiently explained. It was difficult to understand what the derived quantities imply in a real retrieval setting and how the theoretical results translate into practical guidance for choosing between shared and dual projections. Hence, I suggest that the authors to include more intuitive explanations when presenting the main mathematical results along with concrete examples to make the paper easier to follow.
-
The theoretical boundary relies on a local Gaussian model with isotropic noise. Although the experiments support similar qualitative trends on real embeddings, it remains unclear under what data conditions and practical settings the proposed boundary is expected to hold. Hence, more guidance on the assumptions required for the boundary to be useful in practice would help clarify the scope and applicability of the theoretical results.
Questions for authors
-
Could the authors provide more intuitive explanations or concrete examples to clarify the meaning of the key theoretical quantities during the theoretical derivations?
-
Under what data conditions and practical settings is the proposed theoretical boundary expected to hold? In particular, how sensitive is it to deviations from the local Gaussian assumption, and with what data would one expect such a boundary to hold?
3 comments
Response to Reviewer mAwG (2/2)
Question2 (& Weakness2): Under what data conditions and practical settings is the proposed theoretical boundary expected to hold? In particular, how sensitive is it to deviations from the local Gaussian assumption, and with what data would one expect such a boundary to hold?
Thank you for asking when the boundary applies. For fixed dimension and rank, our first-order analysis considers departures of order from a rank- Shared operator with a positive spectral gap at the rank cutoff. Query contributions are i.i.d. with uniformly bounded fourth moments, and CARS uses random splits into equal halves. The variance cost depends on covariance in the Dual-only directions; this comparison does not assume Gaussian or isotropic noise. In practical retrieval settings, bounded fourth moments hold automatically for L2-normalized frozen embeddings, and the boundary is expected to hold whenever the target rank is chosen at a clear singular-value drop (spectral gap) relative to the query budget .
We added a noise-sensitivity simulation to test deviations from Gaussianity and isotropy. For each spectral gap and sample size, the 36 settings cross three noise laws (Gaussian, Student-, and contaminated Gaussian), three covariance allocations, and four signal strengths. The table reports agreement with the geometry of lower mean operator risk. Mean CARS uses the sign of the score averaged over simulation draws.
| Spectral gap | Prediction | 64 queries | 256 | 1,024 | 4,096 |
|---|---|---|---|---|---|
| 0.15 | Isotropic boundary | 30/36 | 30/36 | 30/36 | 30/36 |
| 0.15 | Covariance-adjusted boundary | 30/36 | 30/36 | 30/36 | 36/36 |
| 0.15 | Mean CARS | 30/36 | 30/36 | 33/36 | 36/36 |
| 0.30 | Isotropic boundary | 30/36 | 30/36 | 30/36 | 30/36 |
| 0.30 | Covariance-adjusted boundary | 30/36 | 30/36 | 36/36 | 36/36 |
| 0.30 | Mean CARS | 30/36 | 33/36 | 36/36 | 36/36 |
| 1.00 | Isotropic boundary | 31/36 | 30/36 | 30/36 | 30/36 |
| 1.00 | Covariance-adjusted boundary | 35/36 | 36/36 | 36/36 | 36/36 |
| 1.00 | Mean CARS | 29/36 | 36/36 | 36/36 | 36/36 |
As the cutoff gap grows from 0.15 to 0.30 to 1.00, the first tested sample size at which both the covariance-adjusted boundary and mean CARS agree in all 36 settings falls from 4,096 to 1,024 to 256 queries. At a gap of 0.30 and 1,024 queries, both agree in 36/36 settings across heavy-tailed and contaminated noise, compared with 30/36 for the isotropic boundary. This explains why CARS remains reliable on real benchmarks (FEVER, NQ, MS MARCO, ArguAna, and SciFact), where unit-norm embeddings naturally bound tail moments and compact ranks () enjoy well-separated leading singular values.
You can find more information in the revised manuscript (Sections 3.2 and 5.4 and Appendices C.2 and I).
Thank you again for your comments and for helping us improve the manuscript.
Response to Rebuttal
Thanks to the authors for the clarifications and for addressing my concerns. I retain my acceptance recommendation.
Response to Reviewer mAwG (1/2)
We sincerely appreciate your time in reviewing the manuscript and your thoughtful feedback. We reply to your comments below.
Thank you for this constructive feedback. We have added concrete geometric and numerical examples throughout the revised manuscript.
The practical implication is to use Shared when data are scarce and directional signal is weak, and to favor Dual when the reproducible signal outweighs its estimation cost. When the CARS score is close to zero or varies across splits, we recommend comparing the two fits on validation NDCG@10.
You can find more information in the revised manuscript (Sections 2, 3, 5.4, and 6.)