- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper claims retrieval embeddings only look blind to evidence sufficiency because prior tests used additive probes. It shows an InferSent-style interaction probe beats cosine across 9 encoders, adds a cosine-matched swap test, an erasure analysis claiming the signal is “distributed,” and a conformal gate.
Critical strengths
None
Critical weaknesses
-
The “blindness” being refuted is completely self-constructed. The abstract and intro attribute the null to “prior work,” but no cited work actually runs an additive (q,d) probe (+ global PCA) and concludes the geometry is blind: Joren et al. (2025) use LLM autoraters. Ethayarajh (2019) and Timkey & van Schijndel (2021) are about anisotropy, not sufficiency. The null in Section 3 is the authors’ own reproduction of an unnamed “prior practice.” Separately, the fact that a linear model on a concatenation cannot express interactions is exactly why InferSent introduced |u−v| in 2017. The entire premise of the paper seems self-contructed.
-
Scoring 5 pairs with a cross-encoder costs nothing next to the LLM generation it precedes (orders of magnitude more per answer).
Questions for authors
- Look at the weaknesses section.
- Why is theorem 1 even stated if there is no use for it?
Response to Reviewer WsAA
Thank you. Your first weakness is correct, and the controls prompted by the other reviews show that the positive claim built on that premise does not hold either.
W1. No cited work ran an additive probe and concluded the geometry is blind; the null was ours. Moreover, in our own data the additive probe on [q; d] matches the interaction probe on ArguAna (0.704 against 0.685), and on SciFact the interaction probe's gain disappears once gold documents are held out (general response, point 1). The premise and the claim are both withdrawn.
W2. Agreed. With five retrieved documents, a cross-encoder on five pairs is negligible next to the generation it precedes. The cost argument is withdrawn.
Q2. Theorem 1 supported no empirical claim and should not have been stated as it was.