- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2(+1)
- Soundness
- 2
Read review
Paper summary
The paper constructs three-dimensional MaxSim embeddings that exactly reproduce inner products of arbitrary-dimensional nonnegative sparse vectors with representation length proportional to sparsity. It proves a limitation for signed inner products under a precisely defined one-vector-per-nonzero-coordinate encoding, proposes Signed MaxSim with a post-argmax sign, and relates ordinary MaxSim to positive-CNF evaluation. A trained continuous-weight version, Fallon, improves exclusionary retrieval over a ColBERT baseline, especially on synthetic vocabulary and query-structure shifts. Despite the value of its theoretical contributions, the work suffers from a significant lack of sufficient experiments to demonstrate the method's effectiveness in practice and application, and the writing is difficult to follow.
Critical strengths
- Interesting research question. Understanding what MaxSim can represent, and how that capacity relates to sparse matching and exclusionary retrieval, is a worthwhile question for retrieval research.
- Substantive theoretical analysis. The constructive arguments, the qualified impossibility result for standard MaxSim with signed inputs, and the connection to positive-CNF evaluation offer a useful way to reason about late-interaction scoring (Sections 3–5).
Critical weaknesses
- The exposition makes the central argument difficult to follow. The main text moves among a lot of concepts and theorems without clearly explaining which theoretical property each experiment tests. Much of the space is devoted to formal statements and proof intuition, while essential empirical information is deferred to Appendix. In particular, the generation procedure and distribution shifts of the synthetic task—one of only two experimental settings—are needed to interpret the large gains in Table 1 and should be summarized in the main text. A clearer narrative connecting the assumptions, the model design, and the evidence would make the contribution easier to assess.
- The experimental evidence is too limited for the paper's retrieval claims. The experiments cover one synthetic retrieval task (with three test settings) and ExcluIR, and compare Fallon only with a ColBERT/MaxSim baseline (Tables 1–2). The abstract and introduction discuss the empirical advantage of late interaction over sparse and dense retrieval, yet no sparse or dense retriever is evaluated here. The paper also lacks analyses or case studies showing how Fallon behaves differently: for example, whether it learns to penalize excluded concepts, or how negative query tokens affect final scores. For a retrieval paper, even one centered on theory, targeted comparisons and mechanism-level evidence are needed to connect the formal results to the practical conclusions.
- Generalization language overreaches the evidence. The largest gains in Table 1 come from templated synthetic data; the realistic ExcluIR gains are smaller. The paper should distinguish exact representational capacity from the learned model's generalization and avoid calling Fallon a clear choice broadly.
- Retrieval scalability is unresolved. Fallon uses exhaustive scoring because Signed MaxSim is not directly compatible with PLAID. Online cost and indexing would matter for the claimed retrieval utility.
Questions for authors
See the weaknesses
3 comments
The paper should distinguish exact representational capacity from the learned model's generalization and avoid calling Fallon a clear choice broadly.
This is a reasonable point, but even in-domain Fallon is statistically better than ColBERT. With that said, it would be good to be more metered with our language and ensure we highlight Fallon is clearly better out-of-domain (e.g. 1.0 nDCG@10 vs 0.597 nDCG@10 for Different Vocabulary), but in-domain the gains are less substantial.
Retrieval scalability is unresolved. Fallon uses exhaustive scoring because Signed MaxSim is not directly compatible with PLAID. Online cost and indexing would matter for the claimed retrieval utility.
As we mentioned in Appendix E.3 “...our method does not preclude an approximate search technique, but developing one was outside the scope of this work.” As developing efficient retrieval techniques often are stand-alone papers we believe this shouldn’t be considered a critical weakness of our paper.
Thank you for the detailed response. The rebuttal addressed most of my concerns and strengthened my assessment of the paper. I will raise my overall rating accordingly.
Thank you for taking the time to review our work and provide meaningful feedback. We are glad to hear that you learned something useful from our work! Our responses to your concerns and suggestions are below; we hope that you will consider raising your rating if you find that they address your concerns sufficiently.
The exposition makes the central argument difficult to follow.
We are sorry to hear the central argument of our work was difficult to follow. Given the space constraints we had to make decisions about what was most important for our paper and move the remaining details to the appendix. We tried our best to keep the narrative clear in this process, but it is difficult to fit everything in the space we had. In the future version, with an additional page, we will add back some of the details we had to move and hopefully address some of your concerns.
Much of the space is devoted to formal statements and proof intuition, while essential empirical information is deferred to Appendix
As the core contribution is our theoretical work we did prioritize the proofs in the main body of the paper (although many of the full proofs are still in the appendix). With that said, we will add some additional experimental details in the next edition to make the experiments easier to interpret without going to the appendix.
A clearer narrative connecting the assumptions, the model design, and the evidence would make the contribution easier to assess
We agree that we could more clearly connect our experiments to the theoretical findings and make our hypothesis about why ColBERT will do worse on negation tasks more explicit. Specifically, we will clarify that we believe exclusion queries are a natural candidate to leverage negative query-level scores, something which our theory shows ColBERT struggles with. Although our theory does not preclude ColBERT from handling negations. Our experiments show that Signed MaxSim is able to benefit from expressing negative query-level scores as shown by the strong results and ablation with the non-negative Fallon variant.
The abstract and introduction discuss the empirical advantage of late interaction over sparse and dense retrieval, yet no sparse or dense retriever is evaluated here.
We originally did not include a single-vector baseline because the superiority of ColBERT compared with single-vector models is well established in the literature (https://aclanthology.org/2022.naacl-main.272, https://arxiv.org/pdf/2607.27178) and the failure modes have been clearly shown in prior work, such as the poor performance on the LIMIT dataset (https://proceedings.iclr.cc/paper_files/paper/2026/hash/130b10e31f5c3d730162323b2685bd19-Abstract-Conference.html). Because of this, we felt that the addition of a single-vector baseline would not contribute much to our work and focused on the more direct comparison with ColBERT. With that said, we can see that it would be a useful comparison point and have trained and evaluated a matched dense retrieval baseline on the synthetic retrieval task. The results are available in the updated paper PDF in Table 1. In brief, the dense model performs worse than both Fallon and ColBERT on the In-Domain and Different Vocabulary test and does better than ColBERT but worse than Fallon on the Negation Only task. These results align with our theory that dense models would struggle to maintain all of the independent features while being capable of directly penalizing negations. For more discussion on the dense baseline please see our top-level comment about it.
The paper also lacks analyses or case studies showing how Fallon behaves differently: for example, whether it learns to penalize excluded concepts
Please see our top-level comment “Visualization of Fallon’s Weights” or Appendix F.2 in the updated paper. We did additional analysis and visualization of the weights Fallon produces and found they were as expected with negated terms getting negative values (-1.78) and necessary terms receiving high positive weights (e.g. +4.81).
Generalization language overreaches the evidence. The largest gains in Table 1 come from templated synthetic data; the realistic ExcluIR gains are smaller.
The most striking results are from synthetic data, but both ExcluIR and the synthetic in-domain data show that ColBERT is fairly close to Fallon (although Fallon is still significantly better). We believe when trained in-domain ColBERT is capable of learning exclusion fairly well, but the approach it learns is not generalizable as we show in the out-of-domain evaluation using synthetic data. We believe for similar out-of-domain data for ExcluIR (or other realistic tasks) ColBERT would fail in similar ways to the synthetic data, but unfortunately to the best of our knowledge, this data doesn’t exist which is why we leveraged synthetic data for our experiments.