acceptodds
Under review as a conference paper at ICLR 2027

Quantifying and Expanding the Theoretical Capacity of Late-Interaction Retrieval Models

Abstract

Late-interaction retrieval models that use the MaxSim similarity function have shown strong empirical performance, often outperforming single-vector dense and sparse retrieval models. Despite these empirical findings, little is known about the theoretical representation power of MaxSim and how it compares to other retrieval approaches. This paper shows by construction that MaxSim similarity can exactly replicate the inner product between any two non-negative -sparse vectors with possibly infinite dimension, requiring only representation space. Moreover, there exist similarities that MaxSim can express while standard vector inner products with the same representation space cannot. Leveraging our theoretical framework, we introduce Signed MaxSim which allows late-interaction models to exactly replicate any real-valued inner product, something we prove standard MaxSim is not capable of. We also show that MaxSim can act as an aggregation of soft-OR operations and as an evaluator of logical expressions in positive Conjunctive Normal Form. Our findings show that MaxSim is at least as capable as standard vector inner products for any non-negative vectors and our extension, Signed MaxSim, is as capable for any vectors. Both similarities possess additional capabilities that inner product cannot replicate, marking one of the first theoretical justifications and quantifications of late-interaction methods. Our theoretical findings are supported empirically: on a retrieval task featuring queries with negations, Signed MaxSim improves out-of-domain performance significantly over a standard ColBERT/MaxSim baseline with nDCG@10 increasing from 0.597 to 1.000 under a vocabulary shift and from 0.008 to 0.788 on negation-only queries.

Primary area:
learning theory
Keywords:
Late-Interaction, Retrieval Model, MaxSim, Theory

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_RaNkOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2(+1)
Soundness
2
Read review
Paper summary

The paper constructs three-dimensional MaxSim embeddings that exactly reproduce inner products of arbitrary-dimensional nonnegative sparse vectors with representation length proportional to sparsity. It proves a limitation for signed inner products under a precisely defined one-vector-per-nonzero-coordinate encoding, proposes Signed MaxSim with a post-argmax sign, and relates ordinary MaxSim to positive-CNF evaluation. A trained continuous-weight version, Fallon, improves exclusionary retrieval over a ColBERT baseline, especially on synthetic vocabulary and query-structure shifts. Despite the value of its theoretical contributions, the work suffers from a significant lack of sufficient experiments to demonstrate the method's effectiveness in practice and application, and the writing is difficult to follow.

Critical strengths
  1. Interesting research question. Understanding what MaxSim can represent, and how that capacity relates to sparse matching and exclusionary retrieval, is a worthwhile question for retrieval research.
  2. Substantive theoretical analysis. The constructive arguments, the qualified impossibility result for standard MaxSim with signed inputs, and the connection to positive-CNF evaluation offer a useful way to reason about late-interaction scoring (Sections 3–5).
Critical weaknesses
  1. The exposition makes the central argument difficult to follow. The main text moves among a lot of concepts and theorems without clearly explaining which theoretical property each experiment tests. Much of the space is devoted to formal statements and proof intuition, while essential empirical information is deferred to Appendix. In particular, the generation procedure and distribution shifts of the synthetic task—one of only two experimental settings—are needed to interpret the large gains in Table 1 and should be summarized in the main text. A clearer narrative connecting the assumptions, the model design, and the evidence would make the contribution easier to assess.
  2. The experimental evidence is too limited for the paper's retrieval claims. The experiments cover one synthetic retrieval task (with three test settings) and ExcluIR, and compare Fallon only with a ColBERT/MaxSim baseline (Tables 1–2). The abstract and introduction discuss the empirical advantage of late interaction over sparse and dense retrieval, yet no sparse or dense retriever is evaluated here. The paper also lacks analyses or case studies showing how Fallon behaves differently: for example, whether it learns to penalize excluded concepts, or how negative query tokens affect final scores. For a retrieval paper, even one centered on theory, targeted comparisons and mechanism-level evidence are needed to connect the formal results to the practical conclusions.
  3. Generalization language overreaches the evidence. The largest gains in Table 1 come from templated synthetic data; the realistic ExcluIR gains are smaller. The paper should distinguish exact representational capacity from the learned model's generalization and avoid calling Fallon a clear choice broadly.
  4. Retrieval scalability is unresolved. Fallon uses exhaustive scoring because Signed MaxSim is not directly compatible with PLAID. Online cost and indexing would matter for the claimed retrieval utility.
Questions for authors

See the weaknesses

3 comments

Thank you for taking the time to review our work and provide meaningful feedback. We are glad to hear that you learned something useful from our work! Our responses to your concerns and suggestions are below; we hope that you will consider raising your rating if you find that they address your concerns sufficiently.

The exposition makes the central argument difficult to follow.

We are sorry to hear the central argument of our work was difficult to follow. Given the space constraints we had to make decisions about what was most important for our paper and move the remaining details to the appendix. We tried our best to keep the narrative clear in this process, but it is difficult to fit everything in the space we had. In the future version, with an additional page, we will add back some of the details we had to move and hopefully address some of your concerns.

Much of the space is devoted to formal statements and proof intuition, while essential empirical information is deferred to Appendix

As the core contribution is our theoretical work we did prioritize the proofs in the main body of the paper (although many of the full proofs are still in the appendix). With that said, we will add some additional experimental details in the next edition to make the experiments easier to interpret without going to the appendix.

A clearer narrative connecting the assumptions, the model design, and the evidence would make the contribution easier to assess

We agree that we could more clearly connect our experiments to the theoretical findings and make our hypothesis about why ColBERT will do worse on negation tasks more explicit. Specifically, we will clarify that we believe exclusion queries are a natural candidate to leverage negative query-level scores, something which our theory shows ColBERT struggles with. Although our theory does not preclude ColBERT from handling negations. Our experiments show that Signed MaxSim is able to benefit from expressing negative query-level scores as shown by the strong results and ablation with the non-negative Fallon variant.

The abstract and introduction discuss the empirical advantage of late interaction over sparse and dense retrieval, yet no sparse or dense retriever is evaluated here.

We originally did not include a single-vector baseline because the superiority of ColBERT compared with single-vector models is well established in the literature (https://aclanthology.org/2022.naacl-main.272, https://arxiv.org/pdf/2607.27178) and the failure modes have been clearly shown in prior work, such as the poor performance on the LIMIT dataset (https://proceedings.iclr.cc/paper_files/paper/2026/hash/130b10e31f5c3d730162323b2685bd19-Abstract-Conference.html). Because of this, we felt that the addition of a single-vector baseline would not contribute much to our work and focused on the more direct comparison with ColBERT. With that said, we can see that it would be a useful comparison point and have trained and evaluated a matched dense retrieval baseline on the synthetic retrieval task. The results are available in the updated paper PDF in Table 1. In brief, the dense model performs worse than both Fallon and ColBERT on the In-Domain and Different Vocabulary test and does better than ColBERT but worse than Fallon on the Negation Only task. These results align with our theory that dense models would struggle to maintain all of the independent features while being capable of directly penalizing negations. For more discussion on the dense baseline please see our top-level comment about it.

The paper also lacks analyses or case studies showing how Fallon behaves differently: for example, whether it learns to penalize excluded concepts

Please see our top-level comment “Visualization of Fallon’s Weights” or Appendix F.2 in the updated paper. We did additional analysis and visualization of the weights Fallon produces and found they were as expected with negated terms getting negative values (-1.78) and necessary terms receiving high positive weights (e.g. +4.81).

Generalization language overreaches the evidence. The largest gains in Table 1 come from templated synthetic data; the realistic ExcluIR gains are smaller.

The most striking results are from synthetic data, but both ExcluIR and the synthetic in-domain data show that ColBERT is fairly close to Fallon (although Fallon is still significantly better). We believe when trained in-domain ColBERT is capable of learning exclusion fairly well, but the approach it learns is not generalizable as we show in the out-of-domain evaluation using synthetic data. We believe for similar out-of-domain data for ExcluIR (or other realistic tasks) ColBERT would fail in similar ways to the synthetic data, but unfortunately to the best of our knowledge, this data doesn’t exist which is why we leveraged synthetic data for our experiments.

The paper should distinguish exact representational capacity from the learned model's generalization and avoid calling Fallon a clear choice broadly.

This is a reasonable point, but even in-domain Fallon is statistically better than ColBERT. With that said, it would be good to be more metered with our language and ensure we highlight Fallon is clearly better out-of-domain (e.g. 1.0 nDCG@10 vs 0.597 nDCG@10 for Different Vocabulary), but in-domain the gains are less substantial.

Retrieval scalability is unresolved. Fallon uses exhaustive scoring because Signed MaxSim is not directly compatible with PLAID. Online cost and indexing would matter for the claimed retrieval utility.

As we mentioned in Appendix E.3 “...our method does not preclude an approximate search technique, but developing one was outside the scope of this work.” As developing efficient retrieval techniques often are stand-alone papers we believe this shouldn’t be considered a critical weakness of our paper.

Thank you for the detailed response. The rebuttal addressed most of my concerns and strengthened my assessment of the paper. I will raise my overall rating accordingly.

Reviewer_A6CaOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the expressive power of MaxSim and introduces Signed MaxSim. It constructs low-dimensional representations that exactly recover nonnegative sparse inner products, proves a limitation for signed inner products under a specified encoding class, and shows how the signed extension overcomes it. It also connects MaxSim to positive CNF evaluation and tests the proposed model, Fallon, on exclusionary retrieval.

Critical strengths
  • The constructive results explain how MaxSim can recover sparse inner products and why the maximum operation matters. The signed and positive-CNF extensions fit naturally into the same framework, and the proofs appear sound under their stated assumptions.

  • The experiments test behaviors motivated by the theory. The synthetic settings separate vocabulary shift from negation-only queries, while the ExcluIR results provide evidence beyond the synthetic setup.

Critical weaknesses
  • The discussion in §7 overstates the theory. Theorem 3.2 concerns exact signed inner-product recovery under a particular sparsity-preserving encoding class, which is narrower than an inability to handle semantic negation. The construction also uses real-valued coordinates whose magnitudes grow with coordinate indices, so the representation-size result should be distinguished from a fixed-precision memory bound.

  • Fallon adds a five-layer weight head and relaxes signs to unrestricted real-valued weights. The experiments therefore change both the matching rule and the model’s weighting capacity. Without a matched nonnegative-weight control, it is unclear how much of the improvement specifically requires negative contributions.

Questions for authors
  • Which claims about semantic negation follow from the theorems, and which are interpretations of the experiments? Please make this distinction clearer in the discussion.

  • How does Fallon perform with the same weight head constrained to produce nonnegative values, using a comparable tuning budget?

  • How do the required embedding magnitudes and numerical precision grow with vocabulary size in the constructive encoding?

2 comments

Thank you for taking the time to review our work and provide meaningful feedback. We are glad to hear that you learned something useful from our work! Our responses to your concerns and suggestions are below; we hope that you will consider raising your rating if you find that they address your concerns sufficiently.

The discussion in §7 overstates the theory.

We agree that we slightly overstated the theory and we should be careful to not apply our theoretical results more broadly than what is justified. The theoretical results provide some insight about the experimental results, but we should be careful to not assert the exact mechanism or cause. We will change the language to make this clearer in the next version.

Without a matched nonnegative-weight control, it is unclear how much of the improvement specifically requires negative contributions.

We agree. See our top-level comment about the Non-Negative Fallon Ablation or the newly added Appendix F.1 which includes the full results and discussion. To summarize, we found that a nonnegative-weight control was largely similar to ColBERT except for the Negation Only subset where it did a lot better. We believe that this was in large part because the model could zero-out many terms that might not be useful and allowed it to act more like a dense retriever without having to worry about the maximum. In any case, the standard Fallon model significantly outperforms the constrained ablation which demonstrates the need for negative values for optimal performance.

Which claims about semantic negation follow from the theorems, and which are interpretations of the experiments? Please make this distinction clearer in the discussion.

We agree and will update the discussion to be clearer. To answer your question, our theory does not preclude ColBERT from handling negation, but it does show it is limited in its ability to have a negative contribution for a query embedding (as shown in the real-value inner product limitation proof). We hypothesized that directly penalizing a negated query term, by assigning a negative value to it, is an intuitive mechanism for handling semantic negations and one that standard ColBERT would struggle with. We will make this clearer in our discussion.

How does Fallon perform with the same weight head constrained to produce nonnegative values, using a comparable tuning budget?

See our prior comment and the new experiments we ran.

How do the required embedding magnitudes and numerical precision grow with vocabulary size in the constructive encoding?

It is important to remember our constructions are not intended to be directly used and mostly serve as proof machinery that have been used in similar ways in other proofs. With that said, to get a quick idea of the limitations we ran a preliminary analysis using Numpy to calculate the similarity using the construction in Theorem 3.1. We were able to use a vocabulary size of 10 million with FP64 without substantial errors although the stability is highly dependent on the weights (i.e. values in the original sparse vectors).

Additionally, we added an extension to Theorem 3.1 in Appendix B.4 which largely maintains the findings from Theorem 3.1 while constraining all embeddings to be l2-normalized. This should address your concerns with magnitude growth as the results are largely unchanged even when magnitudes are constant for all embeddings.

Thanks for the additional results. The ReLU ablation strengthens the case that signed weights contribute beyond the extra weighting head, and Appendix B.4 addresses the normalization concern through a clearly stated ranking-preservation result. These additions make me more confident in my positive assessment. Please still revise Section 7, since the theorem does not establish a general inability of MaxSim to handle semantic negation. Overall, I find the theoretical contribution useful and the supporting experiments encouraging, and will retain my positive rating of 3.

Reviewer_4jaxOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the expressive ability of the MaxSim similarity used by late-interaction retrievers. It shows by construction that MaxSim can exactly reproduce the inner product of any two non-negative k-sparse vectors of arbitrary dimension (Thm 3.1), no single-vector representation in R^d can do this (Thm 4.1), and standard MaxSim cannot recover signed inner products under a sparsity-preserving encoding (Thm 3.2). Based on which it also propose Signed MaxSim, which applies query and document signs after the max. Empirically, a ColBERT variant with Signed MaxSim (Fallon) is compared with a matched ColBERT on synthetic negation data and ExcluIR.

Critical strengths
  • The main statement that late interaction is strictly more expressive than single-vector inner products for sparse/high-dimensional and Boolean-style matching, is interesting and useful for the community. The constructions are simple and the proofs are easy to check.
  • Signed MaxSim is a simple modification with a clear motivation, and the negation-only result (ColBERT nDCG@10 0.008 vs 0.788) illustrates a real failure mode of MaxSim.
  • The paper is clearly written and honest about concurrent work.
Critical weaknesses
  • The theory does not connect strongly to the application. The theorems are existence constructions with polynomial coefficients growing with the index that an l2-normalized trained model would not realize, and the paper does not propose how to train models to exploit this capacity. Thm 3.2 is about an encoding class, not about what a trained ColBERT can learn, so it does not directly explain the experimental results.
  • Empirical evaluation is narrow. Only negation/exclusion datasets are used, although Signed MaxSim is proposed as a general replacement. A standard benchmark is needed to show it does not hurt normal retrieval. There is only one baseline (matched ColBERT) and no single-vector model is included even though single vs. multi-vector is the theoretical headline
Questions for authors
  • How does Fallon compare with ColBERT (and a single-vector baseline) on a standard benchmark such as BEIR or MS MARCO?
  • Is there a good explanation for why Fallon reach 1.000 nDCG@10 and 1.000 AP on the different-vocabulary set, above its in-domain score?
3 comments

Thank you for taking the time to review our work and provide meaningful feedback. We are glad to hear that you learned something useful from our work! Our responses to your concerns and suggestions are below; we hope that you will consider raising your rating if you find that they address your concerns sufficiently.

The theorems are existence constructions with polynomial coefficients growing with the index that an l2-normalized trained model would not realize. We agree this was a limitation with our original submission. In the updated version we have uploaded we have added Appendix B.4 which adds an extension to Theorem 3.1 for l2-normalized embeddings. It does require a cap of the possible weight for the “document” sparse vector and only can replicate the sparse inner product up to a fixed constant, as well as requiring five embedding dimensions instead of three. Although these add constraints we believe they are reasonable and fit with real-world usage where the ranking order is what ultimately matters. The l2-normalization construction can also be trivially applied to the Signed MaxSim with similar constraints.

Thm 3.2 is about an encoding class, not about what a trained ColBERT can learn, so it does not directly explain the experimental results. With the additional extension from Appendix B.4 our formulation better reflects how ColBERT models are trained. It is worth emphasizing we are not claiming to explain exactly how ColBERT models train or actually utilize MaxSim, our contribution is demonstrating the theoretical capacity of MaxSim and showing how this differs from standard inner products between finite vectors and how MaxSim relates to sparse inner products. We believe this capacity helps to explain why ColBERT works well empirically, but we do not believe ColBERT models are necessarily learning to produce polynomial embedding like those used in our construction.

A standard benchmark is needed to show it does not hurt normal retrieval. We agree this is a useful addition and have done additional experiments with Fallon and ColBERT on general retrieval datasets. See our top-level comment “General Retrieval Data Comparison” for details.

There is only one baseline (matched ColBERT) and no single-vector model is included even though single vs. multi-vector is the theoretical headline We originally did not include a single-vector baseline because the superiority of ColBERT compared with single-vector models is well established in the literature (https://arxiv.org/pdf/2112.01488, https://arxiv.org/pdf/2607.27178) and the failure modes have been clearly shown in prior work, such as the poor performance on the LIMIT dataset (https://arxiv.org/pdf/2508.21038). Because of this, we felt that the addition of a single-vector baseline would not contribute much to our work and focused on the more direct comparison with ColBERT. With that said, we can see that it would be a useful comparison point and have trained and evaluated a matched dense retrieval baseline on the synthetic retrieval task. The results are available in the updated paper PDF in Table 1. For more discussion on the dense baseline please see our top-level comment about it.

Is there a good explanation for why Fallon reach 1.000 nDCG@10 and 1.000 AP on the different-vocabulary set, above its in-domain score? We believe the reason is that there is a smaller vocabulary for this subset which resulted in more positive documents per query (more documents have the relevant attribute as there are fewer attributes to choose from). For the in-domain there were 67.80 positives per query on average versus 336.56 positives per query in the different vocabulary set. Because there are more positives this likely makes retrieval easier and the limited vocabulary might similarly reduce the complexity of the retrieval task. We will be sure to add these statistics and a more detailed explanation in the discussion. It is also worth highlighting even with the reduced complexity ColBERT is unable to compete with Fallon.

Thanks for the response! I believe the added explanation should make the contribution clearer. I acknowledge the limited time for adding experiments, so even I don't really agree with comparing methods in their early training stage, I am fine with it for now. Please add complete results if the paper get accepted. I would like to keep my score to vote for an acceptance.

We agree that the complete results will be the most informative and appreciate your understanding for this limitation. We will be sure to include the complete result in the final paper. As late-interaction models tend to train in fewer steps than other types of retrieval models, such as dense retrieval models, we believe the early stage results are still informative and likely reflect a reasonably accurate portrait of the generalization of Fallon models.

We thank you again for taking the time to provide a review for our work and thank you for responding to our rebuttal.

Comments on the paper

3 comments

Thank you to all the reviewers for your time and well considered suggestions. We have taken a lot of time to run additional experiments and analysis based on your reviews. We are posting the core additions as a top-level comment as several additions are relevant to multiple reviews. Additionally, we have updated our PDF with the new content, although it is not fully integrated yet so please keep in mind we are planning to move things around a bit.

Dense Retrieval Baseline

Two reviewers suggested an additional dense retrieval baseline. We believed it would be a useful point of comparison so we added the results using a dense baseline to the synthetic data results. The new dense baseline results can be viewed in the updated Table 1 or below.

The results show that as expected the dense retrieval model is not able to do well on the difficult multi-aspect queries and underperforms both ColBERT and Fallon In-Domain and out-of-domain with Different Vocab. For the Negation Only data the dense baseline outperforms ColBERT but is still substantially worse than Fallon. This aligns with our theory as dense retrieval models can encode negation and negative values in a straightforward way so it is reasonable that it surpasses ColBERT in the Negation Only task. Additionally, the Negation Only task has far more positives than the prior two tasks and is the simplest with most queries only containing a single negated requirement which explains why the dense model performs best on this task. Overall the dense results support our theoretical claims about the capacity of dense models and the limitation of ColBERT models when it comes to representing negations.

+---------+---------------------------+---------------------------+---------------------------+
|         |         In-Domain         |     Different Vocab.      |       Negation Only       |
| Model   +---------+--------+--------+---------+--------+--------+---------+--------+--------+
|         | nDCG@10 |  P@10  |   AP   | nDCG@10 |  P@10  |   AP   | nDCG@10 |  P@10  |   AP   |
+---------+---------+--------+--------+---------+--------+--------+---------+--------+--------+
| ColBERT |   0.982 |  0.875 |  0.943 |   0.597 |  0.587 |  0.530 |   0.008 |  0.009 |  0.766 |
| Dense   |   0.387 |  0.336 |  0.185 |   0.371 |  0.353 |  0.109 |   0.487 |  0.497 |  0.821 |
| Fallon  |   0.997 |  0.893 |  0.997 |   1.000 |  0.955 |  1.000 |   0.788 |  0.789 |  0.977 |
+---------+---------+--------+--------+---------+--------+--------+---------+--------+--------+

Training Details

We used the same training approach and hyperparameters as ColBERT and Fallon. We followed the same approach to perform an LR sweep and select the best candidate model to use. We used an embedding dimension of 768 with an MLP after the transformer encoder to match the ColBERT and Fallon implementations. We used mean pooling for the model.

General Retrieval Data Comparison

Two reviewers mentioned that comparing Fallon and ColBERT on standard retrieval datasets would be interesting and provide evidence that Signed MaxSim is a generally useful method and not one that is only useful for exclusionary queries. To evaluate this we trained three models on RLHN 250k one Fallon model and two ColBERT baselines. We used the best LR for Fallon and ColBERT on ExcluIR and for ColBERT also tried the best LR for Fallon to minimize risk of different learning rates contributing to the outcome. Due to the larger training corpus size and minimal time we could not do a full LR sweep.

Otherwise, the same hyperparameters and training setup was used as our prior experiments. For validation we used TREC DL 21 + 22 using only the QREL judged documents to form the pool (including those marked with 0). We selected the best checkpoint for each model at or before 30k steps (due to limited time) that had the best nDCG@10. Evaluating on TREC DL 19 we find that Fallon outperforms both of the ColBERT baselines. Evaluating on a subset of the smaller BEIR tasks we see Fallon performs better in four out of five tasks.
These results suggest that Fallon is equally capable on general retrieval tasks and may be slightly better than ColBERT on average.

TREC DL 19
ModelnDCG@10
ColBERT LR 1e-50.6716
ColBERT LR 1.25e-60.6051
Fallon0.6883
DatasetFallonColBERT LR 1e-5ColBERT LR 1.25e-6
NFCorpus0.35430.30790.3264
SciFact0.74750.69520.6943
ArguAna0.63280.69960.6103
SciDocs0.20120.15670.1613
FiQA0.40790.38040.3532

Non-Negative Fallon Ablation

One reviewer suggested that the gains we see from Fallon are not clearly attributable to the ability to represent negatives because we add an additional weight head that might produce some of the benefit. To understand the importance of negative weights on Fallon’s performance we trained a Fallon model on the synthetic task with a ReLU activation on the weights produced, effectively forcing all weights to be $\ge 0$.

The results and additional details are included in Appendix F.1. We also include the result table here for convenience. The results show that the constrained Fallon model is closer in capacity to the ColBERT model in terms of overall performance than it is to the Fallon model. For both In-Domain and Different Vocabulary the performance of constrained Fallon is within 0.04 nDCG@10 of ColBERT. There is a more substantial difference between constrained Fallon and ColBERT on the Negation Only split (0.424 vs 0.008 nDCG@10) although it is still far below standard Fallon (0.788). Our analysis suggests at least two factors that explain constrained Fallon’s improvement in the Negation Only split: (1) 284/2000 queries have every query weight set to $0$ which leaves the ranking as essentially random. As this split has a high number of positives per query even random is fairly strong and helps to improve the metrics over standard ColBERT (2) for many successful queries the weights for the entities are set to 0 leaving only boilerplate tokens like those used for formatting or special tokens. This might allow the model to reduce the number of possible maximums and thus focus information into only a few tokens to enable the negation to be expressed more clearly.

There may also be additional training dynamics that might help the constrained Fallon model to learn a more general process for scoring which helps especially in the Negation Only setting. In any case, Fallon still significantly outperforms the constrained Fallon variant on all metrics across all tasks which supports our hypothesis that allowing negative weights is crucial for high performance on exclusionary tasks.

┌──────────────────┬─────────────────────────┬─────────────────────────┬─────────────────────────┐
│ Model            │ In-Domain               │ Different Vocab.        │ Negation Only           │
├──────────────────┼─────────┬───────┬───────┼─────────┬───────┬───────┼─────────┬───────┬───────┤
│                  │ nDCG@10 │ P@10  │ AP    │ nDCG@10 │ P@10  │ AP    │ nDCG@10 │ P@10  │ AP    │
├──────────────────┼─────────┼───────┼───────┼─────────┼───────┼───────┼─────────┼───────┼───────┤
│ Fallon           │ 0.997   │ 0.893 │ 0.997 │ 1.000   │ 0.955 │ 1.000 │ 0.788   │ 0.789 │ 0.977 │
│ Fallon w/ ReLU   │ 0.974   │ 0.868 │ 0.963 │ 0.633   │ 0.615 │ 0.599 │ 0.424   │ 0.426 │ 0.897 │
│ ColBERT          │ 0.982   │ 0.875 │ 0.943 │ 0.597   │ 0.587 │ 0.530 │ 0.008   │ 0.009 │ 0.766 │
└──────────────────┴─────────┴───────┴───────┴─────────┴───────┴───────┴─────────┴───────┴───────┘

Extending Theorem 3.1 to l2-Normalized Embeddings

One reviewer noted that our current theorems do not reflect how ColBERT models are generally trained, which is with l2-normalized embeddings. To address this we added Appendix B.4 which extends Theorem 3.1 to l2-normalized embeddings. As it is difficult to render here we suggest you look at the theorem in the updated paper. Note we have not had the time to polish it as much as the other theorems, but we plan to for a future final submission.

The theorem shows that we can replicate the inner product of two non-negative sparse vectors up to some fixed positive constant. This implies any ranking produced with the replication would be exactly equivalent to the ranking produced by the sparse inner products. To achieve this we have minimal constraints, the most significant is now the values of the sparse “document” vector must be less than some arbitrary constant B. For any finite set of vectors this will be possible by setting B to the maximum element in the set of vectors. Another change is we need to increase the embedding dimension from three to five. Otherwise the results are essentially the same as the unconstrained construction.

This result shows that: (1) normalized embeddings used with MaxSim can have many of the same properties we prove exist for un-constrained embeddings (2) our theorem is directly applicable to the standard way of training MaxSim models which use l2-normalized embeddings (3) it is possible to replicate sparse inner products up to a constant while keeping the magnitude fixed.

Visualization of Fallon’s Weights

One reviewer suggested we should do additional analysis to understand what Fallon’s weights produce. We added some example visualizations of Fallon’s weights for the different synthetic tasks and some discussion to Appendix F.2.

The visualized weights show that for queries from every subset Fallon learns to assign semantic negations negative weights with the specific negated tokens generally having the most negative score in the query. We also see that non-negated terms have high positive scores. Interestingly, Fallon also learns to assign positive and negative weights to tokens without a clear semantic meaning. For example, a lot of the shared text, such as “People who like”, have a slight negative weight likely because they have minimal usefulness in distinguishing relevant documents.

See the updated paper for the full analysis and visualizations. The main takeaway is Fallon learns to assign strong negative to excluded terms and vice versa to required ones which supports how we expected Fallon to behave.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.