acceptodds
Under review as a conference paper at ICLR 2027

EigenLI: Spectral Approximations to Late Interaction

Abstract

Late-interaction models such as ColBERT achieve strong effectiveness by representing each document with many token-level vectors, but this expressivity leads to large indexing cost, storage footprints and expensive MaxSim scoring. We show that late-interaction representations exhibit an intrinsic low-rank structure: document token embeddings concentrate in a low-dimensional subspace that preserves most of the retrieval signal. Leveraging this observation, we introduce EigenLI, a spectral approximation framework that compresses late-interaction representations via document-specific low-dimensional subspaces. Unlike clustering or pooling methods, EigenLI identifies the dominant eigendirections of each document and uses them to construct reduced interaction representations. Empirically, -EigenLI with outperforms k-means and Ward clustering based pooling methods on ColBERTv2 and AnswerAI-ColBERT-small; GTE-ModernColBERT exhibits a different tradeoff at , where clustering methods perform better. The same spectral construction also yields EigenLI-SV, an ANN-compatible single-vector representation derived from the second-order summary of the reduced structure. Across multiple datasets and all three text models, EigenLI-SV consistently outperforms comparable single-vector surrogates such as MUVERA.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
late-interaction, retrieval, compression

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_gbC6OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes EigenLI, a training-free method for compressing late-interaction representations. Each document is represented by the leading eigenspace of its token-embedding second-moment matrix. Relevance is then measured by the sum of squared projections of query embeddings onto this subspace. A quadratic feature map gives an exactly equivalent single-vector formulation, EigenLI-SV. Experiments cover several text retrieval models on BEIR and a visual document retriever on ViDoRe-v3, with additional analyses of query compression, centering, and quantization.

Critical strengths
  • The construction is simple and its components fit together well. Document compression, the projection score, and the equivalent single-vector representation follow from the same matrix formulation (§§3–3.1). The method can be applied to existing late-interaction embeddings without retraining. The eigenvalue-weighted formulation for query compression is also a useful consequence of this approach.

  • The experiments reveal meaningful differences between embedding models. The evaluation covers multiple text models and visual retrieval, and the paper reports settings where the method works less well, including GTE-ModernColBERT. The gains over clustering-based compression for ColBERTv2 and AnswerAI provide evidence that document-specific subspaces are worth investigating. Reporting the unfavorable cases makes these observations more informative.

Critical weaknesses
  • The relationship to the closest prior work is not developed sufficiently to establish the contribution. Piwowarski et al., Filtering Documents with Subspaces (ECIR 2010), already construct document subspaces from the leading eigenvectors of a document-specific second-moment matrix and score matches using summed projection quadratic forms, equivalently a trace expression (§§2.1–2.2 of that paper). These are mathematically close to EigenLI’s document representation and score. The brief discussion in §2 does not explain the overlap or identify the new elements precisely. The application to contextual token embeddings, the single-vector implementation, and the empirical findings may provide a contribution, but the current presentation leaves too much of that distinction implicit.

  • The approximation analysis and experiments leave the practical retrieval advantage unresolved. Low-rank document structure alone does not establish preservation of MaxSim scores or rankings. For example, changing the signs of document vectors leaves their second-moment matrix unchanged while potentially changing their MaxSim interactions. The bound in §3.2 and Appendix A is one-sided and concerns a related score, so its implications for ranking quality need to be stated carefully. The efficiency evidence has a separate limitation: comparing an 8,256-dimensional encoding with a padded (512 * 128) representation does not establish actual index savings. Moreover, the largely brute-force evaluation does not show the quality–latency tradeoff of candidate generation followed by reranking. The difficulty reproducing MUVERA’s published effectiveness further complicates that comparison.

Questions for authors
  • Could you provide an explicit mathematical comparison with Piwowarski et al. (2010), identifying which parts of the subspace construction and scoring function are inherited, and which algorithmic or empirical contributions are new?

  • What explains the lower reproduced MUVERA results? Please clarify the candidate-generation and reranking protocols. A comparison using the same reranker, a matched retrieval-quality target, and storage measured from actual non-padding document vectors would be particularly helpful in assessing the practical contribution.

  • In the normalization step of the proof on p.14, dividing a projected vector by a norm at most one does not necessarily increase its inner product with the query when that inner product is negative. The zero-projection case also needs handling. Could this step be replaced by a direct Cauchy–Schwarz argument? This appears to be a repairable proof issue rather than a counterexample to the stated bound.

Reviewer_ZHa4OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper is proposing a new method of reducing the number of vectors used to represent a query in multi-vector retrieval models. Their proposed method uses spectral properties of late-interaction representations. It works through leaving the queries unchanged, but mapping the documents vectors to document-specific subspaces. They also proposed a paired scoring function. The methods yields improvement when compared to previous methods speeding up multi-query scoring through reducing the representations dimensionality.

Critical strengths

I like the proposed method, which makes elegant use of linear algebra, and I suspect there may be a deeper mathematical insight underlying it. For some models, the method also achieves substantial improvements over existing pooling approaches.

Critical weaknesses

The improvement seems to vary by the model choice. I think that the paper could be written in a clearer way.

Questions for authors

Clarity:

  • Could the authors introduce MaxSim in one sentence in the introduction?
  • Line 161: could the authors expand on what the differences are?
  • Could the authors add more 'storytelling' to section 3? Right now, as a reader you see the definition of MDM_D and there is no explanation why. There is a lemma 1 below but it's also not narrated properly, (that is in natural, non-mathematical language).
  • On what hardware was the speed up calculated?

General questions:

  • Is there a relationship of the proposed method to eliptical lemma [1] (or covariance matrix estimation)?
  • Is the scoring speed up meaningful in comparison to the speed of a whole retrieval pipeline?
  • Across different tasks, where and to what extent does the advantage of the multi-query models show up?

[1] Lemma 19.4 in https://tor-lattimore.com/downloads/book/book.pdf

Reviewer_9C7AOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose EigenLI, a training-free method for compressing late-interaction document representations using their leading spectral directions and a projection-based scoring function. They also introduce EigenLI-SV, which expresses the score exactly as a single-vector dot product, enabling compatibility with approximate nearest-neighbor (ANN) search. Experiments on BEIR and ViDoRe-v3 show improvements over clustering-based compression, particularly for ColBERTv2. However, results vary across embedding models, and practical end-to-end retrieval efficiency remains unevaluated.

Critical strengths

S1. Simple and useful spectral compression framework.

EigenLI offers a mathematically coherent, training-free alternative to clustering-based compression. Its exact single-vector formulation and query-compression error bound are particularly valuable for practical retrieval applications.

S2. Promising empirical results.

The evaluation covers multiple text and visual retrieval models. For ColBERTv2, EigenLI-32 improves mean relative nDCG@10 by 8.0% over Ward pooling and 15.0% over KMeans++. The paper also transparently reports negative results for certain embedding models.

Critical weaknesses

W1. Limited understanding of model-dependent effectiveness and novelty.

Table 1 shows that EigenLI-32 underperforms Ward across all reported metric–cutoff combinations for GTE-ModernColBERT, including a 10.2% relative decrease in nDCG@10. Section 5 suggests that degenerate low-rank embedding geometry explains this behavior, but provides no direct evidence. Since exploiting low-rank structure is the central motivation, the authors should characterize which geometric properties determine when EigenLI succeeds or fails. Furthermore, the closely related document-subspace construction in Piwowarski et al. (2010) deserve a more precise comparison to clarify the methodological novelty.

W2. Insufficient evaluation of practical retrieval efficiency.

Although Table 4 reports scoring speedups and Table 7 shows faster offline compression than clustering, neither establishes end-to-end query-latency improvements against existing retrieval systems. Similarly, Section 3.1 compares theoretical representation dimensions rather than actual disk usage or index size. Since reducing retrieval costs is one of the central motivation, a controlled comparison against optimized Ward-based retrieval or PLAID, reporting retrieval effectiveness, latency, and index storage, would substantially strengthen the contribution.

Questions for authors

Q1. What geometric properties explain why EigenLI improves ColBERTv2 but underperforms Ward on GTE-ModernColBERT? Can the authors provide supporting spectral analysis or narrow the explanation?

Q2. Can the authors provide end-to-end retrieval latency and actual index-storage comparisons against existing systems?

Comments on the paper

2 comments

Dear Reviewers, as the discussion period is ending soon, can you please respond to the Authors. In particular, can you state which of your critical concerns are still outstanding, which ones were solved by the author response, and whether this affects your final score.

Thank you!

Dear Reviewers ZHa4 and gbC6, once again, can you please clarify whether the author response addresses your concerns, and whether that affects your scores?

Participating in discussion period is a part of the duties of the Reviewers.

Thanks in advance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.