- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Generative Late Interaction Embeddings (GLIE), a post-hoc method that compresses the ~ 1000 patch embeddings per page of late-interaction visual document retrievers to representitive vectors. The key idea is from observation that per-page embeddings lie on the unit sphere and have low intrinsic dimension (5-6 when estimated with TwoNN). Therefore, they first use normalized k-means centroids to build compressed representation that preserves unit norm property. Then, a zero initialized cross attention refiner adjusts the centroids. Finally, a shared decoder expands the refined vectors back to vectors for reranking. At query time, all pages are scored with their stored vectors, and only the top-L candidates are decoded and reranked. Due to this decoder, they can discard full patch embeddings from the storage and only maintain k vectors per page. They also keep the encoder frozen, and only train cross-attention refiner and MLP decoder, which only needs few GPU minutes for training. On ViDoRe v1 and v2, they outperforms raw k-means, token pooling and cluster merging. They also show they can outperform Light-ColPali on small training budget.
Critical strengths
- Practical and well-motivated method. The paper propose solution to practical problem from simple and well-explained observation. Table 2 shows that per-page patch embeddings have low intrinsic dimension consistently across three encoders with different dimensions. This well motivates compressing a patch embeddings with fewer vectors. Overall solution is low-cost. It works post-hoc on a frozen, public encoder, and only require few GPU minutes to train their refiner and decoder.
- Strong results, tested across different backbones. They empirically show their method is efficient with smaller number of vectors compared with other training-free methods. Finally, they also show thorough evaluation across ViDoRe v1 and v2, two different encoders, and different values for .
- Analysis of k-means centroids. The paper proposes k-means centroids as intuitive compressing method. Normalization of these centroids are proposed as free solution to fix underestimation problem of k-means centroids (Proposition 1). Renormalizing cluster means itself is not new (Light-ColPali already does it, as noted in Appendix C), but Proposition 1 gives a useful explanation of why it helps, and the gain shrinks as k grows as predicted.
Critical weaknesses
- Critical baseline missing from the main comparison. Main comparison (Tables 3, 4 and Figure 1) does not compare GLIE against normalized k-means baseline (i.e. GLIE without refiner, decoder and rerank). Against it, GLIE gains +0.052 at k=4 and almost nothing at (Table 7), which changes the reading of the headline claims. Additionally, the use of k-means centroids is also not justified nor compared with alternatives (e.g. spherical k-means).
- Clarify stage 2 trade-off on memory cost vs performance. The benefit of the generative decoder over reranking with the original embeddings is limited to storage on disk and transfer at query time, not memory during search. At stage 2 holding the L shortlisted pages as regenerated vectors or as original patch embeddings requires the same RAM. The decoder therefore lets one discard the original embeddings X entirely, at the cost of lower accuracy than reranking with X (e.g., 0.657 vs. 0.782 for the shortlist oracle at k=4 on ViDoRe v1, Table 3). This is still meaningful when total storage is the bottleneck, but the paper should state this trade-off more clearly, explaining clearly the benefit of the decoder.
- The geometric analysis is not tightly connected to the method or the experiments. The paper measures an intrinsic dimension of about 5 or 6, but no design choice depends on it (e.g., the budget k is not chosen from it). It is also not discussed what this intrinsic dimension implies for the number of vectors needed per page.
- Presentation can be clearer. Same component appears under many different names (e.g. normalized k-means, normalized clustering, spherical anchoring, etc). Limitation of k-means suddenly appear in Section 3.1 without clearly stating it will be used as compression method. Figure 4 starts y-axis at 0.5 and do not show raw numbers for each components, which distort the relative size of the others. Some statements are stronger than it is. e.g., 'training can only move it where the objective improves' is not guaranteed (Stage 1 in DocVQA or InfoVQA is actually degraded compared to training-free stage).
Questions for authors
See weaknesses.
3 comments
Rebuttal
W3. The link between the geometry and the method.
We agree that the intrinsic dimension does not set k. Its role is to explain why a few vectors per page can work at all. The part of the geometry that the method is built on is the sphere. Page vectors have unit norm, so Proposition 1 gives the score that averaging loses. This predicts the gain of spherical anchoring and its decay at large k (+0.093 at k = 4, +0.030 at k = 64), and it is why the decoder keeps its outputs on the sphere. On text, with the late-interaction model ColBERTv2 on two BEIR test sets, the intrinsic dimension is low as well (TwoNN 2.9 on SciFact and 2.8 on NFCorpus, against 25.7 and 25.3 for a Gaussian with the same covariance). Below we show that the prediction of Proposition 1 also holds there.
| nDCG@10, ColBERTv2 | k = 4 | k = 8 | k = 16 | k = 32 | uncompressed |
|---|---|---|---|---|---|
| SciFact, raw k-means | 0.342 | 0.460 | 0.526 | 0.596 | 0.692 |
| SciFact, normalized k-means | 0.450 | 0.544 | 0.585 | 0.633 | |
| NFCorpus, raw k-means | 0.180 | 0.216 | 0.265 | 0.296 | 0.344 |
| NFCorpus, normalized k-means | 0.212 | 0.245 | 0.289 | 0.312 |
Normalizing the k-means centroids improves nDCG@10 at every budget, and the gain shrinks with k as Proposition 1 predicts (+0.108 at k = 4 and +0.037 at k = 32 on SciFact). We will revise the geometry section so that each measurement is followed by the design choice it supports, with the intrinsic dimension presented as motivation.
W4. Presentation.
We agree with all four points and will make these edits.
- One name per object: "spherical anchoring" for the step and "normalized k-means" for the training-free code it produces. "Normalized clustering" is removed.
- Section 3.1 will state that k-means is the compression step before its centroids are analyzed.
- Figure 4 will start its y-axis at 0 and print the value of each component.
- "Training can only move it where the objective improves" becomes: "Training starts from it. On the source holdout the objective can only improve. On a transferred corpus the learned code can fall below it, as the reviewer notes for DocVQA and InfoVQA."
I thank the authors for their rebuttal. The clarifications addressed my main concerns.
I believe clarifying the motivation of the method and its important baselines, as done in the rebuttal, will improve the presentation in the final version (i.e. more about the geometry of page vectors, rather than the generative part as said by the authors).
I am raising my score from 2 (weak reject) to 3 (weak accept).
Rebuttal
We thank the reviewer for the constructive comments. We answer each point below.
W1 (first part). Normalized k-means in the main comparison, and the gain of the learned components.
The reviewer is right that normalized k-means belongs in the main comparison. It was only in the ablation table (Table 7) because the main tables compare complete systems with prior work. We will add it as a row to Tables 3 and 4 and as a curve to Figure 1. The reviewer's reading of the gains is also right: over normalized k-means the learned components add about +0.05 at k = 4, and the gain shrinks toward k = 32, where normalized k-means alone already keeps 94% of the uncompressed score (0.784 of 0.836). The scope of the paper is the geometry of page vectors, the training-free correction that follows from it, and a minimal generative read-out. We believe decoder design is a large topic of its own, so we kept the read-out minimal and limited the paper to two results that show this direction is promising: (1) the design of the decoder matters (Table 10, re-run with three seeds in our reply to reviewer tiqY), and (2) a headroom of 0.13 nDCG@5 at k = 4 remains between GLIE and the original vectors on the same shortlist (Figure 4). The generative read-out is therefore a first step in this direction, and we will reflect this scope more clearly in the revision.
W1 (second part). Why k-means, and the comparison with spherical k-means.
k-means is the clustering that Proposition 1 points to. For unit-norm tokens, the k-means objective equals the total amount by which the centroids fall inside the sphere, which is the quantity that spherical anchoring corrects. We also compared it with other training-free ways to choose the k vectors.
Averaging and then normalizing beats selecting existing tokens at every budget. Spherical k-means projects the centroids at every iteration, where we project once. Run in the same solver, it stays within about 0.01 of normalized k-means at every budget (0.010 higher at k = 4, equal from k = 32). GLIE runs unchanged on a spherical k-means partition, so we adopt the reviewer's suggestion as a variant and will add both comparisons to the appendix.
W2. The trade-off of stage 2.
The reviewer is right that the read-out saves storage and not working memory. Its benefit is a higher score from the same stored bytes, for the case where the original vectors cannot be kept. The same index supports three deployments.
When the original vectors can be kept, re-scoring the shortlist with them is the stronger second stage (the row the paper calls the shortlist oracle). GLIE targets the case where they cannot be kept: at one million pages they take 258 GB, against 1.0 GB for the GLIE index. The index can shrink further: with the stored code quantized to 4 bits per value, without retraining, a page takes 280 bytes (0.28 GB at one million pages) and the score of GLIE does not change (within 0.001, table in our reply to reviewer Q8Xh, W4). Working memory at query time is the same in both cases (20 pages of full-size vectors, about 5 MB), and the read-out adds about 2 ms per query at every corpus size. We will add this table to the paper.