- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces SaMer - a token compression method for multi-vector vision-language retrieval. SaMer compresses projected image tokens into a fixed-sized token set through soft-clustering. The clustering builds upon feature-spatial distance metric and performs a single GMM clustering-like iteration to generate the final centroids. During training, SaMer further utilizes "object-aware merge prior" which augments the distance metric by encouraging tokens to be re-assigned to representatives whose prominent tokens share the same object bounding boxes. This augmentation arguably yields the projector to yield more object-consistent centroids. During training, vision and text encoders are kept frozen so only the projection layer is tuned. SaMer is evaluated on multiple benchmarks (Flickr30K, MSCOCO, ImageCoDe, DocVQA) and shows slight improvements over previous works. Ablation studies include the different method components and number of compressed tokens. In addition, an interpretability study analyzes the concept-level visual evidence which supports the "object-consistent centroids" hypothesis.
Critical strengths
- The paper is clearly written and is easy to follow
- The authors conducted many ablation studies as well as an interpretability analysis which strengthens the results
Critical weaknesses
- The empirical contribution of the method is quite small despite the complexity of the method. Specifically, since the authors chose to finetune SaMer on Flickr30, comparison with other methods on Flickr30k (which were not fine-tuned on Flickr30k) is unfair. Also, SaMer without fine-tuning is slightly worse than previous methods.
- The novelty of the paper is quite limited. The token merging mechanism itself is essentially a single GMM-clustering iteration. While the idea of using detected objects during training is interesting, it has been only investigated through the object-aware merge prior and is not thoroughly investigated.
Questions for authors
Clarification requests:
- How are the locations and features (i,e and ) of the representatives are initialized?
- SaMer is clearly designed for natural images (where natural objects appear), which is reflected both in the distance metric (which utilizes spatial coherence) and in the object merge prior. I find it very weird that improvement on the DocVQA resembles the improvement on benchmarks that consist of natural images (and specifically Flickr30K, which SaMer was trained on). Particularly, why finetuning SaMer helps in that case?
Further questions/suggestions:
- For fair comparison, the authors should report zero-shot performance on both Flickr and COCO. To achieve this, the authors can train SaMer using another dataset (like VisualGenome, which contains bbox annotations as well).
- The authors should compare SaMer against more vanilla baseline methods as well (K-means clustering, universal query tokens) in which the distance metric can be augmented with the object-merging prior as well.
4 comments
Response to Reviewer Rnam (Part 2/3)
2. K-means, Learned Query Pooling, and Object-aware Prior Analysis
The novelty of the paper is quite limited. The token merging mechanism itself is essentially a single GMM-clustering iteration.
While the idea of using detected objects during training is interesting, it has been only investigated through the object-aware merge prior and is not thoroughly investigated.
The authors should compare SaMer against more vanilla baseline methods as well (K-means clustering, universal query tokens) in which the distance metric can be augmented with the object-merging prior as well.
To further examine the novelty of our approach and the contribution of the object-aware prior, we introduced K-means clustering and Learned Query Pooling as additional baselines. We also progressively incorporated spatial information and the object-aware prior into K-means to isolate the contribution of each component. Learned Query Pooling assigns tokens through attention rather than a distance metric, so we apply the prior only to K-means.
| Method | Backbone | Flickr30K R@1 | MSCOCO R@1 |
|---|---|---|---|
| K-means (Feature) | ColPali | 80.3 | 49.4 |
| ColQwen2 | 79.3 | 47.5 | |
| K-means (+ Spatial) | ColPali | 80.3 | 49.5 |
| ColQwen2 | 78.9 | 47.4 | |
| K-means (+ Spatial + Prior) | ColPali | 82.2 | 51.4 |
| ColQwen2 | 79.2 | 47.5 | |
| Learned Query Pooling | ColPali | 78.6 | 46.8 |
| ColQwen2 | 77.7 | 45.3 | |
| SaMer w/o Prior | ColPali | 80.4 | 49.8 |
| ColQwen2 | 79.1 | 47.5 | |
| SaMer w/ Prior | ColPali | 82.4 | 51.6 |
| ColQwen2 | 79.3 | 47.5 |
In addition to retrieval performance, to directly examine whether object-aware merging improves object-instance consistency, we introduce three diagnostic metrics evaluated on the Flickr30K-Entities test set. Let denote the sum of inference-time soft-assignment weights from tokens belonging to object instance to centroid , computed without bounding boxes. We define the foreground mass as the sum of over all object instances, excluding background, and as the normalized instance distribution.
- Instance Purity:
- Cross-instance Mixing:
- Small-object Coverage: the fraction of annotated objects occupying less than 2% of image area that dominate at least one centroid satisfying .
Purity and Mixing are foreground-mass-weighted and pooled across images. Bounding boxes are used only for post-hoc evaluation, with each token position assigned to the smallest enclosing box or background. Small objects without token support remain in the coverage denominator. We discuss both results below.
| Backbone | Method | Instance Purity ↑ | Mixing ↓ | Small Coverage ↑ |
|---|---|---|---|---|
| ColPali | w/o Prior | 60.1 | 51.5 | 23.3 |
| SaMer | 61.4 | 50.1 | 24.7 | |
| ColQwen2 | w/o Prior | 73.5 | 35.4 | 26.5 |
| SaMer | 74.4 | 34.2 | 28.0 |
SaMer (with the prior) outperforms Learned Query Pooling across both backbones, improving R@1 by +3.8 and +4.8 percentage points on Flickr30K and MSCOCO with ColPali, respectively. Compared with K-means, which iterates to convergence, SaMer achieves comparable performance with a single soft-assignment step (80.4 vs. 80.3 R@1 on Flickr30K with ColPali, both without the prior). Adding the object-aware prior to K-means improves R@1 by +1.9 percentage points on both datasets with ColPali, showing that its benefits extend beyond soft assignment. For ColQwen2, however, the prior yields smaller R@1 changes. ColQwen2 produces fewer tokens per image (approximately 238 on Flickr30K and 127 on MSCOCO, Table 5), making K = 64 less aggressive. Its merged tokens also show higher instance consistency without the prior, with Instance Purity of 73.5 versus 60.1 for ColPali and Mixing of 35.4 versus 51.5. This suggests less cross-instance mixing for the prior to address, potentially explaining its smaller impact on retrieval accuracy. Although object-aware K-means and SaMer perform similarly, these results support that our primary contribution lies in object-aware merging for late-interaction retrieval rather than a fundamentally new clustering algorithm.
With the object-aware prior, both ColPali and ColQwen2 show increased Instance Purity and Small-object Coverage, along with reduced Cross-instance Mixing. Although the improvements are modest, the consistent trends across all metrics suggest that the proposed prior helps reduce cross-instance evidence mixing and preserve evidence from small objects. Together with the grounding results in Table 4, these findings provide further evidence for the effectiveness of object-aware merging. We will incorporate these additional analyses into the revised manuscript to clarify the contribution of the proposed method.
Response to Reviewer Rnam (Part 3/3)
3. Centroid Initialization
How are the locations and features of the representatives initialized?
As described in Section 4.1, SaMer initializes its representatives by selecting K seed tokens uniformly across the spatial image-token grid (e.g., an 8 × 8 grid for K = 64). The feature centroid and spatial centroid are initialized using the feature embedding and spatial coordinates of each selected seed token, respectively. SaMer then computes the feature-spatial distances and constructs the final merged tokens through a single soft-assignment step followed by an L2-normalized weighted average. Therefore, no iterative clustering convergence procedure is required. We will clarify this initialization procedure in the revised manuscript.
4. Fine-tuning Effects and Scope of Applicability on DocVQA
I find it very weird that improvement on the DocVQA resembles the improvement on benchmarks that consist of natural images (and specifically Flickr30K, which SaMer was trained on). Particularly, why finetuning SaMer helps in that case?
SaMer primarily focuses on preserving object- and region-level visual evidence in natural-image retrieval. In contrast, DocVQA relies heavily on sparse OCR text and document layout information. We therefore included DocVQA as a supplementary evaluation benchmark to examine the generalizability and scope of applicability of our approach across different domains. In fact, the benefits of fine-tuning on DocVQA are considerably smaller than those on Flickr30K. To examine where these gains come from, we also report the full-token models under the same adaptation.
| Backbone | Dataset | Method | w/o FT R@1 | w/ FT R@1 | ΔR@1 |
|---|---|---|---|---|---|
| ColPali | Flickr30K | Full-token | 77.0 | 83.9 | +6.9 |
| SaMer | 73.5 | 82.4 | +8.9 | ||
| DocVQA | Full-token | 51.0 | 47.6 | −3.4 | |
| SaMer | 44.6 | 45.7 | +1.1 | ||
| ColQwen2 | Flickr30K | Full-token | 73.6 | 78.8 | +5.2 |
| SaMer | 71.3 | 79.3 | +8.0 | ||
| DocVQA | Full-token | 53.2 | 51.3 | −1.9 | |
| SaMer | 47.2 | 48.8 | +1.6 |
Full-token w/o FT and SaMer values are taken from Table 1 of the paper. Full-token w/ FT denotes the full-token model trained with the same projection-only adaptation as SaMer.
Fine-tuning improves R@1 on Flickr30K by +8.9 percentage points for ColPali and +8.0 for ColQwen2, whereas the improvements on DocVQA are limited to +1.1 and +1.6 points. Notably, the same adaptation decreases DocVQA performance for the full-token models (51.0 → 47.6 for ColPali and 53.2 → 51.3 for ColQwen2), suggesting a domain mismatch between natural-image training and document retrieval. The modest improvements of SaMer on DocVQA may instead reflect compression-aware adaptation, which helps align the projection space with merged tokens (Section 4.3). Consistently, the object-aware prior provides little additional benefit on DocVQA (45.9 without vs. 45.7 with the prior for ColPali). Nevertheless, SaMer remains below the adapted full-token models on DocVQA, highlighting the limitations of aggressive compression for OCR- and layout-intensive retrieval. We will clarify DocVQA's role as a supplementary evaluation benchmark and discuss these limitations in the revised manuscript.
Gentle Reminder to Reviewer Rnam
Thank you again for your thoughtful review and valuable feedback. As the discussion period is approaching its end, we would like to kindly ask whether our responses and the additional experiments in the revised manuscript have addressed your concerns. If you have any remaining questions or concerns, we would be happy to provide further clarification.
Response to Reviewer Rnam (Part 1/3)
We sincerely appreciate the reviewer's valuable and constructive feedback. To address the concerns regarding fair comparisons, methodological novelty, centroid initialization, and the effects of fine-tuning on DocVQA, we conducted additional experiments and analyses.
1. Zero-shot Generalization with Visual Genome
We agree that fair comparisons require a clear separation between training and evaluation data. To address this concern, we trained SaMer on Visual Genome (VG) instead of Flickr30K-Entities and evaluated it on Flickr30K and MSCOCO without additional fine-tuning. Importantly, we removed all images overlapping with the evaluation sets from the Visual Genome training data to prevent data leakage.
Compared with the original full-token zero-shot retrievers, VG-trained SaMer improves R@1 by +2.8 and +3.3 percentage points on Flickr30K and MSCOCO with ColPali, and by +3.1 and +4.5 percentage points with ColQwen2, respectively. These results demonstrate that SaMer's improvements are not solely dependent on in-domain fine-tuning on Flickr30K, and that strong retrieval performance can be achieved when training on an external dataset. We will include these results in the revised manuscript to further clarify the generalizability and fairness of our evaluation.