acceptodds
Under review as a conference paper at ICLR 2027

Do All Visual Tokens Matter Equally? Object-Evidence Preserving Token Merging for Vision-Language Retrieval

Abstract

Multi-vector vision-language retrieval preserves fine-grained visual evidence through maximum-similarity late interaction, but dense image-side tokens make storage and scoring expensive. Existing token compression methods reduce this cost, yet they can remove or collapse object- and region-level evidence that future query tokens may need to select. We propose SaMer, an object-aware token merging framework that compresses image-side post-projector tokens into representative centroids while preserving the original late-interaction interface. SaMer uses object annotations only during training as a merge prior to discourage cross-instance mixing, requires no ground-truth bounding boxes or detectors at inference time, and adapts only the shared projection layer with frozen vision and language backbones. With , SaMer removes more than 93% of image-side tokens and reduces ColPali storage by , while improving R@1 on Flickr30K and MSCOCO. These gains arise because object-aware merging preserves query-selectable object evidence that pruning or feature-only pooling can remove or collapse. SaMer also outperforms compression baselines and shows stronger phrase-level grounding, suggesting that efficient multi-vector retrieval depends not only on reducing token count, but on preserving the evidence future query tokens need to select.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision-Language Retrieval, Visual Token Compression, Object-Aware Token Merging, Late Interaction, Efficient Multimodal Retrieval

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_RnamOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SaMer - a token compression method for multi-vector vision-language retrieval. SaMer compresses projected image tokens into a fixed-sized token set through soft-clustering. The clustering builds upon feature-spatial distance metric and performs a single GMM clustering-like iteration to generate the final centroids. During training, SaMer further utilizes "object-aware merge prior" which augments the distance metric by encouraging tokens to be re-assigned to representatives whose prominent tokens share the same object bounding boxes. This augmentation arguably yields the projector to yield more object-consistent centroids. During training, vision and text encoders are kept frozen so only the projection layer is tuned. SaMer is evaluated on multiple benchmarks (Flickr30K, MSCOCO, ImageCoDe, DocVQA) and shows slight improvements over previous works. Ablation studies include the different method components and number of compressed tokens. In addition, an interpretability study analyzes the concept-level visual evidence which supports the "object-consistent centroids" hypothesis.

Critical strengths
  1. The paper is clearly written and is easy to follow
  2. The authors conducted many ablation studies as well as an interpretability analysis which strengthens the results
Critical weaknesses
  1. The empirical contribution of the method is quite small despite the complexity of the method. Specifically, since the authors chose to finetune SaMer on Flickr30, comparison with other methods on Flickr30k (which were not fine-tuned on Flickr30k) is unfair. Also, SaMer without fine-tuning is slightly worse than previous methods.
  2. The novelty of the paper is quite limited. The token merging mechanism itself is essentially a single GMM-clustering iteration. While the idea of using detected objects during training is interesting, it has been only investigated through the object-aware merge prior and is not thoroughly investigated.
Questions for authors

Clarification requests:

  1. How are the locations and features (i,e μk\mu_k and sks_k) of the representatives are initialized?
  2. SaMer is clearly designed for natural images (where natural objects appear), which is reflected both in the distance metric (which utilizes spatial coherence) and in the object merge prior. I find it very weird that improvement on the DocVQA resembles the improvement on benchmarks that consist of natural images (and specifically Flickr30K, which SaMer was trained on). Particularly, why finetuning SaMer helps in that case?

Further questions/suggestions:

  1. For fair comparison, the authors should report zero-shot performance on both Flickr and COCO. To achieve this, the authors can train SaMer using another dataset (like VisualGenome, which contains bbox annotations as well).
  2. The authors should compare SaMer against more vanilla baseline methods as well (K-means clustering, universal query tokens) in which the distance metric can be augmented with the object-merging prior as well.
4 comments
AuthorsOpenReview ↗

Response to Reviewer Rnam (Part 1/3)

We sincerely appreciate the reviewer's valuable and constructive feedback. To address the concerns regarding fair comparisons, methodological novelty, centroid initialization, and the effects of fine-tuning on DocVQA, we conducted additional experiments and analyses.

1. Zero-shot Generalization with Visual Genome

Since the authors chose to finetune SaMer on Flickr30, comparison with other methods on Flickr30k (which were not fine-tuned on Flickr30k) is unfair.

For fair comparison, the authors should report zero-shot performance on both Flickr and COCO.

We agree that fair comparisons require a clear separation between training and evaluation data. To address this concern, we trained SaMer on Visual Genome (VG) instead of Flickr30K-Entities and evaluated it on Flickr30K and MSCOCO without additional fine-tuning. Importantly, we removed all images overlapping with the evaluation sets from the Visual Genome training data to prevent data leakage.

BackboneDatasetFull-token Zero-shot R@1VG-trained SaMer R@1R@5nDCG@10
ColPaliFlickr30K77.079.894.488.7
MSCOCO47.450.775.266.6
ColQwen2Flickr30K73.676.793.086.9
MSCOCO43.347.872.363.9

Compared with the original full-token zero-shot retrievers, VG-trained SaMer improves R@1 by +2.8 and +3.3 percentage points on Flickr30K and MSCOCO with ColPali, and by +3.1 and +4.5 percentage points with ColQwen2, respectively. These results demonstrate that SaMer's improvements are not solely dependent on in-domain fine-tuning on Flickr30K, and that strong retrieval performance can be achieved when training on an external dataset. We will include these results in the revised manuscript to further clarify the generalizability and fairness of our evaluation.

AuthorsOpenReview ↗

Response to Reviewer Rnam (Part 2/3)

2. K-means, Learned Query Pooling, and Object-aware Prior Analysis

The novelty of the paper is quite limited. The token merging mechanism itself is essentially a single GMM-clustering iteration.

While the idea of using detected objects during training is interesting, it has been only investigated through the object-aware merge prior and is not thoroughly investigated.

The authors should compare SaMer against more vanilla baseline methods as well (K-means clustering, universal query tokens) in which the distance metric can be augmented with the object-merging prior as well.

To further examine the novelty of our approach and the contribution of the object-aware prior, we introduced K-means clustering and Learned Query Pooling as additional baselines. We also progressively incorporated spatial information and the object-aware prior into K-means to isolate the contribution of each component. Learned Query Pooling assigns tokens through attention rather than a distance metric, so we apply the prior only to K-means.

MethodBackboneFlickr30K R@1MSCOCO R@1
K-means (Feature)ColPali80.349.4
ColQwen279.347.5
K-means (+ Spatial)ColPali80.349.5
ColQwen278.947.4
K-means (+ Spatial + Prior)ColPali82.251.4
ColQwen279.247.5
Learned Query PoolingColPali78.646.8
ColQwen277.745.3
SaMer w/o PriorColPali80.449.8
ColQwen279.147.5
SaMer w/ PriorColPali82.451.6
ColQwen279.347.5

In addition to retrieval performance, to directly examine whether object-aware merging improves object-instance consistency, we introduce three diagnostic metrics evaluated on the Flickr30K-Entities test set. Let mkbm_{kb} denote the sum of inference-time soft-assignment weights from tokens belonging to object instance bb to centroid kk, computed without bounding boxes. We define the foreground mass FkF_k as the sum of mkbm_{kb} over all object instances, excluding background, and pkb=mkb/Fkp_{kb} = m_{kb}/F_k as the normalized instance distribution.

  • Instance Purity: Purity=∑kFkmax⁡bpkb∑kFk\mathrm{Purity}=\frac{\sum_k F_k\max_b p_{kb}}{\sum_k F_k}
  • Cross-instance Mixing: Mixing=∑kFk(1−∑bpkb2)∑kFk\mathrm{Mixing}=\frac{\sum_k F_k(1-\sum_b p_{kb}^2)}{\sum_k F_k}
  • Small-object Coverage: the fraction of annotated objects occupying less than 2% of image area that dominate at least one centroid satisfying Fk>mk,bgF_k>m_{k,\mathrm{bg}}.

Purity and Mixing are foreground-mass-weighted and pooled across images. Bounding boxes are used only for post-hoc evaluation, with each token position assigned to the smallest enclosing box or background. Small objects without token support remain in the coverage denominator. We discuss both results below.

BackboneMethodInstance Purity ↑Mixing ↓Small Coverage ↑
ColPaliw/o Prior60.151.523.3
SaMer61.450.124.7
ColQwen2w/o Prior73.535.426.5
SaMer74.434.228.0

SaMer (with the prior) outperforms Learned Query Pooling across both backbones, improving R@1 by +3.8 and +4.8 percentage points on Flickr30K and MSCOCO with ColPali, respectively. Compared with K-means, which iterates to convergence, SaMer achieves comparable performance with a single soft-assignment step (80.4 vs. 80.3 R@1 on Flickr30K with ColPali, both without the prior). Adding the object-aware prior to K-means improves R@1 by +1.9 percentage points on both datasets with ColPali, showing that its benefits extend beyond soft assignment. For ColQwen2, however, the prior yields smaller R@1 changes. ColQwen2 produces fewer tokens per image (approximately 238 on Flickr30K and 127 on MSCOCO, Table 5), making K = 64 less aggressive. Its merged tokens also show higher instance consistency without the prior, with Instance Purity of 73.5 versus 60.1 for ColPali and Mixing of 35.4 versus 51.5. This suggests less cross-instance mixing for the prior to address, potentially explaining its smaller impact on retrieval accuracy. Although object-aware K-means and SaMer perform similarly, these results support that our primary contribution lies in object-aware merging for late-interaction retrieval rather than a fundamentally new clustering algorithm.

With the object-aware prior, both ColPali and ColQwen2 show increased Instance Purity and Small-object Coverage, along with reduced Cross-instance Mixing. Although the improvements are modest, the consistent trends across all metrics suggest that the proposed prior helps reduce cross-instance evidence mixing and preserve evidence from small objects. Together with the grounding results in Table 4, these findings provide further evidence for the effectiveness of object-aware merging. We will incorporate these additional analyses into the revised manuscript to clarify the contribution of the proposed method.

AuthorsOpenReview ↗

Response to Reviewer Rnam (Part 3/3)

3. Centroid Initialization

How are the locations and features of the representatives initialized?

As described in Section 4.1, SaMer initializes its representatives by selecting K seed tokens uniformly across the spatial image-token grid (e.g., an 8 × 8 grid for K = 64). The feature centroid μk\mu_k and spatial centroid sks_k are initialized using the feature embedding and spatial coordinates of each selected seed token, respectively. SaMer then computes the feature-spatial distances and constructs the final KK merged tokens through a single soft-assignment step followed by an L2-normalized weighted average. Therefore, no iterative clustering convergence procedure is required. We will clarify this initialization procedure in the revised manuscript.


4. Fine-tuning Effects and Scope of Applicability on DocVQA

I find it very weird that improvement on the DocVQA resembles the improvement on benchmarks that consist of natural images (and specifically Flickr30K, which SaMer was trained on). Particularly, why finetuning SaMer helps in that case?

SaMer primarily focuses on preserving object- and region-level visual evidence in natural-image retrieval. In contrast, DocVQA relies heavily on sparse OCR text and document layout information. We therefore included DocVQA as a supplementary evaluation benchmark to examine the generalizability and scope of applicability of our approach across different domains. In fact, the benefits of fine-tuning on DocVQA are considerably smaller than those on Flickr30K. To examine where these gains come from, we also report the full-token models under the same adaptation.

BackboneDatasetMethodw/o FT R@1w/ FT R@1ΔR@1
ColPaliFlickr30KFull-token77.083.9+6.9
SaMer73.582.4+8.9
DocVQAFull-token51.047.6−3.4
SaMer44.645.7+1.1
ColQwen2Flickr30KFull-token73.678.8+5.2
SaMer71.379.3+8.0
DocVQAFull-token53.251.3−1.9
SaMer47.248.8+1.6

Full-token w/o FT and SaMer values are taken from Table 1 of the paper. Full-token w/ FT denotes the full-token model trained with the same projection-only adaptation as SaMer.

Fine-tuning improves R@1 on Flickr30K by +8.9 percentage points for ColPali and +8.0 for ColQwen2, whereas the improvements on DocVQA are limited to +1.1 and +1.6 points. Notably, the same adaptation decreases DocVQA performance for the full-token models (51.0 → 47.6 for ColPali and 53.2 → 51.3 for ColQwen2), suggesting a domain mismatch between natural-image training and document retrieval. The modest improvements of SaMer on DocVQA may instead reflect compression-aware adaptation, which helps align the projection space with merged tokens (Section 4.3). Consistently, the object-aware prior provides little additional benefit on DocVQA (45.9 without vs. 45.7 with the prior for ColPali). Nevertheless, SaMer remains below the adapted full-token models on DocVQA, highlighting the limitations of aggressive compression for OCR- and layout-intensive retrieval. We will clarify DocVQA's role as a supplementary evaluation benchmark and discuss these limitations in the revised manuscript.

AuthorsOpenReview ↗

Gentle Reminder to Reviewer Rnam

Thank you again for your thoughtful review and valuable feedback. As the discussion period is approaching its end, we would like to kindly ask whether our responses and the additional experiments in the revised manuscript have addressed your concerns. If you have any remaining questions or concerns, we would be happy to provide further clarification.

Reviewer_NvXtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The manuscript addresses the storage and scoring cost of multi-vector vision-language retrieval, where each image is indexed as roughly one thousand post-projector tokens and scored by maximum-similarity late interaction. Results compare SaMer at K = 64 with single-vector, VLM, multi-vector, and compression baselines on Flickr30K, MSCOCO, ImageCoDe, and DocVQA with ColPali and ColQwen2, plus grounding metrics and a storage and throughput analysis.

Critical strengths
  1. Section 3.2 separates post-projector index compression from encoder-side reduction and makes the query-dependence of patch importance explicit, the correct distinction for late-interaction retrieval.
  2. The efficiency accounting is internally consistent: 1030 to 64 tokens at FP16 with D = 128 gives 263.7 GB to 16.4 GB per million images, the stated 16.09 times, and MaxSim operations fall from 36.87 to 2.29 trillion with QPS rising from 625.5 to 2716.3 on Flickr30K.
  3. The grounding evaluation tests the proposed mechanism rather than accuracy alone, reporting three complementary quantities for every baseline at a matched budget.
  4. Evaluation spans two retrievers and four domains, including a deliberate boundary case where the object-centric assumption should fail, which is more informative than four natural-image benchmarks.
Critical weaknesses
  1. The headline comparison confounds compression with in-domain adaptation. SaMer without adaptation reaches 73.5 and 45.1 R@1, below uncompressed ColPali at 77.0 and 47.4, so compression alone loses 3.5 and 2.3 points, while adaptation lifts it to 82.4 and 51.6. Appendix D shows the same pattern in a control the authors already ran: random pruning with identical projection-only adaptation reaches 79.9 and 48.1, above uncompressed ColPali on both benchmarks. Most of the gain over full tokens is therefore attributable to three epochs of in-domain projection training on Flickr30K-Entities, not to preserving object evidence. A full-token ColPali arm with the same adaptation budget is absent from every table and is the control that decides the claim.
  2. DocVQA contradicts the thesis and is under-reported. Recall@1 falls from 51.0 to 45.7 and Recall@5 from 66.5 to 60.5, a relative loss larger than any ImageCoDe gain, in the one domain where evidence is sparse text rather than redundant patches. Section 5.3 calls this a boundary case and the Abstract omits it, yet it is precisely the failure the paper argues against, since merging cannot recover OCR evidence averaged away. The regression should be stated where the claims are made, with a K = 128 or text-preserving fallback test.
  3. No table reports variance, seeds, or paired tests, although adaptation involves initialization, batching, and a soft-assignment temperature.
Questions for authors

N/A

4 comments
AuthorsOpenReview ↗

Response to Reviewer NvXt (Part 1/3)

We sincerely appreciate the reviewer's valuable and constructive feedback. To address the concerns regarding fair comparisons, performance degradation on DocVQA, and statistical reliability, we conducted additional experiments and analyses.

1. Separating the Effects of Full-token Retrieval and Adaptation

The headline comparison confounds compression with in-domain adaptation. SaMer without adaptation reaches 73.5 and 45.1 R@1, below uncompressed ColPali at 77.0 and 47.4, so compression alone loses 3.5 and 2.3 points, while adaptation lifts it to 82.4 and 51.6. Appendix D shows the same pattern in a control the authors already ran: random pruning with identical projection-only adaptation reaches 79.9 and 48.1, above uncompressed ColPali on both benchmarks. Most of the gain over full tokens is therefore attributable to three epochs of in-domain projection training on Flickr30K-Entities, not to preserving object evidence. A full-token ColPali arm with the same adaptation budget is absent from every table and is the control that decides the claim.

We agree that a fair comparison under identical adaptation conditions is essential. To address this concern, we additionally evaluated full-token ColPali and ColQwen2 with the same projection-only fine-tuning as SaMer, using identical training data, optimizer, and training schedule.

(a) ColPali

MetricMethodFlickr30KMSCOCOImageCoDeDocVQA
R@1Full + FT83.952.66.047.6
SaMer82.451.65.945.7
R@5Full + FT96.476.418.365.6
SaMer96.375.517.060.5
nDCG@10Full + FT91.667.914.959.4
SaMer90.967.114.455.6

(b) ColQwen2

MetricMethodFlickr30KMSCOCOImageCoDeDocVQA
R@1Full + FT78.846.04.951.3
SaMer79.347.55.148.8
R@5Full + FT93.870.615.565.2
SaMer94.772.316.262.5
nDCG@10Full + FT88.262.012.760.9
SaMer88.763.512.958.1

SaMer results are from Table 1 of the original manuscript (K = 64), while Full + FT uses the same projection-only adaptation settings.

These results confirm the reviewer's observation that fine-tuning contributes to the improvements over the original full-token retrievers. However, SaMer maintains retrieval performance close to that of full-token models even under identical fine-tuning conditions. For ColPali, the R@1 differences on Flickr30K and MSCOCO are only −1.5 and −1.0 percentage points, respectively. For ColQwen2, SaMer performs on par with or slightly above Full + FT on the same benchmarks. Importantly, SaMer reduces ColPali's image tokens from 1,030 to 64, achieving approximately 16.1× reductions in both storage and MaxSim operations while maintaining strong retrieval performance.

Moreover, Table 4 shows that adding the object-aware merge prior improves R@1 by +2.0 percentage points on Flickr30K and +1.8 percentage points on MSCOCO. All variants in Table 4 are trained with the same projection-only adaptation as SaMer (same training data, optimizer, and schedule), so this difference reflects the merge rule rather than adaptation; we will state this explicitly in the revised manuscript. The same matched comparison also shows that the prior improves grounding: BoxMass increases from 47.8 to 54.2 and CoverageIoU from 13.1 to 16.4, indicating that the accuracy gain is accompanied by phrase relevance more concentrated within the annotated object regions. Under the same adaptation conditions, SaMer also outperforms random pruning, as discussed in more detail below. This indicates that the improvements cannot be attributed solely to applying adaptation to an arbitrary compression method.

Therefore, our main contribution is not that compression itself improves accuracy, but that SaMer effectively preserves query-relevant visual evidence and retrieval performance under aggressive token compression. We will clarify this distinction in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer NvXt (Part 2/3)

2. DocVQA Performance Degradation and Scope of Applicability

DocVQA contradicts the thesis and is under-reported. Recall@1 falls from 51.0 to 45.7 and Recall@5 from 66.5 to 60.5, a relative loss larger than any ImageCoDe gain, in the one domain where evidence is sparse text rather than redundant patches. Section 5.3 calls this a boundary case and the Abstract omits it, yet it is precisely the failure the paper argues against, since merging cannot recover OCR evidence averaged away. The regression should be stated where the claims are made, with a K = 128 or text-preserving fallback test.

We agree that the performance degradation on DocVQA should be reported and discussed more explicitly.

SaMer is primarily designed to preserve object- and region-level visual evidence in natural-image retrieval. In contrast, DocVQA heavily relies on sparse OCR text and document layout information, where object-centric merging may discard important textual evidence at a small token budget. Below, we examine whether this loss is inherent to merging or depends on the budget.

Following the reviewer's suggestion, we conducted additional DocVQA experiments with larger token budgets of K=128 and K=256.

BackboneMetricFull + FTSaMer (K=64)SaMer (K=128)SaMer (K=256)
ColPaliR@147.643.8 ± 1.646.1 ± 0.246.8 ± 0.1
Δ\Delta R@1—−3.8−1.5−0.8
R@565.660.8 ± 0.363.1 ± 0.464.2 ± 1.2
nDCG@1059.455.1 ± 0.557.0 ± 0.358.5 ± 0.1
ColQwen2R@151.348.2 ± 0.651.2 ± 0.351.7 ± 0.3
Δ\Delta R@1—−3.1−0.1+0.4
R@565.262.7 ± 0.265.2 ± 0.464.7 ± 0.4
nDCG@1060.958.4 ± 0.361.0 ± 0.361.4 ± 0.1

DocVQA results are reported as Mean ± Std over three random seeds. Full + FT denotes the full-token model with the same projection-only adaptation (single run, see comment 1 of this response). The paper reported a single run at K = 64 (45.7 R@1), and we will report seed means in the revision.

Increasing the token budget improves ColPali's R@1 from 43.8 to 46.8 (+3.0 percentage points) and ColQwen2's R@1 from 48.2 to 51.7 (+3.6 percentage points). These results suggest that retaining more tokens can help mitigate the loss of sparse textual evidence in document retrieval.

Interestingly, even the full-token models exhibit performance degradation on DocVQA after Flickr30K-based fine-tuning (see the Full + FT results in comment 1 of this response): ColPali's R@1 decreases from 51.0 to 47.6, while ColQwen2's decreases from 53.2 to 51.3. This suggests that the degradation on DocVQA may arise not only from token compression but also from the domain mismatch between natural-image adaptation and document retrieval.

Compared with Full + FT, which shares the same adaptation, the gap largely closes as K increases (last column of the table above): ColQwen2 at K = 128 matches Full + FT (51.2 vs. 51.3 R@1; 65.2 vs. 65.2 R@5), and ColPali at K = 256 is within 0.8 R@1 points (46.8 vs. 47.6) while still compressing its 1,030 tokens by about 4×. These results suggest that a larger token budget can mitigate retrieval performance degradation in document retrieval, although preserving sparse OCR evidence remains challenging under aggressive compression. We leave text-aware merging, such as protecting OCR-dense regions from aggressive merging, to future work. We will explicitly discuss the DocVQA performance degradation and clarify the intended scope of SaMer in the Abstract and Section 5.3.

AuthorsOpenReview ↗

Response to Reviewer NvXt (Part 3/3)

3. Statistical Reliability and Significance Analysis

No table reports variance, seeds, or paired tests, although adaptation involves initialization, batching, and a soft-assignment temperature.

We appreciate the reviewer's suggestion to strengthen the statistical reliability of our results. To address this concern, we additionally evaluated SaMer using three different random seeds (0, 42, and 100) for both training and inference, and report Mean ± Std. We also conducted paired bootstrap analysis with 95% confidence intervals to assess the reliability of performance differences between key methods.

The results across three random seeds are summarized below.

BackboneMetricFlickr30KMSCOCOImageCoDeDocVQA
ColPali + SaMerR@183.1 ± 0.651.9 ± 0.35.8 ± 0.143.8 ± 1.6
R@596.4 ± 0.175.7 ± 0.217.1 ± 0.160.8 ± 0.3
nDCG@1091.2 ± 0.367.4 ± 0.214.4 ± 0.055.1 ± 0.5
ColQwen2 + SaMerR@178.9 ± 0.347.0 ± 0.44.9 ± 0.248.2 ± 0.6
R@594.4 ± 0.371.8 ± 0.515.8 ± 0.462.7 ± 0.2
nDCG@1088.4 ± 0.363.2 ± 0.312.9 ± 0.158.4 ± 0.3

The three-seed evaluation assesses training variability, while the paired bootstrap analysis uses the original single-run SaMer results and their corresponding baseline predictions to evaluate the reliability of performance differences.

Comparison (ColPali)DatasetΔ\Delta R@1 (pp)95% CI
SaMer vs Full-token + FTFlickr30K−1.5[−2.1, −0.9]
SaMer vs Full-token + FTMSCOCO−1.0[−1.3, −0.6]
SaMer vs w/o Object Prior + FTFlickr30K+2.0[+1.8, +2.3]
SaMer vs w/o Object Prior + FTMSCOCO+1.8[+1.7, +1.9]
SaMer vs Random Pruning + FTFlickr30K+2.5[+1.6, +3.4]
SaMer vs Random Pruning + FTMSCOCO+3.5[+2.9, +4.0]

To assess statistical reliability, we performed paired bootstrap resampling on matched evaluation queries and computed 95% confidence intervals for the R@1 differences. Compared with Full-token + FT, SaMer exhibits relatively small accuracy reductions of −1.5 percentage points on Flickr30K (95% CI: [−2.1, −0.9]) and −1.0 percentage points on MSCOCO (95% CI: [−1.3, −0.6]). In contrast, adding the object-aware merge prior improves R@1 by +2.0 percentage points on Flickr30K (95% CI: [+1.8, +2.3]) and +1.8 percentage points on MSCOCO (95% CI: [+1.7, +1.9]), with both confidence intervals excluding zero.

Furthermore, compared with Random Pruning under identical fine-tuning conditions, SaMer improves R@1 by +2.5 percentage points on Flickr30K (95% CI: [+1.6, +3.4]) and +3.5 percentage points on MSCOCO (95% CI: [+2.9, +4.0]). Both confidence intervals exclude zero, providing further evidence that the proposed merging strategy preserves retrieval-relevant visual evidence more effectively than random pruning.

We will incorporate the three-seed Mean ± Std results and paired bootstrap analysis into the revised manuscript to provide a clearer and more reliable assessment of the stability and effectiveness of SaMer. We will also conduct an additional sensitivity analysis of the soft-assignment temperature τs\tau_s and report its impact on retrieval performance in the revised manuscript.

AuthorsOpenReview ↗

Gentle Reminder to Reviewer NvXt

Thank you again for your thoughtful review and valuable feedback. As the discussion period is approaching its end, we would like to kindly ask whether our responses and the additional experiments in the revised manuscript have addressed your concerns. If you have any remaining questions or concerns, we would be happy to provide further clarification.

Comments on the paper

6 comments
AuthorsOpenReview ↗

Response to Reviewer SyTA (Part 1/4)

We sincerely appreciate the reviewer's valuable and constructive feedback. To address the concerns regarding fair comparisons, the contribution and generalizability of the object-aware prior, novelty, and practical retrieval efficiency, we conducted additional experiments and analyses.

1. Separating the Effects of Full-token Retrieval and Adaptation

The comparison with full-token retrieval confounds compression with adaptation. SaMer(w/FT) is trained with retrieval supervision, while the headline full ColPali and ColQwen2 rows are zero-shot. Table 2 adapts compression baselines, but it still does not include a full-token projection-only adapted model. Without that upper bound, the reported improvement over the original retriever cannot establish an accuracy benefit from merged tokens.

Can the authors report full-token ColPali and ColQwen2 results under exactly the same projection-only adaptation, optimizer, schedule, and training data?

We agree that a fair comparison under identical adaptation conditions is essential. To address this concern, we additionally evaluated full-token ColPali and ColQwen2 with the same projection-only fine-tuning as SaMer, using identical training data, optimizer, and training schedule.

BackboneDatasetFull + FT R@1SaMer R@1Full + FT R@5SaMer R@5Full + FT nDCG@10SaMer nDCG@10
ColPaliFlickr30K83.982.496.496.391.690.9
ColPaliMSCOCO52.651.676.475.567.967.1
ColPaliImageCoDe6.05.918.317.014.914.4
ColPaliDocVQA47.645.765.660.559.455.6
ColQwen2Flickr30K78.879.393.894.788.288.7
ColQwen2MSCOCO46.047.570.672.362.063.5
ColQwen2ImageCoDe4.95.115.516.212.712.9
ColQwen2DocVQA51.348.865.262.560.958.1

SaMer results are taken from Table 1 of the original manuscript with K=64.

These results confirm the reviewer's observation that fine-tuning contributes to the improvements over the original full-token retrievers. Nevertheless, SaMer maintains retrieval performance close to that of full-token models under identical fine-tuning conditions. For ColPali, the R@1 differences on Flickr30K and MSCOCO are −1.5 and −1.0 percentage points, respectively. For ColQwen2, SaMer performs on par with or slightly above Full + FT on the same benchmarks. With this upper bound, the adapted H-Pool and SAP baselines in Table 2 fall 3.7–4.7 R@1 points below Full + FT on Flickr30K and 2.9–5.3 points on MSCOCO for ColPali, whereas SaMer stays within 1.5 and 1.0 points. Importantly, SaMer achieves this while reducing ColPali’s image tokens from 1,030 to 64, yielding approximately 16.1× reductions in both storage and MaxSim operations.

Therefore, our main contribution is not that compression itself improves accuracy, but that SaMer effectively preserves query-relevant visual evidence and retrieval performance under aggressive token compression, as further supported by the controlled ablation and grounding analyses discussed below. We will clarify this distinction in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer SyTA (Part 2/4)

2. Contribution of the Object-aware Prior and Cross-domain Generalization

The strongest gains are on in-domain Flickr30K, while ImageCoDe and DocVQA gains are small or negative for ColPali. It is unclear how much of the result transfers when object annotations are unavailable during adaptation.

What happens when SaMer is adapted without object annotations but with the same projection-only supervision? This would isolate gains from the prior itself.

We appreciate the reviewer's suggestion to further examine the contribution and generalizability of the object-aware prior. First, Table 4 provides a controlled ablation in which only the prior is removed, while maintaining identical projection-only adaptation and retrieval supervision.

MethodFlickr30K R@1MSCOCO R@1BoxMassRegionHitCoverageIoU
w/o Object Prior80.449.847.868.013.1
SaMer (w/ Prior)82.451.654.268.316.4

Adding the prior improves R@1 by +2.0 percentage points on Flickr30K and +1.8 on MSCOCO, while increasing BoxMass from 47.8 to 54.2 and CoverageIoU from 13.1 to 16.4. These results demonstrate its contribution beyond projection-only adaptation.

To further evaluate annotation-free adaptation, we trained two models on Flickr30K-Entities with and without the object-aware prior, then applied identical projection-only fine-tuning on MSCOCO without bounding-box annotations for either model. No bounding boxes were required at inference.

AdaptationDatasetw/o Prior R@1w/ Prior R@1ΔR@1ΔR@5ΔnDCG@10
Flickr30K OnlyFlickr30K80.482.4+2.0+0.8+1.2
MSCOCO49.851.6+1.8+1.6+1.6
ImageCoDe5.95.9+0.0+0.2+0.2
DocVQA45.945.7−0.2+0.70.0
+ MSCOCO FT (No bbox)Flickr30K79.180.9+1.9+0.8+1.1
MSCOCO53.254.8+1.7+1.6+1.5
ImageCoDe5.75.6−0.1+0.1+0.0
DocVQA43.844.3+0.4−0.6+0.2

Δ\Delta = w/ Prior − w/o Prior (percentage points), calculated before rounding. All experiments use K=64.

The prior's advantage remains nearly unchanged after annotation-free adaptation, from +2.0 to +1.9 percentage points on Flickr30K and +1.8 to +1.7 on MSCOCO. This suggests a practical approach: training with the object-aware prior on annotated source data, followed by projection-only adaptation without bounding boxes. Grounding results show improvements in RegionHit (68.3 → 79.1) and CoverageIoU (16.4 → 22.1), although BoxMass decreases (54.2 → 31.3), indicating a trade-off between spatial localization and object-specific evidence concentration. Meanwhile, the prior provides limited gains on ImageCoDe and DocVQA, where domain mismatch also contributes to performance degradation, as reflected in the full-token ColPali results (51.0 → 47.6 R@1 on DocVQA). We will clarify these findings and domain-specific limitations in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer SyTA (Part 3/4)

3. Novelty and Distinction from Existing Methods

The contribution is incremental and the cross-domain evidence is limited. The core mechanism combines unsupervised feature-spatial clustering with a training-time object-annotation prior; both components are plausible but familiar.

We appreciate the reviewer's observation and agree that the distinction between SaMer and existing methods should be articulated more clearly.

The main contribution of SaMer is not the introduction of a new clustering algorithm itself, but rather the design of token merging specifically to preserve object-level visual evidence that future queries may need to select under late-interaction retrieval.

Existing token compression methods primarily rely on feature similarity or token importance for merging or pruning. However, these strategies may merge evidence from different object instances into a single representative or discard region-level information required by future queries.

To address this issue, SaMer directly incorporates object annotations as a merge prior during training to discourage cross-instance mixing. This is achieved without introducing an auxiliary grounding loss or a learnable merging module, while preserving the original MaxSim interface and requiring neither bounding boxes nor detectors at inference time.

The effectiveness of this design is supported by our existing experiments. In Table 2, under the same token budget K=64K=64 and adaptation conditions, SaMer achieves a Flickr30K R@1 of 82.4 with ColPali, outperforming H-Pool (79.2) and SAP (80.2). Furthermore, Table 4 and the grounding analysis show that the object-aware prior improves not only retrieval accuracy but also the concentration of visual evidence within the correct object regions. Table 4 also shows where the gain comes from: the familiar feature-spatial clustering alone yields no improvement over feature-only merging (80.4 vs. 80.7 R@1 on Flickr30K), whereas the object-aware prior accounts for the +2.0 point gain. This indicates that SaMer’s effectiveness does not come from combining existing components, but primarily from the object-aware prior.

Therefore, the distinction of SaMer lies not merely in applying feature-spatial clustering, but in guiding merge assignments to preserve object-instance evidence required for retrieval and jointly validating this design through retrieval accuracy and phrase-level grounding. We will emphasize this distinction more explicitly in the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.