- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Retrieval-Aware Representation Compression (RARC), a training-free framework for reducing the storage and computational costs of multi-vector visual document retrieval. RARC generates page-conditioned synthetic queries and uses MaxSim winner assignments to construct retrieval-response fields. Response magnitudes serve as proxies for token retrieval utility, while response directions characterize the query directions supported by each token. Based on these signals, the method protects tokens with distinctive retrieval roles and applies utility-weighted merging to the remaining tokens. The compression is performed offline without retraining the encoders or modifying the online retrieval scorer.
Experiments on ViDoRe V2 and V3 with ColQwen2 and ColPali show improvements of 1.27–3.19 nDCG@5 points over matched clustering baselines in the reported benchmark averages at a 10% vector retention ratio. With H-Pool, RARC reduces index storage by approximately 90% and achieves 7.17–8.20× faster pure MaxSim scoring compared with uncompressed retrieval.
Critical strengths
-
The paper highlights an important limitation of semantic-based compression: semantically similar document tokens may serve different retrieval roles. The proposed retrieval-response fields provide a principled connection to local MaxSim sensitivity, offering an interpretable way to guide token protection and aggregation. This is a useful perspective on preserving retrieval evidence under aggressive compression.
-
RARC improves benchmark-average retrieval performance across two retrievers, three clustering backends, and two benchmarks at a 10% vector budget. The evaluation also covers multiple retention ratios and includes component ablations and diagnostic analyses. Importantly, the method preserves the original encoders and online MaxSim scoring function, making it compatible with existing retrieval systems.
Critical weaknesses
-
The paper’s main novelty is the use of retrieval-response directions to identify tokens with distinctive retrieval roles. However, Table 12 shows that RARC improves over the simpler Top-TRU selection strategy by only 0.137 and 0.090 nDCG@5 points on ColQwen2 and ColPali, respectively. The improvements over shuffled-direction selection are also small. Although the overall gains over clustering baselines are meaningful, these results suggest that much of the benefit may come from utility-based token selection and weighted aggregation rather than directional distinctiveness itself. Without statistical significance testing or stronger evidence of its independent contribution, the central novelty claim remains insufficiently supported.
-
Although the paper evaluates several clustering and pruning baselines, some important comparisons with relevant prior work are still missing. In particular, Light-ColPali/Light-ColQwen2 (Findings of ACL 2025) provides a directly relevant, established baseline for patch-level compression in visual document retrieval. The authors should also clarify whether their Attention Pooling baseline faithfully reproduces the attention-guided clustering (AGC) method from SIGIR 2026. Furthermore, MAGIC (arXiv 2026) is a closely related concurrent approach that also performs training-free, retrieval-aware compression of frozen multi-vector representations.
Questions for authors
-
Table 12 shows that RARC improves over Top-TRU selection by only 0.137 and 0.090 nDCG@5 points for ColQwen2 and ColPali, respectively. The gains over shuffled-direction selection are similarly small. Could the authors provide stronger evidence that directional distinctiveness contributes meaningfully beyond utility-based token selection and weighted aggregation? In particular, are there specific retrieval scenarios where directional protection provides a clear advantage over Top-TRU?
-
Could the authors clarify whether the reported Attention Pooling baseline faithfully reproduces AGC (SIGIR 2026)?
2 comments
Response to Reviewer BQTY (2/2)
W2/Q2. Is Attention Pooling AGC, and how do the closest methods compare?
No. Our Table 1 baseline chooses anchors by patch attention, assigns patches by cosine similarity, and averages each group. AGC (Qin et al., SIGIR 2026) uses learned universal queries to identify anchors and attention weights to aggregate patches. We will identify it as an attention baseline and report AGC separately.
At 64 vectors per page on V2, our ColQwen2.5 results are 54.9 for H-Pool and 56.5 for RARC(H-Pool), and 53.3 for K-Means and 57.7 for RARC(K-Means). For context, Qin et al. report 56.7 for their trained AGC model at this budget (Table 3). The MAGIC scores in Yuan et al. (Table 6) average 55.8 across the three multilingual domain averages and ESG-Human. These are published reference values; the comparisons with our own backends isolate the effect of RARC. MAGIC estimates demand from a shared query pool and allocates it to representatives by optimal transport. RARC uses probes specific to each page to decide which original patches warrant their own vectors before merging the remainder.
| Method | Bio. | Econ. | ESG | ESG-H | Avg. |
|---|---|---|---|---|---|
| AGC | 49.2 | 50.4 | 52.3 | 53.1 | 51.3 |
| MAGIC | 57.8 | 56.8 | 53.2 | 55.4 | 55.8 |
| H-Pool | 56.9 | 54.5 | 52.0 | 56.1 | 54.9 |
| K-Means | 55.5 | 53.0 | 48.7 | 56.0 | 53.3 |
| RARC(H-Pool) | 58.4 | 53.6 | 53.0 | 60.9 | 56.5 |
| RARC(K-Means) | 58.0 | 53.4 | 53.8 | 65.6 | 57.7 |
Light-ColPali/Light-ColQwen2 (Ma et al., Findings of ACL 2025) are also relevant. That study evaluates training-free merging and then fine-tunes the retriever with merging for its final models. Our H-Pool result tests semantic merging on frozen embeddings; it is not a reproduction of those trained models. We have no matched result against their checkpoints and will make that distinction explicit in the paper.
References
[1] Hanxiang Qin, Alexander Martin, Rohan Jha, Chunsheng Zuo, Reno Kriz, and Benjamin Van Durme. "Multi-vector index compression in any modality." SIGIR, 2026, pp. 1519–1530.
[2] Yubo Ma, Jinsong Li, Yuhang Zang, Xiaobao Wu, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Haodong Duan, Jiaqi Wang, Yixin Cao, and Aixin Sun. "Towards storage-efficient visual document retrieval: An empirical study on reducing patch-level embeddings." Findings of ACL, 2025, pp. 19568–19580.
[3] Xu Yuan, Hua Liu, Wenqi Fan, and Qing Li. "MAGIC: Marginal-guided compression with optimal transport for efficient visual document retrieval." arXiv:2609.21018, 2026.
We appreciate your time and valuable feedback. We hope our responses address your concerns, and we would be grateful if you could reconsider your score.
Response to Reviewer BQTY (1/2)
Thank you for asking how much direction adds once utility already guides compression. Table 12 is the right comparison for that question: Top-TRU uses the same high-utility candidates, the same number of protected tokens on each page, and the same utility-weighted merging as RARC. Only the choice of which candidates to protect changes.
W1/Q1. What does direction add?
On ViDoRe V2 at 10% retention, the mean gains over weighted Top-TRU are +0.137 and +0.090 nDCG@5 points (ColQwen2 and ColPali). Their paired 95% bootstrap intervals are [−0.52, +0.87] and [−0.16, +0.50]. The gains over shuffled directions, averaged over five shuffles, are +0.17 and +0.08, also nonsignificant. We therefore agree that Table 12 does not establish an average advantage over a strong utility-weighted control. We will distinguish this result from the gains of the complete method over unweighted clustering. However, this is not enough to show that directional filtering has not made a sufficient contribution. Protection and weighting have overlapping benefits. In all eight model–V2-subset comparisons, their combined gain is smaller than the sum of their separate gains, consistent with Table 2(a). This helps explain why weighted Top-TRU recovers much of RARC's overall gain.
We also tested the choice of protected tokens with uniform H-Pool merging. At the same 10% budget on V2, choosing by utility alone gives 52.47/53.22; choosing by direction alone gives 49.89/52.14. Screening by utility and then choosing by local directional difference gives 53.06/53.77. Thus, among high-utility candidates, direction adds 0.59/0.55 points when the remaining clusters are averaged uniformly. The smaller increment in Table 12 has a practical explanation: on ESG-Human, weighting alone already improves 16 of the 18 queries helped by protection alone for ColQwen2, and 8 of 10 for ColPali.
With weighting enabled, we can still identify cases where the choice of protected patch changes the page ranking. For a regional footprint percentage query (ColQwen2), we changed only one competing page in the weighted Top-TRU index. Substituting RARC's protected vectors reduced that page's MaxSim score from 14.24 to 13.84 and moved the judged-relevant page from rank 6 to 5 (query nDCG@5: 0 to 38.69). Substituting only RARC's centroids left the competitor at 14.22 and the relevant page at rank 6. For a minority protection policy query, the relevant page moved from rank 2 to 1 on both retrievers. On ColQwen2, with the rest of the RARC index fixed, none of the 15 equal-count alternatives to one direction-selected patch recovered rank 1. These selected cases show what local direction can preserve beyond utility; the paired intervals above indicate how far that observation generalizes.
RARC still corrects errors left by weighted Top-TRU. On ESG-human, the gains increase to +0.70/+0.65 points for ColQwen2/ColPali, which demonstrates that RARC performs better in real-world queries than Top-TRU.