acceptodds
Under review as a conference paper at ICLR 2027

Retrieval-Aware Representation Compression for Visual Document Retrieval

Abstract

Multi-vector visual document retrieval matches queries against patch-level embeddings to capture fine-grained evidence, but storing and scoring many vectors per page is costly. An effective compressor must reduce the vector count while preserving the query-specific evidence that supports retrieval accuracy. This emerges as a critical challenge due to the fact that semantically similar patches may match different queries, so merging them can weaken query-specific matches. Patch importance alone is also insufficient, as it does not indicate whether that evidence is already represented by semantically similar patches. In light of this, we present a training-free framework termed Retrieval-Aware Representation Compression (RARC), which jointly considers retrieval utility and local role distinctiveness under a fixed vector budget. RARC uses page-conditioned synthetic queries to construct retrieval-response fields from MaxSim assignments. These fields characterize local score sensitivity, with their magnitudes estimating patch utility and their directions describing the supported query directions. RARC protects useful patches whose response directions differ from those of their semantic neighbors and merges the remaining patches using utility-based weights. All operations are performed offline, without retraining the encoders or modifying online scoring. Experiments with two retrievers and three clustering backends on ViDoRe V2 and V3 show improvements of 1.27-3.19 nDCG@5 points in every benchmark average over the corresponding clustering baseline at a 10% vector budget. With H-Pool, indexes are approximately 90% smaller and MaxSim scoring is 7.17-8.20 faster than with uncompressed indexes. These results demonstrate the value of preserving locally distinctive retrieval evidence for accurate retrieval under aggressive compression.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Efficient Information Retrieval, Visual Document Retrieval, Representation Compression, Multimodal RAG

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_BQTYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Retrieval-Aware Representation Compression (RARC), a training-free framework for reducing the storage and computational costs of multi-vector visual document retrieval. RARC generates page-conditioned synthetic queries and uses MaxSim winner assignments to construct retrieval-response fields. Response magnitudes serve as proxies for token retrieval utility, while response directions characterize the query directions supported by each token. Based on these signals, the method protects tokens with distinctive retrieval roles and applies utility-weighted merging to the remaining tokens. The compression is performed offline without retraining the encoders or modifying the online retrieval scorer.

Experiments on ViDoRe V2 and V3 with ColQwen2 and ColPali show improvements of 1.27–3.19 nDCG@5 points over matched clustering baselines in the reported benchmark averages at a 10% vector retention ratio. With H-Pool, RARC reduces index storage by approximately 90% and achieves 7.17–8.20× faster pure MaxSim scoring compared with uncompressed retrieval.

Critical strengths
  1. The paper highlights an important limitation of semantic-based compression: semantically similar document tokens may serve different retrieval roles. The proposed retrieval-response fields provide a principled connection to local MaxSim sensitivity, offering an interpretable way to guide token protection and aggregation. This is a useful perspective on preserving retrieval evidence under aggressive compression.

  2. RARC improves benchmark-average retrieval performance across two retrievers, three clustering backends, and two benchmarks at a 10% vector budget. The evaluation also covers multiple retention ratios and includes component ablations and diagnostic analyses. Importantly, the method preserves the original encoders and online MaxSim scoring function, making it compatible with existing retrieval systems.

Critical weaknesses
  1. The paper’s main novelty is the use of retrieval-response directions to identify tokens with distinctive retrieval roles. However, Table 12 shows that RARC improves over the simpler Top-TRU selection strategy by only 0.137 and 0.090 nDCG@5 points on ColQwen2 and ColPali, respectively. The improvements over shuffled-direction selection are also small. Although the overall gains over clustering baselines are meaningful, these results suggest that much of the benefit may come from utility-based token selection and weighted aggregation rather than directional distinctiveness itself. Without statistical significance testing or stronger evidence of its independent contribution, the central novelty claim remains insufficiently supported.

  2. Although the paper evaluates several clustering and pruning baselines, some important comparisons with relevant prior work are still missing. In particular, Light-ColPali/Light-ColQwen2 (Findings of ACL 2025) provides a directly relevant, established baseline for patch-level compression in visual document retrieval. The authors should also clarify whether their Attention Pooling baseline faithfully reproduces the attention-guided clustering (AGC) method from SIGIR 2026. Furthermore, MAGIC (arXiv 2026) is a closely related concurrent approach that also performs training-free, retrieval-aware compression of frozen multi-vector representations.

Questions for authors
  1. Table 12 shows that RARC improves over Top-TRU selection by only 0.137 and 0.090 nDCG@5 points for ColQwen2 and ColPali, respectively. The gains over shuffled-direction selection are similarly small. Could the authors provide stronger evidence that directional distinctiveness contributes meaningfully beyond utility-based token selection and weighted aggregation? In particular, are there specific retrieval scenarios where directional protection provides a clear advantage over Top-TRU?

  2. Could the authors clarify whether the reported Attention Pooling baseline faithfully reproduces AGC (SIGIR 2026)?

2 comments
AuthorsOpenReview ↗

Response to Reviewer BQTY (1/2)

Thank you for asking how much direction adds once utility already guides compression. Table 12 is the right comparison for that question: Top-TRU uses the same high-utility candidates, the same number of protected tokens on each page, and the same utility-weighted merging as RARC. Only the choice of which candidates to protect changes.

W1/Q1. What does direction add?

On ViDoRe V2 at 10% retention, the mean gains over weighted Top-TRU are +0.137 and +0.090 nDCG@5 points (ColQwen2 and ColPali). Their paired 95% bootstrap intervals are [−0.52, +0.87] and [−0.16, +0.50]. The gains over shuffled directions, averaged over five shuffles, are +0.17 and +0.08, also nonsignificant. We therefore agree that Table 12 does not establish an average advantage over a strong utility-weighted control. We will distinguish this result from the gains of the complete method over unweighted clustering. However, this is not enough to show that directional filtering has not made a sufficient contribution. Protection and weighting have overlapping benefits. In all eight model–V2-subset comparisons, their combined gain is smaller than the sum of their separate gains, consistent with Table 2(a). This helps explain why weighted Top-TRU recovers much of RARC's overall gain.

MethodColQwen2ColPali
H-Pool50.8251.75
Direction-only49.8952.14
TRU-only52.4753.22
TRU-Direction (TRU + direction gate)53.0653.77
RARC53.7754.17

We also tested the choice of protected tokens with uniform H-Pool merging. At the same 10% budget on V2, choosing by utility alone gives 52.47/53.22; choosing by direction alone gives 49.89/52.14. Screening by utility and then choosing by local directional difference gives 53.06/53.77. Thus, among high-utility candidates, direction adds 0.59/0.55 points when the remaining clusters are averaged uniformly. The smaller increment in Table 12 has a practical explanation: on ESG-Human, weighting alone already improves 16 of the 18 queries helped by protection alone for ColQwen2, and 8 of 10 for ColPali.

With weighting enabled, we can still identify cases where the choice of protected patch changes the page ranking. For a regional footprint percentage query (ColQwen2), we changed only one competing page in the weighted Top-TRU index. Substituting RARC's protected vectors reduced that page's MaxSim score from 14.24 to 13.84 and moved the judged-relevant page from rank 6 to 5 (query nDCG@5: 0 to 38.69). Substituting only RARC's centroids left the competitor at 14.22 and the relevant page at rank 6. For a minority protection policy query, the relevant page moved from rank 2 to 1 on both retrievers. On ColQwen2, with the rest of the RARC index fixed, none of the 15 equal-count alternatives to one direction-selected patch recovered rank 1. These selected cases show what local direction can preserve beyond utility; the paired intervals above indicate how far that observation generalizes.

RetrieverTop-TRU (Table 12)Shuffled-directionRARC
ColQwen258.5058.5559.20
ColPali58.1958.3258.84

RARC still corrects errors left by weighted Top-TRU. On ESG-human, the gains increase to +0.70/+0.65 points for ColQwen2/ColPali, which demonstrates that RARC performs better in real-world queries than Top-TRU.

AuthorsOpenReview ↗

Response to Reviewer BQTY (2/2)

W2/Q2. Is Attention Pooling AGC, and how do the closest methods compare?

No. Our Table 1 baseline chooses anchors by patch attention, assigns patches by cosine similarity, and averages each group. AGC (Qin et al., SIGIR 2026) uses learned universal queries to identify anchors and attention weights to aggregate patches. We will identify it as an attention baseline and report AGC separately.

At 64 vectors per page on V2, our ColQwen2.5 results are 54.9 for H-Pool and 56.5 for RARC(H-Pool), and 53.3 for K-Means and 57.7 for RARC(K-Means). For context, Qin et al. report 56.7 for their trained AGC model at this budget (Table 3). The MAGIC scores in Yuan et al. (Table 6) average 55.8 across the three multilingual domain averages and ESG-Human. These are published reference values; the comparisons with our own backends isolate the effect of RARC. MAGIC estimates demand from a shared query pool and allocates it to representatives by optimal transport. RARC uses probes specific to each page to decide which original patches warrant their own vectors before merging the remainder.

MethodBio.Econ.ESGESG-HAvg.
AGC49.250.452.353.151.3
MAGIC57.856.853.255.455.8
H-Pool56.954.552.056.154.9
K-Means55.553.048.756.053.3
RARC(H-Pool)58.453.653.060.956.5
RARC(K-Means)58.053.453.865.657.7

Light-ColPali/Light-ColQwen2 (Ma et al., Findings of ACL 2025) are also relevant. That study evaluates training-free merging and then fine-tunes the retriever with merging for its final models. Our H-Pool result tests semantic merging on frozen embeddings; it is not a reproduction of those trained models. We have no matched result against their checkpoints and will make that distinction explicit in the paper.

References

[1] Hanxiang Qin, Alexander Martin, Rohan Jha, Chunsheng Zuo, Reno Kriz, and Benjamin Van Durme. "Multi-vector index compression in any modality." SIGIR, 2026, pp. 1519–1530.

[2] Yubo Ma, Jinsong Li, Yuhang Zang, Xiaobao Wu, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Haodong Duan, Jiaqi Wang, Yixin Cao, and Aixin Sun. "Towards storage-efficient visual document retrieval: An empirical study on reducing patch-level embeddings." Findings of ACL, 2025, pp. 19568–19580.

[3] Xu Yuan, Hua Liu, Wenqi Fan, and Qing Li. "MAGIC: Marginal-guided compression with optimal transport for efficient visual document retrieval." arXiv:2609.21018, 2026.

We appreciate your time and valuable feedback. We hope our responses address your concerns, and we would be grateful if you could reconsider your score.

Reviewer_sJkrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses document-side vector compression for multi-vector visual document retrievers (ColPali, ColQwen2).

To estimate which patches matter, RARC generates ~10 synthetic queries per page with Qwen3-VL-8B and encodes them with the frozen query encoder. For each patch it then computes a "retrieval-response field" RiR_i: the sum of the query-token vectors whose MaxSim winner is patch ii. With unique winners, RiR_i is the gradient of the aggregate probe score with respect to xix_i. The paper uses ∥Ri∥\|R_i\| as a utility score and Ri/∥Ri∥R_i/\|R_i\| as a "role direction".

These signals drive two offline, training-free steps:

  1. Protection. A patch is kept as a singleton if its utility exceeds τ and its direction deviates by at least γdir\gamma_{dir} from the mean direction of its semantic k-NN.
  2. Weighted merging. The remaining patches are clustered by a standard backend (Ward/H-Pool, K-Means, or a von Mises–Fisher mixture). Each cluster is merged into a normalized utility-weighted mean.

At a 10% vector budget, RARC improves every benchmark-average nDCG@5 over its matched backend: +2.42 to +3.19 on ViDoRe V2 and +1.27 to +1.88 on V3. The appendix contains extensive diagnostics.

Critical strengths
  1. The method is training-free and offline. It keeps the encoders frozen, leaves the online MaxSim scorer unchanged, and plugs into any clustering backend; the paper demonstrates three.

  2. All 12 retriever x backend x benchmark averages improve. On V3, gains hold at every tested budget from 5% to 30% for H-Pool and K-Means, and are generally larger at aggressive compression.

  3. Aggregating the query vectors that win MaxSim at a patch is a sensible importance estimate under late interaction. Its link to the local gradient of the score is correct.

  4. The discussion and diagnostics are thorough and honest. The appendix covers:

    • probe count and cross-page contamination;
    • count-matched alternative selectors (Table 12) and a Shapley-style component attribution (Table 6);
    • a 25-point threshold grid on both benchmarks;
    • bootstrap CIs with Holm correction for the mechanism analyses.

    The authors are careful to state what these studies do not establish.

  5. The prompts, hyperparameters, backend settings, and tie-breaking rules are all documented for reproducibility.

Critical weaknesses

W1. The directional component, the paper's central conceptual claim, is not shown to add value beyond utility

The abstract, introduction, and Fig. 1 argue that "patch importance alone is insufficient" and that local role distinctiveness must also be considered. The paper's own controls seem to only support a weaker claim.

  • In Table 12, RARC beats Top-TRU (utility only) by ~0.1 nDCG@5 for ColQwen2 / ColPali. It beats shuffled directions by a slightly larger but still thin margin. No significance test is reported.
    • For scale: resampling the probes alone changes nDCG@5 by an amount (Tables 18–19), several times larger than these gaps.
    • In the separate similar-utility replacement experiment (App. B.3), some replacements match or exceed RARC on ColQwen2.
  • The "Scalar support" and "Direction only" rows in Table 12 are not defined in the text. "Direction only" is the closest test of direction without utility, and it performs poorly.
  • In Table 6a, each module/step alone recovers most of RARC's gain over H-Pool, and adding the second contributes little. So the gain is not additive

Requests:

  • Paired significance tests for RARC vs. Top-TRU and vs. shuffled direction, on V2 and V3, for both retrievers, ideally also at the 5% budget.
  • Definitions for all Table 12 rows.

If directional deviation does not significantly beat utility-only selection, the contribution should be reframed as probe-derived utility guiding protection and merging, with direction as a secondary refinement.

W2. The comparison does not control for RARC's additional information and compute

RARC runs an 8B VLM over every indexed page. All compared baselines use only document embeddings or encoder internals. This leaves it unclear how much of the gain comes from RARC's specific design, and how much from simply having page-specific query information.

W3. Offline cost is incompletely reported

The dominant offline step is VLM query generation for every page, and its total time "was not recorded" (App. B.2). Query-encoding times are estimates extrapolated from a 200-page sample. Please report generation throughput and GPU-hours relative to corpus encoding, and discuss when the trade-off is worthwhile (e.g., static vs. frequently updated corpora).

The headline "~90% smaller indexes" and "7.17–8.20× faster MaxSim" follow from the 10% budget and apply equally to H-Pool. The paper notes this in Sec. 5.2, but the abstract presents them as RARC results.

W4. No significance tests for the main results, and V2 gains are uneven across splits

  • The bootstrap tests explicitly exclude the main-table nDCG differences. Please add paired tests (bootstrap or randomization over queries) per split and for the averages, for RARC vs. backend and RARC vs. Full.
  • ColQwen2 V2 gains are concentrated in ESG-Human:
    • That split accounts for 62–76% of the V2 average gain across the three backends.
    • The other three splits change by −0.55 to +3.24.
    • On ESG-Human, RARC exceeds Full for two of the three backends (+1.81 K-Means, +1.37 H-Pool), whereas H-Pool alone loses about 7 points.

Please report per-split query counts with CIs, and analyze this split. As noted in W2, it is human-labelled, which makes this analysis valuable.

  • The "exceeds Full" claims rest on differences of 0.05–0.42 points and should read "matches Full" unless shown to be significant.
  • The "functional denoising" interpretation depends on unjudged competitor pages, and its sign depends on pair selection. I would move it to the appendix.
Questions for authors
  1. Paired significance tests for Table 1 and for RARC vs. Top-TRU / shuffled direction (Table 12)? What are the "Scalar support" and "Direction only" rows?
  2. Results with same-probe controls (win-count or win-mass pruning; weighting only) on both benchmarks?
  3. Total GPU-hours for probe generation on V2 and V3, compared with corpus encoding? Variance over generation draws or generators?
  4. Per-split query counts and an analysis of the ESG-Human gains?
3 comments
AuthorsOpenReview ↗

Response to Reviewer sJkr (1/3)

Thank you for asking us to separate two effects. Paired tests support RARC’s gain over its backend. Direction changes some rankings, but its average benefit is small once utility weighting is active.

R1. Direction and utility (W1, Q1).

In Table 12, even Top-TRU uses utility-weighted merging. At matched per-page protection counts, it selects the largest utilities among high-utility candidates. Scalar support selects candidates with the lowest mean neighbor utility; inverted direction selects the least directionally distinctive candidates; shuffled direction permutes directions within each page before recomputing deviations (five seeds). Global-random selects uniformly from all page tokens (five seeds). Direction-only chooses the largest deviations among all tokens with valid directions, without the utility gate. RARC selects locally distinctive candidates after the utility gate. However, this is not enough to show that directional filtering has not made a sufficient contribution. Protection and weighting have overlapping benefits. This is consistent with Table 6a: on ESG-Human, weighting alone improves 16 of the 18 queries helped by protection on ColQwen2 and 8 of 10 on ColPali.

To test which patches merit individual protection, we kept the candidate pool and the number of protected vectors per page fixed and uniformly averaged the remaining clusters.

MethodColQwen2ColPali
H-Pool50.8251.75
Direction-only49.8952.14
TRU-only52.4753.22
TRU-Direction (TRU + direction gate)53.0653.77
RARC53.7754.17

Under this control, changing selection from highest utility to high utility plus local directional disagreement raised nDCG@5 from 52.47 to 53.06 and from 53.22 to 53.77 at 10%. Ranking by direction alone gave 49.89/52.14. The joint rule protects just 5.97/7.00 vectors per page on average. With so few slots, which patch keeps its own vector can still change a top-five ranking.

With uniform merging on V2 at 10%, adding direction to TRU-based protection raises nDCG@5 from 52.47 to 53.06 (ColQwen2) and 53.22 to 53.77 (ColPali). With weighted merging, the corresponding gains shrink to +0.14 (paired 95% bootstrap CI −0.52 to +0.87) and +0.09 (−0.16 to +0.50). Across V2/V3, both retrievers and 5%/10% retention, none of the 16 comparisons against weighted Top-TRU or weighted shuffled direction is significant (Holm-adjusted p ≥ 0.70; eight tests per budget and metric).

Some similar-utility replacements also match RARC. We will distinguish the benefit of retrieval-aware protection and merging from the smaller contribution of direction when utility already guides both operations.

RetrieverTop-TRU (Table 12)RARC
ColQwen258.5059.20
ColPali58.1958.84

RARC still corrects errors left by weighted Top-TRU. On ESG-human, the gains increase to +0.70/+0.65 points for ColQwen2/ColPali, which demonstrates that RARC performs better in real-world queries than Top-TRU.

R2. Identical probes (W2, Q2).

All controls received identical page probes, embeddings and a 10% budget. Win-count retains the most frequent MaxSim winners; win-mass retains patches with the greatest summed positive winning similarity. RARC scored above both pruning rules in all four settings on nDCG@5, with gains surviving Holm correction in three of four settings for each rule. Weighting H-Pool centroids without protecting any singletons came much closer to RARC. For example, V2/ColQwen2 scored 50.87 (win-count), 50.81 (win-mass), 53.48 (weighting only) and 53.77 (RARC). We give all four settings in the accompanying statistical detail. The controls match probe generation and encoding; they differ in how they spend the vector budget.

AuthorsOpenReview ↗

Response to Reviewer sJkr (2/3)

R3. Main results and ESG-Human (W4, Q1, Q4).

At 10%, all 12 benchmark-average nDCG@5 gains over the matched backends pass Holm correction (12 tests per metric and comparator); 9/12 do so for Recall@5. The small positive differences from Full do not (adjusted p = 1.00). We will describe those scores as comparable, not superior; ColQwen2 remains significantly below Full on V3 for all three backends.

ESG-Human contributes one quarter of the V2 mean despite having only 52 of 1,152 queries. Against H-Pool, ColQwen2 improves on 21, declines on 10 and ties on 21 (+8.35 points; paired 95% CI +1.37 to +15.91); ColPali improves on 13, declines on 7 and ties on 32 (+4.47; −1.49 to +10.56). Neither split result survives the full per-split Holm correction. Excluding ESG-Human leaves positive mean gains across all three backends (+0.94 to +1.52 for ColQwen2; +1.74 to +2.31 for ColPali). In one percentage-lookup query, changing a single competing page’s protected vectors, with centroids and all other pages fixed, moved the relevant page from sixth to fifth. This explains one ranking change without assuming the competitor is irrelevant. We will move the denoising interpretation to the appendix. We give split sizes and confidence intervals in the accompanying statistical detail.

R4. Offline cost and generation variance (W3, Q3).

At 5.693 s/page, generating probes projects to about 7.2 h for V2 and 30.4 h for V3 at the measured throughput. For comparison, the V3 document/probe encoding figures are 2.85/3.28 h for ColQwen2 and 1.79/1.99 h for ColPali. The generated text is reusable across retrievers, but changed pages need new probes. At a fixed budget, H-Pool already obtains the roughly 90% storage saving and similar MaxSim speedups. The reason to incur the extra indexing cost is better retrieval at this budget, particularly when the index is reused; we will make that distinction in the abstract.

On 100 pages per benchmark with three draws from the same generator, fixed-intent nDCG@5 standard deviations were 0.32/0.45 on V2 and 0.10/0.11 on V3 (ColQwen2/ColPali). Without fixed intents they were 2.00/2.68 and 0.13/0.19. V2 was more sensitive to prompt choice; this small study measures variability across draws, not across generators.

AuthorsOpenReview ↗

Response to Reviewer sJkr (3/3)

The following are paired 95% bootstrap intervals for differences in nDCG@5 (×100) at 10% retention. V2 and V3 benchmark scores give equal weight to their respective splits. Full denotes the uncompressed index.

Benchmark-level differences. Intervals are for paired differences, rather than for either system’s score in isolation. Holm correction was applied to the tests in separate families for each metric and comparator; the intervals shown are unadjusted.

Benchmark / retriever / backendRARC minus backendRARC minus Full
V2 · ColQwen2 · H-Pool[+0.97, +5.06][−3.08, +0.20]
V2 · ColQwen2 · K-Means[+0.77, +5.23][−3.82, −0.33]
V2 · ColQwen2 · vMFMM[+1.11, +4.88][−3.15, +0.01]
V2 · ColPali · H-Pool[+0.76, +4.10][−1.27, +1.33]
V2 · ColPali · K-Means[+1.30, +4.50][−0.96, +1.81]
V2 · ColPali · vMFMM[+1.75, +4.76][−1.58, +1.17]
V3 · ColQwen2 · H-Pool[+1.00, +1.57][−1.43, −0.86]
V3 · ColQwen2 · K-Means[+1.58, +2.17][−1.33, −0.75]
V3 · ColQwen2 · vMFMM[+1.13, +1.71][−1.61, −1.04]
V3 · ColPali · H-Pool[+1.04, +1.50][−0.17, +0.30]
V3 · ColPali · K-Means[+1.19, +1.69][−0.13, +0.35]
V3 · ColPali · vMFMM[+1.14, +1.63][−0.33, +0.15]

H-Pool differences by split. Intervals are for RARC(H-Pool) minus H-Pool. Counts are query entries in the public test splits, including language versions, not counts of independent underlying questions. The paired analyses are at the query-entry level. These intervals are for the H-Pool comparison only.

SplitRelease queriesColQwen2ColPali
V2 / Economics232[−1.39, +2.63][+0.01, +3.15]
V2 / ESG228[−2.06, +3.55][−0.14, +4.31]
V2 / Biomedical640[+0.93, +3.24][+0.58, +2.53]
V2 / ESG-human52[+1.37, +15.91][−1.49, +10.56]
V3 / CS1290[−0.60, +0.83][+0.22, +1.43]
V3 / Energy1848[+0.94, +2.83][+1.38, +3.01]
V3 / Fin. EN1854[+0.29, +1.90][+0.21, +1.55]
V3 / Fin. FR1920[−0.04, +1.53][+0.77, +2.13]
V3 / HR1908[+2.54, +4.29][+1.31, +2.67]
V3 / Industrial1698[+0.10, +1.70][+0.40, +1.78]
V3 / Pharm.2184[+0.91, +2.08][+0.11, +1.02]
V3 / Physics1812[−0.15, +1.40][+0.53, +1.79]

The released splits contain 1,152 V2 and 14,514 V3 entries. ESG-Human has 52 English queries; the other three V2 splits have four language versions, and the public V3 splits have six. We distinguish the nominal intervals above from the multiplicity-adjusted tests: neither ESG-Human gain is significant in the full per-split family.

Same-probe controls. Each column uses the same probes, frozen embeddings and 10% vector budget. Scores are benchmark-average nDCG@5 (×100).

SettingWin-countWin-massWeighting onlyRARC
V2 · ColQwen250.8750.8153.4853.77
V2 · ColPali52.8952.7953.5354.17
V3 · ColQwen240.4440.3242.8142.93
V3 · ColPali39.8139.9441.9642.17

We appreciate your time and valuable feedback. We hope our responses address your concerns, and we would be grateful if you could reconsider your score.

Reviewer_exAiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This study investigates the problem of reducing storage overhead and retrieval costs associated with multi-vector retrieval in visually rich documents. The core argument of this work is that semantic similarity does not imply retrieval redundancy, and motivated by this, it proposes a training-free compression approach called RARC. Specifically, in the page-conditioned query probing step, synthetic queries are generated to observe which document patches are selected, defining this as the retrieval-response field. Afterwards, each response direction is compared against the average response direction, and patches with significantly differing directions are preserved separately. The remaining unpreserved patches then go through a clustering step. As a result, RARC achieves superior performance compared to existing methodologies.

Critical strengths
  • [S1] Well motivated, important and practical problem: The paper addresses a timely and critical challenge in multi-vector retrieval for visually rich documents: storage overhead and computational costs. The motivation is well-founded and insightful, and the proposed training-free framework (RARC) is intuitive and soundly formulated

  • [S2] Retriever-agnostic applicability: As a training-free compression approach that operates offline without requiring encoder retraining or modifications to online late-interaction scoring, the framework is broadly retriever-agnostic and can be flexibly integrated into diverse multi-vector architectures.

  • [S3] Accuracy: The empirical evaluation is extensive and convincing. The authors demonstrate consistent retrieval performance gains (NDCG@5) across multiple visual retrievers, various clustering backends.

Critical weaknesses
  • [W1] Limited methodological novelty: While the motivation is reasonable, the proposed method appears to be a combination of relatively simple heuristics, including synthetic query-based utility estimation, directional token protection, and utility-weighted merging. The retrieval-response field also seems to follow directly from the MaxSim formulation, making the methodological novelty somewhat limited.

  • [W2] Insufficient evidence supporting the main motivation (main concern): The paper argues that semantically similar patches can serve different retrieval roles and that preserving these distinctions improves compression. However, the additional benefit of directional protection over utility-only selection appears marginal. This raises questions about whether retrieval-role distinctiveness is truly responsible for the observed gains. Further ablations across compression budgets and different levels of semantic–retrieval mismatch would help substantiate this claim.

  • [W3] Computational overhead of synthetic query generation: Although RARC is training-free, it requires generating synthetic queries for every document page using a large VLM. The appendix reports query encoding costs but omits the query generation time. A more complete analysis of offline computational overhead, ideally compared with training-based compression methods, would help establish its practical efficiency.

  • [W4] Dependence on synthetic query distribution: RARC estimates retrieval roles using page-conditioned synthetic queries generated from a fixed set of ten retrieval intents. However, actual query distributions may vary substantially across datasets and domains. It remains unclear whether these predefined intents adequately capture the retrieval roles relevant to real queries, or how sensitive the method is to different query generation strategies.

Questions for authors

See weakness

2 comments
AuthorsOpenReview ↗

Response to Reviewer exAi (1/2)

Thank you for asking whether direction helps specifically when semantic similarity is a poor guide to retrieval. We tested this at four vector budgets and at different levels of mismatch. The distinction that matters is whether residual patches are merged uniformly or already weighted by retrieval utility.

W1. What is new?

For fixed, unique winners, the gradient identity follows from MaxSim; our contribution is how we use the vector of winning query-token responses to decide which patches can share an index slot. Two nearby patches can have equally large response magnitudes while supporting different query directions. On V2, among semantic top-20 neighbor pairs with embedding cosine at least 0.9 and a smaller-to-larger nonzero utility ratio of at least 0.9, 22.7% (ColQwen2) and 17.7% (ColPali) still have response cosine at most 0.5. Utility alone does not reveal this directional difference. RARC uses the local direction comparison for the singleton decision and uses utility to build representatives for the remaining patches.

W2. When does direction help?

On ViDoRe V2, we held the candidate pool, the number of protected vectors per page, and the H-Pool backend fixed. With uniform merging, ranking candidates by local directional deviation rather than utility alone improved mean nDCG@5 at each retention level.

RetentionColQwen2: TRU-only → TRU-DirectionColPali: TRU-only → TRU-Direction
5%48.81 → 49.0952.58 → 52.77
20%54.70 → 55.0853.97 → 54.18
30%55.05 → 55.7653.15 → 53.60

At 5%, 10%, 20%, and 30% retention, the gains were +0.28, +0.59, +0.38, and +0.71 points for ColQwen2, and +0.19, +0.55, +0.21, and +0.45 for ColPali.

MethodColQwen2ColPali
H-Pool50.8251.75
Direction-only49.8952.14
TRU-only52.4753.22
TRU-Direction (TRU + direction gate)53.0653.77
RARC53.7754.17

At 10%, direction without the utility gate scored 49.89/52.14 (ColQwen2/ColPali), utility alone scored 52.47/53.22, and the joint selection rule scored 53.06/53.77. The joint rule thus scores higher than either criterion alone in this comparison.

We also grouped V2 queries by the median semantic–response neighborhood mismatch of their judged-relevant pages. We used quartile cutoffs within each retriever and split. Ties gave unequal group sizes; each reported mean gives equal weight to the splits represented in that group.

RetrieverMismatch quartileQueriesTRU-onlyTRU-Direction
ColQwen2Q1 (lowest)14159.0059.23
Q231649.5150.13
Q312555.4356.98
Q4 (highest)57047.7749.55
ColPaliQ1 (lowest)20659.1159.69
Q28054.2154.98
Q345752.3753.56
Q4 (highest)40950.9352.35

From lowest to highest mismatch, the directional gains with uniform merging at 10% were +0.23, +0.62, +1.55, and +1.78 for ColQwen2 and +0.58, +0.77, +1.19, and +1.42 for ColPali. The largest observed gain was in the highest-mismatch group for both retrievers.

Table 12 asks a different question: its Top-TRU control already uses utility-weighted merging. With weighting in place, the V2 gains from direction were only +0.14/+0.09, with paired 95% confidence intervals [−0.52, +0.87]/[−0.16, +0.50]. One example shows what a small average can conceal. For a regional-footprint query, changing only one competing page’s protected vectors moved the relevant page from rank 6 to 5; changing only its weighted centroids left the rank unchanged. This illustrates a selection effect on one query, not its frequency. We will distinguish such cases from the aggregate result.

Protection and weighting have overlapping benefits. On ESG-human, weighting improves 16/18 of the queries improved by protection for ColQwen2 and 8/10 for ColPali. In all eight model–V2-subset comparisons, their combined gain is smaller than the sum of their separate gains, consistent with Table 2(a). This helps explain why weighted Top-TRU recovers much of RARC's overall gain.

RetrieverTop-TRU (Table 12)RARC
ColQwen258.5059.20
ColPali58.1958.84

RARC still corrects errors left by weighted Top-TRU. On ESG-human, the gains increase to +0.70/+0.65 points for ColQwen2/ColPali, which demonstrates that RARC performs better in real-world queries than Top-TRU.

AuthorsOpenReview ↗

Response to Reviewer exAi (2/2)

W3. Offline cost.

For 1,000 V2 pages, probe generation accounts for an estimated 94.89 minutes (5.693 s/page). Probe encoding adds 10.88/5.44 minutes, giving estimated totals of 105.92/100.58 minutes, versus 0.10/0.18 minutes for H-Pool alone, excluding shared page encoding. RARC therefore trades indexing time for retrieval quality at the same online budget. For context, Ma et al. (Findings of ACL 2025, Table 3) report 9.0 hours of training on eight A100s for Light-ColQwen2 2B and 2.6 minutes to encode 500 pages on one A100. Its training cost can be reused across corpora; RARC incurs probing costs on new pages. Different hardware and timing scopes preclude a direct speed ratio.

W4. Query distribution.

On 100 pages from each benchmark, we removed the ten intent categories but kept the page-grounding, language, and output-format instructions. Over three draws, changing from fixed to open prompts moved mean nDCG@5 from 58.65 to 54.39 and 49.37 to 48.32 on V2 (ColQwen2 and ColPali), versus 36.48 to 36.47 and 35.19 to 35.17 on V3. Some pages yielded fewer than ten usable probes, so these differences reflect both prompt choice and the probes it produced. They do not show that ten intents cover unseen user needs; the intents guide offline probing and place no restrictions on the actual retrieval queries.

We appreciate your time and valuable feedback. We hope our responses address your concerns, and we would be grateful if you could reconsider your score.

Reviewer_3g7tOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on compressing multi-vector visual document indexes (e.g., ColPali, ColQwen2). Standard compression methods (like semantic clustering) assume similar patch embeddings can be safely merged, but the authors show that semantically close patches can serve different retrieval roles under MaxSim. To fix this, the authors propose RARC (Retrieval-Aware Representation Compression):

  1. An 8B VLM generates 10 synthetic queries per page to compute "retrieval-response fields" (the sum of query tokens winning MaxSim on each patch). Field magnitude reflects retrieval utility, while direction captures the supported query orientation.
  2. Patches with high utility whose response directions diverge from their semantic neighbors are saved as standalone vectors.
  3. The remaining budget is compressed via standard clustering, but cluster prototypes are aggregated using utility-based weights rather than simple uniform means.
Critical strengths
  1. The paper points out a blind spot in existing multi-vector compression: semantic proximity does not mean retrieval redundancy. Showing that cosine-similar patches can point in opposite query directions under MaxSim gives a solid motivation for going beyond standard pooling.

  2. The authors tested two retrievers, three clustering backends, and 12 benchmark splits across ViDoRe V2 and V3. The thoroughness of the evaluation, including probe scaling and cross-page noise tests, is commendable.

Critical weaknesses
  1. The main selling point is comparing response directions against semantic neighbors. But in Table 12, simple Top-TRU (picking by utility magnitude alone) scores 53.628 / 54.078, which is barely ~0.1 point behind RARC (53.765 / 54.168). In fact, the authors note that swapping protected tokens with similar-utility ones can even beat RARC on ColQwen2 (lines 1051–1054). The directional mechanism feels over-engineered for the marginal gain it delivers.

  2. Calling the method "training-free" is true, but calling it lightweight ignores the elephant in the room. Running an 8B VLM to generate 10 questions per page is massive compute overhead.

  3. Looking at Table 17 and Appendix B.6, RARC typically protects only 4 to 8 tokens per page (well under 1% of the page's original patches, and less than 10% of the compressed budget).

  4. The authors mention that ~88% of patches have zero response field (lines 447–448). This means the probe-derived utility signal is blind to almost 90% of the document content. If user queries land on these unprobed areas, the compression decisions have no guidance there.

Questions for authors
  1. Given that plain Top-TRU in Table 12 is practically on par with RARC, why bother with the directional neighbor comparison δi\delta_i? Did you test if directional filtering shows any meaningful advantage at other retention budgets (like 5% or 20%)?

  2. Could you share the total end-to-end time needed to index 1,000 pages on an L40 GPU, explicitly adding up: (a) Qwen3-VL-8B query generation, (b) query encoding, and (c) RARC clustering and merging? How does this total time compare to standard H-Pool or SAP?

  3. Since ~88% of patches get zero response signals from the 10 probes, how does RARC behave when real test queries match specific figures or fine-print details that were completely missed during probe generation?

  4. What happens on page types where half of these intents don't make sense (e.g., pure artwork, code repositories, or full-page diagrams)? Have you tried open-ended query generation without the rigid template?

  5. Is RARC actually fixing false negatives in the original model, or is it behaving like a re-ranker that changes the retrieval landscape entirely?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 3g7t (1/2)

Thank you for asking whether the choice of a few protected vectors justifies the additional computation. Two different costs matter here: generating probes is expensive, but both Top-TRU and RARC need those probes. Table 12 compares what they do with the same information.

Q1. What does direction add, and at which budgets?

Top-TRU selects by token retrieval utility (TRU) alone but already uses RARC’s utility-weighted merging. At 10% retention on V2, adding direction increases mean nDCG@5 by only 0.137/0.090 points (ColQwen2/ColPali throughout). Neither difference is significant in the paired tests (p = .713/.961). Some similar-utility replacements also score higher on ColQwen2. Thus Table 12 does not establish that direction is necessary for the average gain over H-Pool. However, this is not enough to show that directional filtering has not made a sufficient contribution. Protection and weighting have overlapping benefits. On ESG-human, weighting improves 16/18 of the queries improved by protection for ColQwen2 and 8/10 for ColPali. In all eight model–V2-subset comparisons, their combined gain is smaller than the sum of their separate gains, consistent with Table 2(a). This helps explain why weighted Top-TRU recovers much of RARC's overall gain.

To test which patches merit individual protection, we kept the candidate pool and the number of protected vectors per page fixed and uniformly averaged the remaining clusters.

MethodColQwen2ColPali
H-Pool50.8251.75
Direction-only49.8952.14
TRU-only52.4753.22
TRU-Direction (TRU + direction gate)53.0653.77
RARC53.7754.17

Under this control, changing selection from highest utility to high utility plus local directional disagreement raised nDCG@5 from 52.47 to 53.06 and from 53.22 to 53.77 at 10%. Ranking by direction alone gave 49.89/52.14. The joint rule protects just 5.97/7.00 vectors per page on average. With so few slots, which patch keeps its own vector can still change a top-five ranking.

RetentionColQwen2: TRU-only → TRU-DirectionColPali: TRU-only → TRU-Direction
5%48.81 → 49.0952.58 → 52.77
20%54.70 → 55.0853.97 → 54.18
30%55.05 → 55.7653.15 → 53.60

Repeating the matched utility-versus-joint comparison at 5%, 20%, and 30% gave gains of +0.28/+0.19, +0.38/+0.21, and +0.71/+0.45 points, respectively. These are protection-only comparisons. Weighted merging recovers much of the same benefit; we will distinguish these two comparisons in the paper.

RetrieverTop-TRU (Table 12)RARC
ColQwen258.5059.20
ColPali58.1958.84

RARC still corrects errors left by weighted Top-TRU. On ESG-human, the gains increase to +0.70/+0.65 points for ColQwen2/ColPali, which demonstrates that RARC performs better in real-world queries than Top-TRU.

Q2. What does indexing 1,000 pages cost?

For 1,000 V2 pages (250 per split; 10% retention), Qwen3-VL-8B generation is estimated at 94.89 min from 5.693 s/page. The same generated text serves both retrievers. Probe encoding adds 10.88/5.44 min, clustering and merging take 6.05/10.77 s, and response computation, protection and index saving together take 2.66/3.91 s. The resulting estimates are 105.92/100.58 min for RARC versus 0.10/0.18 min for H-Pool, excluding document encoding shared by both. These are estimates for the three requested stages plus index saving. At the same budget, the benefit over H-Pool is retrieval quality; online costs are similar. We will explicitly report generation cost and restrict the practical claim to settings where an index can be reused.

AuthorsOpenReview ↗

Response to Reviewer 3g7t (2/2)

Q3. What if the probes miss a detail?

The roughly 88% figure counts patches with a zero response field; it does not count content removed from the index. Those patches still enter semantic clusters: Eq. (9) gives even a zero-response patch a positive weight, and a cluster of such patches is averaged uniformly. Sparse matching is inherent to MaxSim, including for real test queries. Across V2’s 1,152 test queries and 6,584 query–relevant-page pairs, this gives mean upper bounds of 4.20%/2.85% of page vectors for ColQwen2/ColPali. A low probe-hit fraction therefore does not, by itself, establish poor retrieval coverage.

For a real query about a biomedical chart, 70.4%/74.1% of the full index’s winning matches were to zero-response patches. After compression, four of the 27 query-token matches on each retriever were to representatives made entirely from such patches. RARC, H-Pool and the full index all scored 63.09 nDCG@5 and 100 Recall@5 on this query (scores ×100). This illustrates how missed evidence can survive. Fine print can still be lost if its vector is diluted during merging; probes give no protection guarantee for an unseen query. The relevant question is whether compression preserves useful matching evidence, rather than how many original vectors the probes activate.

Q4. Are fixed intents necessary?

We removed the intent categories while keeping the grounding, language and output instructions, and compared three generation draws on 100 pages from each benchmark. Mean nDCG@5 changed from 58.65/49.37 (fixed) to 54.39/48.32 (open) on V2 and from 36.48/35.19 to 36.47/35.17 on V3. Some pages yielded fewer than ten usable probes, so this comparison includes variation in probe count. This small test favors the fixed template on V2 and shows little difference on V3. It included diagram pages, but it does not establish the best prompt for standalone artwork or code repositories.

Q5. Does RARC correct false negatives in the original model?

We do not claim a systematic correction of errors made by the uncompressed retriever. Compression changes the vectors scored by the original MaxSim function, so it can change rankings without an online reranker. For a regional-footprint query, we held the rest of the weighted Top-TRU index fixed. Changing only the protected vectors on one competing page lowered its score from 14.24 to 13.84 and moved the unchanged relevant page from rank 6 to 5. Changing only that competitor’s centroids left the relevant page at rank 6. This gain came from reducing a competing match.

We appreciate your time and valuable feedback. We hope our responses address your concerns, and we would be grateful if you could reconsider your score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of RARC’s Contributions, Strengths, and Responses to Concerns

Dear AC, SAC, and PCs,

We sincerely appreciate the time and effort you have devoted to overseeing the review. We also thank the reviewers for their careful reading and for recognizing the practical importance of compressing multi-vector document indexes and the breadth of our evaluation across retrievers and clustering backends. As the discussion concludes, we summarize RARC’s contributions, the strengths the reviewers highlighted, and the additional experiments addressing their main questions about directional selection, baselines, and offline indexing cost. We hope this overview helps you weigh the evidence in your final assessment.

1. Core Contributions

  • Retrieval-aware compression. Semantically close patches can serve different retrieval roles. RARC protects high-utility patches with locally distinctive MaxSim response directions and weights the rest by utility during merging.
  • Accuracy at a fixed budget. At 10% retention, all 12 benchmark-average gains over matched clustering backends on ViDoRe V2 and V3 were significant after Holm correction (1.27–3.19 nDCG@5 points).

2. Strengths Recognized by Reviewers

  • Motivation. 3g7t highlighted the limits of semantic pooling, and BQTY valued the link to MaxSim sensitivity.
  • Applicability. exAi and sJkr noted consistent gains across backends with frozen encoders and an unchanged online scorer.

3. Responses to Key Concerns

We compared directional selection with utility ranking using the same high-utility candidates and the same number of protected patches per page.

ReviewersConcernEvidence from the rebuttal
3g7t, exAi, sJkr, BQTYWhy is RARC close to Top-TRU in Table 12?Both methods use utility-weighted merging, which benefits many of the same queries as protection. In a separate V2 ablation at 10% retention with uniform H-Pool merging, directional selection gained about 0.6 nDCG@5 points per retriever over utility ranking.
3g7t, sJkr, BQTYCan directional selection still improve rankings when weighting is active?On the human-labeled ESG-human split, full RARC averaged 0.70 nDCG@5 points above weighted Top-TRU on ColQwen2 and 0.65 on ColPali. For one query, changing only a competing page’s protected vectors moved the relevant page from sixth to fifth.
exAi, sJkr, BQTYDo the gains persist with identical probes and against prior methods?RARC outperformed scalar pruning with identical probes. On V2 with ColQwen2.5, it also outscored AGC and MAGIC at a matched token budget.
3g7t, exAi, sJkrWhat does probing cost, and how sensitive is retrieval to the prompt?RARC adds an estimated 100–106 minutes per 1,000 pages over H-Pool, mostly for query generation. In the prompt comparison, open-ended prompts lowered mean nDCG@5 on V2 and had little effect on V3.

4. Concluding Remarks

For stable collections, RARC’s one-time indexing cost can be spread over many searches while preserving the online vector budget. We appreciate your consideration.

Thank you again for your time and thoughtful consideration.

Best regards,

The Authors of Submission #1175

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.