Retrieval-Aware Representation Compression for Visual Document Retrieval

Multi-vector visual document retrieval matches queries against patch-level embeddings to capture fine-grained evidence, but storing and scoring many vectors per page is costly. An effective compressor must reduce the vector count while preserving the query-specific evidence that supports retrieval accuracy. This emerges as a critical challenge due to the fact that semantically similar patches may match different queries, so merging them can weaken query-specific matches. Patch importance alone is also insufficient, as it does not indicate whether that evidence is already represented by semantically similar patches. In light of this, we present a training-free framework termed Retrieval-Aware Representation Compression (RARC), which jointly considers retrieval utility and local role distinctiveness under a fixed vector budget. RARC uses page-conditioned synthetic queries to construct retrieval-response fields from MaxSim assignments. These fields characterize local score sensitivity, with their magnitudes estimating patch utility and their directions describing the supported query directions. RARC protects useful patches whose response directions differ from those of their semantic neighbors and merges the remaining patches using utility-based weights. All operations are performed offline, without retraining the encoders or modifying online scoring. Experiments with two retrievers and three clustering backends on ViDoRe V2 and V3 show improvements of 1.27-3.19 nDCG@5 points in every benchmark average over the corresponding clustering baseline at a 10% vector budget. With H-Pool, indexes are approximately 90% smaller and MaxSim scoring is 7.17-8.20$\times$ faster than with uncompressed indexes. These results demonstrate the value of preserving locally distinctive retrieval evidence for accurate retrieval under aggressive compression.