acceptodds
Under review as a conference paper at ICLR 2027

From Static Compression to Query-Adaptive Representation for Efficient Visual Document Retrieval

Abstract

Visual document retrieval aims to efficiently identify fine-grained query-relevant evidence from long and visually rich documents. Multi-vector retrieval models are effective at localizing such evidence through fine-grained late interaction, but storing and matching hundreds of vectors per page incurs substantial memory and computation costs. Existing compression methods reduce this overhead through pruning or merging, yet the resulting page representations are typically fixed during offline indexing and reused for all future queries. This creates a fundamental mismatch between query-agnostic compression and query-dependent retrieval, since different queries may rely on different semantic aspects of the same page. We propose Query-Conditioned Reconstruction (QCR), a training-free framework that preserves compact reconstructable variation during indexing and instantiates query-specific page representations at retrieval time. QCR augments static compressed representatives with low-rank residual memory, selectively activates groups with high query-relevant reconstruction potential, and reconstructs query-conditioned vectors without accessing the original full multi-vector representation. A budgeted MaxSim-coverage objective further selects a compact set of complementary vectors for late interaction. Experiments across multiple retrieval encoders and benchmarks show that QCR better preserves Full-Index retrieval quality under aggressive compression, while improving evidence retrieval and downstream long-document question answering.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multi-Vector Retrieval, Visual Document Retrieval, Long Document Understanding, Compression

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_9umaOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Query-Conditioned Reconstruction (QCR), a training-free framework for query-adaptive compression in visual document retrieval. To address the limitations of static multi-vector compression, QCR preserves cluster centroids and low-rank residual statistics during offline indexing, and dynamically reconstructs query-conditioned document vectors at retrieval time. The framework further introduces Budgeted MaxSim Coverage to select complementary vectors under a fixed representation budget. Experiments across multiple retrieval backbones and visual document benchmarks demonstrate consistent improvements over static compression baselines, with additional gains in evidence retrieval and downstream long-document question answering.

Critical strengths

1.The paper identifies a meaningful limitation of static compression: document representations are constructed without knowledge of future queries, potentially suppressing query-relevant fine-grained information. Deferring part of the representation construction until retrieval time provides a reasonable direction for addressing this limitation.

2.QCR does not require updating the retrieval encoder or accessing the original full multi-vector representations during retrieval. Its combination of low-rank residual memory, query-aware group activation, and closed-form reconstruction offers a potentially useful approach to query adaptation under constrained storage budgets.

Critical weaknesses

1.Although the paper clearly motivates the limitations of static compression, it does not sufficiently explain why cluster centroids and low-rank residual statistics are adequate for preserving query-relevant information. The residual subspace is selected through SVD based on explained residual energy, which is not necessarily aligned with retrieval relevance. Low-energy directions may still contain critical fine-grained evidence, while high-energy directions may capture less discriminative variations. Consequently, the relationship between residual reconstruction fidelity and retrieval-relevant information preservation remains insufficiently established.

2.QCR constructs a continuous reconstruction region using cluster centroids, a low-rank residual basis, and second-order statistics. However, the reconstructed vectors do not necessarily correspond to actual visual embeddings present in the original document. Moreover, the diagonal residual statistics may overlook correlations between feature directions, potentially allowing combinations that were never observed in the original embeddings. Although the trust radius constrains reconstruction, it does not guarantee semantic fidelity. As a result, query-conditioned reconstruction may artificially increase matching scores for irrelevant pages, particularly hard negatives.

3.The reconstruction procedure primarily seeks to maximize cosine similarity between the query and reconstructed vectors. However, increasing query–document similarity does not necessarily improve ranking discrimination across documents. Both relevant and irrelevant pages may benefit from query-conditioned adaptation, and the paper does not sufficiently analyze whether reconstruction systematically improves positive–negative score margins. While the reported NDCG and Recall improvements support the overall effectiveness of the method, they do not fully explain the underlying mechanism responsible for better ranking.

Questions for authors

1.Why should SVD-based residual energy maximization preserve fine-grained query-relevant information? Is there empirical evidence supporting this assumption?

2.Since reconstructed vectors may not correspond to actual visual features, how does QCR avoid introducing spurious matches, particularly for hard negatives?

3.Maximizing query–reconstruction similarity does not necessarily improve retrieval ranking. Could the authors analyze its impact on positive–negative ranking margins?

3 comments

R1: Residual energy versus retrieval relevance.

SVD minimizes residual projection error at a fixed rank, providing a query-agnostic approximation criterion; query-dependent activation subsequently selects useful preserved variation. This does not guarantee preservation of every low-energy discriminative cue. To examine this choice, we compare leading, random, and low-energy subspaces with identical groups and ranks, re-estimating the residual statistics for each variant.

methodNDCG@5 Ret. (%)Recall@5 (%)
Random orthogonal91.785.6
Low-energy SVD89.283.1
SVD (ours)94.688.9

SVD improves NDCG@5 retention and Recall@5 over the random orthogonal, with larger gains over the low-energy SVD subspace. This supports preserving leading residual directions under a fixed rank budget, without implying that every low-energy direction is irrelevant.

R2: Semantic fidelity and spurious matches.

We thank the reviewer for this concern. QCR constructs query-conditioned retrieval vectors from compressed page statistics. Its page-specific residual basis restricts adaptation directions, while group statistics, the trust radius, and the step-size bound constrain the update. These constraints do not guarantee semantic fidelity or rule out spurious matches. To investigate potential spurious matches induced by hard-negative scores, we report Mean inflation score. Scores are normalized by query length, and the positive part of each hard negative's score difference is averaged over a fixed evaluation set.

MethodMean Inflation ↓
Full Index0.0000
Static0.0031
Sculpting0.0048
QCR0.0045

QCR exhibits higher mean inflation than Static but slightly lower inflation than Sculpting. To better assess the ranking impact of potential spurious matches, we further examine positive--negative margins and pairwise accuracy in R3.

R3: Does similarity maximization improve ranking?

We compare Static, Sculpting, and QCR using fixed positive–hard-negative pairs under matched storage budgets. Scores are normalized by query length, relevance labels determine the preferred page, and ties count as incorrect. QCR achieves larger mean margins and higher pairwise accuracy across the three storage ratios. At 5% storage, it increases the mean margin from 0.0121 to 0.0241 and pairwise accuracy from 71.6% to 80.3% relative to Static. Thus, although reconstruction can increase hard-negative scores, the margin improvements indicate that relevant pages benefit more on average over the evaluated pairs.

Storage RatioMethodMean Positive ScoreMean Margin ↑Pairwise Acc. (%) ↑
2%Static0.64520.006864.8
2%Sculpting0.66050.014272.1
2%QCR (ours)0.67460.020777.5
5%Static0.65430.012171.6
5%Sculpting0.66610.020077.2
5%QCR0.67400.024180.3
10%Static0.66230.017977.3
10%Sculpting0.67080.023580.4
10%QCR0.67360.025583.0

QCR consistently achieves larger positive–negative margins and higher pairwise accuracy than both baselines across storage ratios. These results indicate that reconstruction benefits relevant pages more on average and improves ranking discrimination over the evaluated pairs.

The additional experiments and analyses have largely addressed my concerns regarding Questions 1 and 3.Regarding Question 2, I acknowledge that exact semantic fidelity is not necessarily required for compressed retrieval representations. Although the potential for spurious matches remains a limitation, the additional analyses have sufficiently alleviated my concerns about its practical impact. I am willing to raise my rating to 3 and encourage the authors to incorporate these additional experiments and analyses into the revised manuscript.

We sincerely thank you for your support of our additional experiments and analyses. As you suggested, we will fully incorporate these additional experiments and analyses into the revised manuscript.

Best

Reviewer_34i1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Query-Conditioned Reconstruction (QCR), a paradigm that separates the representation stored at indexing time from the representation used for retrieval to answer the query. The idea is to store a compact, reconstructible representation of the page and to reconstruct, or specifically adapt, this representation to focus on the relevant semantic aspects of the query. It makes the representation adapt to the query. The reconstructible representation is based on low-rank residual memory. An experimental validation across multiple retrieval encoders and benchmarks (ViDoRe 1 and 2, MMLongBench, and LongDocURL) demonstrates the benefit of the approach.

Critical strengths
  • The paper is well written, motivated, and well illustrated.
  • A proper formalization
  • Nice experimental section.
  • The paper addresses an important limitation of Visual RAG
Critical weaknesses
  • Some typos: qurey versus query on Figure 2
  • Some technical choices could be more motivated or justified. For instance, the choice to use SVD for low-rank decomposition makes sense, but it could be discussed further, as well as the fact that it only retains the group centers (alternatives should be possible).
  • The query-conditioned reconstruction is optimized to prevent the reconstruction of every group, but couldn’t the maximum be estimated from the moment
Questions for authors
  • How would the method perform with a global database rather than a per-page database?
2 comments

We thank the reviewer for the positive assessment and helpful suggestions. We will correct the noted typo errors.

R1: Global database.

If a global database refers to retrieval over a corpus of pages, QCR can retain a separate compressed memory for each page and apply reconstruction to candidate pages. Page-wise indexing therefore does not restrict retrieval to a single document. If instead it refers to clustering embeddings jointly across pages and sharing residual statistics, this would require maintaining the association between reconstructed vectors and their source pages. Such pooling may also increase within-group heterogeneity. Our current formulation preserves page boundaries to retain local residual structure and an explicit correspondence to retrieval targets.

R2: Motivation for SVD and group centers.

Before normalization, each group center is the mean of its assigned embeddings, which minimizes within-group squared reconstruction error. We then summarize the remaining variation using truncated SVD, which minimizes the Frobenius reconstruction error of the page residual matrix at a fixed rank. Sharing one basis across groups within each page provides a compact representation of this variation. Figure 6 shows the benefit of retaining residual information beyond group centers alone, while the equal-rank comparisons with random and low-energy subspaces in our response to Reviewer 9uma (R1) support the choice of leading SVD directions.

Regarding direct estimation from moments, our activation surrogate already uses stored residual statistics to estimate group relevance. We additionally compare against direct scoring from optimized group similarities, without constructing a shared candidate pool or applying coverage selection, in our response to Reviewer VzLs (R2) .

Dear Reviewers,

Please take a moment to review the authors' rebuttal and provide your feedback on whether your concerns have been adequately addressed. Thank you for your time and effort.

Reviewer_VzLsOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on visual document retrieval. To address the mismatch between query-agnostic compression and query-dependent retrieval, it proposes the Query-Conditioned Reconstruction framework, which constructs query-conditioned vectors during online retrieval based on each group’s potential support for the query, taking into account the variation around its center. Comparisons with baselines and ablation experiments demonstrate the effectiveness of the proposed components.

Critical strengths

The method is intuitive and technically sound. It preserves variation information around group centers and uses it to estimate each group’s potential support for the query. The framework does not require additional training and can be applied to existing multi-vector retrievers, making it practically useful.

Comparisons with baselines support the effectiveness of the method across different retrievers and benchmarks under limited storage budgets. The ablation experiments also show improvements from residual reconstruction, residual-aware activation, and budgeted coverage, supporting the contribution of the designed components.

Critical weaknesses

The motivation, namely the mismatch between query-agnostic compression and query-dependent retrieval, may not fully reflect the core of the method. QCR still stores a fixed, query-agnostic index, while constructing query-dependent representations during retrieval. Its main contribution appears to be preserving and using variation information lost during compression. The paper should more clearly distinguish these two aspects.

The paper does not fully demonstrate the benefit of explicitly constructing query-conditioned vectors. For each query token, the reconstruction step already obtains an optimized similarity under the residual constraints. These similarities may be sufficient for retrieval. Although the reconstructed vectors enable cross-token matching and budgeted coverage, the current ablations do not compare the full pipeline with direct scoring based on these optimized similarities.

Questions for authors

Using the same stored residual statistics and activated groups, how does direct scoring compare with the full pipeline in retrieval accuracy and latency? Specifically, each query token could take its highest optimized group similarity, including the static-center scores, and these scores could then be summed without constructing and selecting a shared candidate set.

Do the authors have an analysis of how reconstruction changes the scores of relevant pages and hard negatives relative to the full index? Since each reconstructed vector is optimized to match a query token, it would be useful to establish whether this improves the separation between relevant and irrelevant pages or also introduces unsupported high similarities.

3 comments

We thank the reviewer for the positive assessment of our work.

R1: Contribution and motivation.

We agree that the stored index remains query-agnostic. QCR preserves residual variation offline and determines the final scoring vectors after observing the query. We will clarify this distinction. Figure 7(b) supports this design: under the same storage budget, QCR achieves 94.6% NDCG@5 retention versus 89.0% for storage-matched static centroids.

R2: Direct scoring versus reconstruction.

We compare three variants using the same residual memory and activated groups. Direct scoring sums each query token's highest optimized group or static-center similarity, without constructing a shared candidate pool. Full Pool MaxSim aggregates similarities over the complete pool of static and reconstructed vectors, while QCR applies greedy coverage to select a budgeted subset before aggregation. Full Pool MaxSim and QCR share the same candidate pool and reuse the computed query–candidate similarities for scoring. Their latency measurements include activation, reconstruction, similarity computation, and aggregation, with greedy selection additionally included for QCR.

MethodNDCG@5 Ret. (%)Recall@5 (%)Latency (ms)
Direct scoring93.387.67.6
Full Pool MaxSim94.288.68.1
QCR (ours)94.688.910.2

QCR improves NDCG@5 retention and Recall@5 over Direct scoring by 1.3 percentage points each, with 2.6 ms additional latency. Compared with Full Pool MaxSim, the corresponding gains are 0.4 and 0.3 points, with 2.1 ms additional latency.

R3: Relevant-page and hard-negative scores.

We thank the reviewer for this suggestion. Relevant details can be found in our response to Reviewer 9uma's Question 2 and 3. In brief, QCR consistently yields larger positive–negative margins and higher pairwise accuracy than Static and Sculpting across storage ratios.

Thank you for the additional experiments. The direct-scoring comparison and hard-negative analysis have largely addressed my concerns. I recommend including these results in the main paper or supplementary material. I maintain my weak accept rating.

We sincerely thank you for your support, and we will include these results in the main paper.

Best

Reviewer_Am5COpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper tackles a well-known headache in multi-vector visual document retrieval (like ColPali/ColQwen): storing hundreds of vectors per page eats up crazy amounts of memory and compute. Existing methods tackle this by pruning or merging tokens offline, but that leaves you with a static, fixed page representation that doesn't care what the upcoming query actually wants.
To fix this query-agnostic mismatch, the authors propose Query-Conditioned Reconstruction (QCR), a training-free framework that splits indexing from retrieval:

  1. Offline Indexing: Pages are clustered into MM groups. Instead of just keeping cluster centers, QCR preserves a low-rank residual basis U∈Rd×rU \in \mathbb{R}^{d \times r} via SVD of within-group residuals, plus group-level diagonal second-order variances Λi\Lambda_i and trust radii τi\tau_i. Crucially, all these auxiliary parameters are strictly accounted for within the byte-level storage budget.
  2. Retrieval-Time Adaptive Instantiation: When a query arrives, QCR uses a fast first-order cosine surrogate to activate only the Top-KK relevant groups per query token. It then analytically solves a 1D optimization problem for the step size β\beta to reconstruct query-adaptive vectors along valid residual directions.
  3. Budgeted MaxSim Coverage: To avoid dumping redundant vectors into the scoring stage, it greedily picks at most B=MB=M vectors from the candidate pool to maximize submodular query token coverage, followed by standard MaxSim scoring.
Critical strengths
  1. Most compression papers in visual doc retrieval just play around with heuristic token pruning or static cluster pooling. Deferring the actual representation instantiation until retrieval time by storing compact residual ellipsoids is genuinely neat.

  2. I think that the framework requires zero additional training or fine-tuning. The derivation of the 1D closed-form optimal step size β0\beta_0 (Eq. 26), the first-order cosine surrogate for group activation (Eq. 4), and framing vector selection as a budgeted submodular coverage problem with a (1−1/e)(1 - 1/e) guarantee make the system mathematically tight and reproducible without introducing messy, fragile hyperparameter grids.

Critical weaknesses
  1. The paper shows an online latency of 10.2 ms for 200 pages (Table 9 and Table 10b). While that looks competitive compared to the uncompressed Full Index on small benchmarks, document retrieval in the real world (or even in standard IR benchmarks like BEIR or large ViDoRe scale-ups) typically searches over 10410^4 to 10610^6 pages, not a few hundred. Because QCR requires running residual-aware activation, on-the-fly SVD vector reconstruction, and greedy candidate selection on every indexed page before scoring, the online overhead scales linearly with corpus size (O(J⋅ops)O(J \cdot \text{ops})). In production, late-interaction engines (e.g., PLAID / ColBERTv2) rely on inverted lists and fast centroid pruning to avoid touching 99% of page vectors. If QCR has to reconstruct representations across all pages, 50,000 pages would take over 2.5 seconds per query!

  2. The paper compares against pruning and merging. However, the real baseline to beat in multi-vector search (like ColBERTv2/PLAID) is residual vector quantization (PQ/SQ) down to 2–4 bits. Without comparing against or combining with PQ under identical byte budgets, it is hard to know if QCR is better than simply quantizing full-resolution vectors.

  3. While the greedy selection in Eq. 18 operates on precomputed inner products, running a sequential BB-step greedy loop per page across thousands of pages on GPU can cause substantial GPU kernel launch overhead and memory thrashing.

  4. Appendix A.1 proposes Overlap@K (Eq. 15) to show the surrogate hˉi(qt)\bar{h}_i(q_t) matches the true SOCP cosine oracle. Yet, the paper never actually reports this overlap number anywhere in the tables or figures.

Questions for authors
  1. How does QCR scale to 10,000+ pages? Can we use static centroids for coarse IVF pruning first (to get Top-100 candidates), and only run QCR reconstruction on those 100 pages?

  2. In Appendix A.1, you introduced Overlap@K to evaluate the first-order surrogate against the exact SOCP oracle (Eq. 15). What is the actual average overlap percentage on ViDoRe v1 and v2?

  3. How sensitive is retrieval quality to the explained-energy threshold η\eta?

  4. Can the residual basis UU and group centers cic_i be quantized into INT8 or INT4?

  5. How long does offline index construction take per 1,000 pages compared to simple k-means clustering? Does running per-page SVD significantly slow down the document ingestion pipeline?

2 comments

We thank the reviewer for the positive assessment of our paper.

R1: Large-scale retrieval and candidate pruning.

Yes, QCR can be integrated with coarse candidate pruning: static centroids first retrieve a shortlist, e.g., the Top-100 pages, followed by QCR reconstruction and reranking within this shortlist. This limits reconstruction to at most 100 pages per query. Overall latency and retrieval quality also depend on the coarse retriever and its candidate coverage. SVD is performed only during offline indexing.

R2: Overlap@KK.

The table compares surrogate and oracle group activation, reporting Overlap@5 between their selected group sets and the corresponding NDCG@5 retention and Recall@5.

DatasetActivation StrategyOverlap@5 (%)NDCG@5 Retention (%)Recall@5 (%)
ViDoRe v1oracle--94.989.4
ViDoRe v1Surrogate86.194.688.9
ViDoRe v2oracle--92.283.5
ViDoRe v2Surrogate81.691.882.6

The surrogate achieves 86.1% and 81.6% Overlap@5 on ViDoRe v1 and v2, respectively. Compared with oracle activation, NDCG@5 retention decreases by 0.3 and 0.4 percentage points, while Recall@5 decreases by 0.5 and 0.9 points. These results suggest that the surrogate maintains retrieval quality despite differences in group selection.

R3: Sensitivity to the energy threshold.

The threshold η\eta controls residual approximation fidelity, but is not a direct measure of retrieval relevance. Under a fixed storage budget, increasing η\eta can retain more residual directions while reducing the number of stored group centers. We therefore vary η\eta while keeping the total byte budget fixed, reallocating capacity between residual directions and group centers for each setting. The table below reports the resulting retrieval quality. We use a fixed-shape setting with N=1024N=1024 and d=128d=128, with the index ratio held at ρ≈0.05\rho\approx0.05 according to the storage formula in the paper. The default value is η=0.90\eta=0.90.

η\etaNDCG@5 Retention (%)Recall@5 (%)
0.7092.987.4
0.8094.188.3
0.9094.688.9
0.9594.288.6
0.9893.988.1

Across the tested thresholds from η=0.80\eta=0.80 to 0.950.95, NDCG@5 retention and Recall@5 vary by at most 0.5 and 0.6 percentage points, respectively. The default η=0.90\eta=0.90 achieves the highest scores among the evaluated settings.

R4: Quantization of the residual basis and group centers.

Yes. Both the residual basis and group centers can be quantized. To isolate quantization effects, we keep the grouping assignments and residual ranks fixed and include quantization metadata in storage accounting. We also compare QCR with PQ and Semantic Clustering with SQ8/SQ4 under the same ρ=0.05\rho=0.05 storage budget, including all auxiliary index data and using identical evaluation settings.

MethodStorage ρ\rhoNDCG@5 Ret. (%)Recall@5 (%)
(a) Quantization sensitivity
QCR (Unquantized) ours0.05094.688.9
QCR (INT8)0.02893.287.5
QCR (INT4)0.01690.485.0
(b) Matched target storage budget
PQ0.05093.287.3
SC + SQ80.05092.186.2
SC + SQ40.05093.687.8

INT8 reduces storage by approximately 44% relative to unquantized QCR, with a 1.4-percentage-point decrease in both metrics, while INT4 incurs larger retrieval losses. At the matched target budget of ρ=0.05\rho=0.05, QCR exceeds SC + SQ4 by 1.0 and 1.1 percentage points in NDCG@5 retention and Recall@5, respectively.

R5: Offline indexing cost.

Table 9 reports offline processing times, including document encoding and index construction, of 0.64 s for QCR and 0.56 s for Clustering on 200 pages. Since pages are indexed independently, approximately linear scaling gives estimated times of 3.2 s and 2.8 s per 1,000 pages, respectively, corresponding to 14.3% additional ingestion time. This overhead includes all additional indexing operations, including SVD. The resulting residual memory is reused across queries, and no SVD is performed during retrieval.

Dear Reviewers,

Please take a moment to review the authors' rebuttal and provide your feedback on whether your concerns have been adequately addressed. Thank you for your time and effort.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.