acceptodds
Under review as a conference paper at ICLR 2027

RetVDQ: Retrieval-Aware Variable-Depth Residual Quantization for Visual Document Retrieval

Abstract

Multi-vector visual document retrieval (VDR) represents each page with multiple embeddings and scores it through MaxSim, which matches each query token to its most similar page embedding. This enables fine-grained retrieval but incurs a large storage cost. Existing compression mainly prunes or merges embeddings, but since different queries rely on different embeddings, removing any of them may discard needed evidence. Quantization keeps every embedding, yet existing quantizers only approximate the original vectors without preserving retrieval rankings, and assign the same precision to all embeddings regardless of their importance. We propose RetVDQ, a retrieval-aware variable-depth residual quantization framework that keeps every embedding and adapts its precision under a fixed budget. RetVDQ learns residual codebooks progressively so that every code prefix preserves the full-precision ranking, and then determines the depth of each embedding by the marginal retrieval utility of additional codes. Both stages are guided by synthetic queries generated offline from each page. The resulting index works with the frozen retriever and its original MaxSim scoring, requiring no model fine-tuning. Experiments on ViDoRe V1 and V2 show that RetVDQ consistently outperforms quantization baselines, and retains 99.3% of full-precision quality on average with only 3.2% of the original storage, far less than token reduction methods need for comparable quality.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
visual document retrieval, multi-vector retrieval, residual quantization, adaptive precision, index compression

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_mU5rOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RetVDQ is a quantization framework for reducing the storage overhead of multi-vector indexes in ColPali-style visual document retrieval. Existing quantizers such as PQ, OPQ, and RQ assign the same precision to every document embedding. Under MaxSim, however, the retrieval contribution of each embedding varies with the query, so uniform precision is not necessarily efficient. The authors propose two main components. First, progressive codebook learning trains the residual codebooks so that each code prefix preserves the full-precision ranking as much as possible. Second, retrieval-aware depth allocation estimates the marginal retrieval utility of adding a code to each embedding and distributes the storage budget greedily. Both stages use offline synthetic queries as supervision, and the retriever backbone remains frozen. RetVDQ achieves higher nDCG@5 than PQ, OPQ, and RQ on ViDoRe V1 and V2, and with ColPali it retains 99.3% of full-precision quality across all 14 subsets while using only 3.2% of the original storage.

Critical strengths
  1. Starting from the observation that embedding importance is query-dependent, the paper replaces uniform precision with variable-depth allocation based on offline estimates of retrieval utility.
  2. A Compression Framework Applicable Without Retriever Fine-tuning.
  3. The ablation in Section 4.3 shows that each component improves performance on its own, and combining them works best. The method does more than add greedy allocation to standard RQ; learning the codebooks themselves in a retrieval-aware way is a positive aspect.
  4. The gains are especially clear under the tight 2B/token budget. On ViDoRe V1 with ColPali, RQ reaches 64.7 nDCG@5 while RetVDQ reaches 74.1. The improvements are also consistent across two backbones and two benchmarks.
Critical weaknesses
  1. Dependence on synthetic query quality needs analysis. Both codebook learning and depth allocation rely on synthetic queries. The limitations section mentions this, but a quantitative analysis is needed of how the number, diversity, and generator of synthetic queries, as well as their gap from the real query distribution, affect final retrieval quality.
  2. Offline index construction cost is not analyzed. Depth allocation recomputes the marginal retrieval utility after each code is added. As the number of embeddings per document and the number of synthetic queries grow, this may require substantial offline computation. The paper demonstrates storage efficiency, but a comparison of the total index construction time, including codebook training and greedy allocation, seems necessary.
  3. Generalization of corpus-specific optimization is unclear. RetVDQ optimizes the codebooks and depth allocation using synthetic queries from the target corpus. This corpus adaptation is a strength, but it is unclear how often the codebooks would need retraining when the corpus changes or new documents are added continuously. In addition, each newly added page requires VLM-generated pseudo queries before depth allocation can be performed.
Questions for authors
  • Comparing with methods such as HPC-ColPali [1] and MarginMerge [2] in future work could further clarify the contribution of RetVDQ.

[1] Duong Bach. Hierarchical Patch Compression for ColPali: Efficient Multi-Vector Document Retrieval with Dynamic Pruning and Quantization. In Proceedings of KDIR, 2025. arXiv:2506.21601.

[2] Coverage Matters: MarginMerge for Compressing Multi-Vector Visual Document Retrievers. arXiv:2608.02969, 2026.

5 comments
AuthorsOpenReview ↗

Response to Reviewer mU5r (1/3)

Thank you for the constructive comments. All revisions are highlighted in the manuscript.

W1: Sensitivity to synthetic-query supervision

We agree that the dependence on synthetic queries needs quantitative analysis. We have added Section 4.5 and Appendix B.3, with experiments on three ViDoRe V1 subsets.

Effect of Pseudo-Query Supervision. To isolate the effect of synthetic-query supervision on depth allocation, we compare Fixed-depth, shuffled, and matched allocation while keeping the same learned codebooks from the first stage. Fixed-depth allocation does not use queries, whereas shuffled allocation randomly assigns each page queries generated for another page. As shown in Table R1 below, our matched allocation outperforms both alternatives across all budgets, showing that depth allocation benefits from synthetic queries and, importantly, their correct correspondence to the target pages. (Table 10 in revised manuscript)

Table R1. Effect of page-query correspondence.

Code Budgets (B/token)246
Fixed-depth67.874.676.3
Shuffled70.474.375.8
Matched (Ours)73.575.976.8

Sensitivity on Query Quantity. We vary the number of page-grounded queries used for depth allocation while keeping the codebooks fixed. As shown in Table R2, even a single page-grounded query provides useful retrieval signals for depth allocation, with further improvements as the number of queries increases to 8. Beyond 8 queries, the performance saturates, which may reflect overlap in the retrieval information provided by additional queries. We thus adopt 8 queries per page as the default setting. (Figure 4 in revised manuscript)

Table R2. Sensitivity to the Number of Queries.

B/token1248121624
271.672.072.773.573.773.774.0
474.774.875.375.976.076.076.0
676.276.276.476.876.776.776.8

Sensitivity on Query Generator. To assess the impact of the query generator, we replace Qwen3-VL-8B-Instruct with Qwen2.5-VL-7B or InternVL3.5-4B. As shown in the table below, the two alternatives lower macro-averaged nDCG@5 by at most 1.7 and 0.9 points, respectively. Even with the smallest query-generation backbone, InternVL3.5-4B, the gains over the strongest baseline, RQ, are 8.0, 2.6, and 1.4 points at 2, 4, and 6 B/token, respectively. These results demonstrate the robustness of RetVDQ to the choice of query generator. (Table 7 in revised manuscript)

Table R3. Sensitivity to the Query Generator.

Method246
Qwen3-VL-8B73.575.976.8
Qwen2.5-VL-7B71.875.275.8
InternVL3.5-4B72.675.676.3
RQ64.673.074.9

Sensitivity on Query Intent Coverage. Our synthetic queries cover four main intents: Extractive (retrieving specific information), Numerical (numerical reasoning), Comparative (comparing items), and Enumerative (listing multiple items), accounting for 63.14%, 10.10%, 18.14%, and 1.31% of queries on average, respectively. To evaluate the effect of incomplete intent coverage, we remove each of the four query intents (Extractive, Numerical, Comparative, and Enumerative) from training while keeping them in the test set. As shown in Table R4, excluding individual intents causes only small performance drops, which become smaller at higher storage budgets. These results suggest that RetVDQ can preserve retrieval-relevant information even when some query intents are absent from the synthetic supervision. (Table 9 in revised manuscript)

Table R4. Sensitivity to Query Intent.

Code Budgets (B/token)246
w/o Numerical72.575.776.4
w/o Comparative72.575.376.3
w/o Enumerative73.275.876.4
w/o Extractive73.175.776.3
Ours73.575.976.8

Query Distribution Gap. The official benchmark queries are independently provided, with most derived from existing QA datasets and a small portion generated by LLMs. None of them are used by our query generator or during compression. Thus, all reported retrieval results are evaluated on real queries that are independent of the synthetic supervision. Furthermore, the intent-removal experiments above deliberately introduce distribution shifts in query intents between synthetic training queries and real test queries. The relatively small performance drops suggest that RetVDQ is robust to such shifts in query intent distributions.

AuthorsOpenReview ↗

Response to Reviewer mU5r (2/3)

Changes in the paper: We have added Section 4.5, including Figure 4 and Table 7. Furthermore, to present the additional sensitivity analyses, we have added two experiments in Appendix B.3, with the results reported in Tables 9 and 10.

W2: Offline index construction cost and computational efficiency.

We agree that offline construction cost is an important consideration. The original submission includes a component-wise runtime analysis in Appendix C.6 (Table 12 in the revised manuscript). All reported timings were measured using a single NVIDIA L40S GPU. We summarize the results here to clarify the main sources of computational cost. On 500 InfoVQA pages, each containing 1,030 embeddings, standard RQ takes 1.8 minutes in total. RetVDQ additionally requires 1.84 hours for synthetic query generation, 123.5 minutes for progressive codebook learning, 7.9 minutes for joint codebook learning, and 1.8 minutes for greedy depth allocation. Encoding takes 0.03 minutes for both methods.

Most of the additional cost comes from synthetic query generation and codebook learning. Codebooks are learned only once per corpus, while query generation and depth allocation are performed independently for each page and can therefore be parallelized across pages to accelerate offline index construction. Notably, depth allocation itself takes only approximately 0.2 seconds per page, despite recomputing utilities after each code. Importantly, all additional computations are performed during offline index construction and do not need to be repeated for individual queries. Online retrieval retains the original query encoder and MaxSim scoring, requiring only an additional code-decoding step. Moreover, with ColPali, RetVDQ consistently outperforms RQ under matched code budgets, achieving maximum nDCG@5 improvements of 9.4 and 5.8 points on ViDoRe V1 and V2, respectively. From another perspective, at 4 B/token, RetVDQ matches or surpasses RQ at 7 B/token on both datasets, reducing code storage by approximately 43%. Overall, RetVDQ’s additional one-time indexing cost translates into improved retrieval quality under the same storage budget, substantially reduced storage for comparable retrieval quality, and minimal additional online overhead.

Table 12 (in the revised manuscript). Offline index construction cost on a representative ViDoRe V1 workload. Times are measured from cached document embeddings and exclude document encoding by the retriever.

ComponentTime
Synthetic query generation1.84 h
RQ codebook learning1.76 min
RQ encoding0.03 min
RetVDQ progressive learning123.54 min
RetVDQ joint optimization7.88 min
RetVDQ depth allocation1.84 min
RetVDQ encoding0.03 min

Reference in the paper: Appendix C.6 and Table 12 (in the revised manuscript).

W3: Corpus adaptation and incremental indexing.

Thank you for raising this important concern. We agree that corpus-specific optimization raises a practical consideration when the corpus is continuously updated. To address this, RetVDQ separates corpus-level codebook learning from page-level depth allocation. For newly added pages, the existing codebooks can be directly reused, and only synthetic query generation and depth allocation are required. This incremental process takes approximately 13 seconds per page in our setting, with query generation accounting for most of the time. Existing index entries remain unchanged.

Codebook retraining is therefore unnecessary for routine document additions and is expected only when the corpus distribution shifts substantially, for example, when a large number of documents from a substantially different domain or document type are introduced. Thus, the need for retraining depends on the extent of distribution shift rather than the frequency of document additions. This is consistent with other learned quantizers, such as QINCo2 (ICLR 2025) and CrossQ (ICML 2026), whose codebooks are also fitted to the target corpus.

Changes in the paper: We have revised the Limitations paragraph.

AuthorsOpenReview ↗

Response to Reviewer mU5r (3/3)

Q1: Comparisons with additional VDR compression methods.

Thank you for these references. We have added MarginMerge to our storage–quality comparison on both ColPali and ColQwen2.5 (Figure 3). RetVDQ achieves similar or higher nDCG@5 with much less storage. Selected results are shown in Tables R7 and R8.

As shown in Tables R7 and R8, RetVDQ consistently achieves higher FP retention than MarginMerge across all three compression levels and both backbones. The advantage is especially pronounced under high compression: RetVDQ retains 98.6% and 98.7% of full-precision performance using only 3.6% and 3.3% storage, respectively, whereas MarginMerge requires approximately 10% storage to retain 96.3% and 97.3%. These results demonstrate that RetVDQ provides a more favorable storage–quality trade-off across both backbones.

Table R7. Storage–Accuracy Trade-off on ViDoRe V1 subsets with ColPali Backbone.

ComparisonMethodRemaining Storage (%)FP Retention (%)
Low storageRetVDQ2.094.8
MarginMerge2.186.5
Medium storageRetVDQ2.898.0
MarginMerge3.090.3
High compressionRetVDQ3.698.6
MarginMerge10.096.3

Table R8. Storage–Accuracy Trade-off on ViDoRe V1 subsets with ColQwen2.5 Backbone.

ComparisonMethodRemaining Storage (%)FP Retention (%)
Low storageRetVDQ1.895.3
MarginMerge2.186.9
Medium storageRetVDQ2.597.5
MarginMerge3.190.5
High compressionRetVDQ3.398.7
MarginMerge10.197.3

HPC-ColPali. Since HPC-ColPali combines two distinct modules, quantization and attention-guided token pruning, a fair comparison should isolate its quantization component. This component assigns a single code to each embedding, essentially corresponding to single-level RQ, which is already included in our baselines. Therefore, its quantization component has already been covered in our comparisons. Its pruning component further reduces the number of embeddings, which is complementary to our adaptive depth allocation. Combining token pruning with adaptive depth allocation is a promising direction for future work, as noted in our Limitations.

Changes in the paper: We have added MarginMerge to Figure 3 and its discussion in Section 4.2, and added both MarginMerge and HPC-ColPali to the related work in Appendix A.2.

We sincerely hope our responses have addressed all your concerns, and we would be grateful if you would consider reflecting this in your final rating. If you have any further questions, please feel free to follow up, and we are glad to address them.

Reviewer_BxT2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes RetVDQ, a retrieval-aware quantization method for visual document retrieval. Instead of removing document embeddings, it assigns different quantization depths based on their retrieval importance. The method combines progressive codebook learning with a greedy depth allocation strategy, both guided by synthetic queries. Experiments on ViDoRe V1 and V2 show strong compression-performance tradeoffs.

Critical strengths
  1. Interesting. Allocating quantization precision based on retrieval importance rather than reconstruction error is intuitive and well motivated, especially for MaxSim-based retrieval. I like the overall approach.

  2. Strong results and clear presentation. The paper is well written and easy to follow. The improvements over standard quantization baselines are substantial and consistent across benchmarks and backbones. The ablations also support the effectiveness of the main components.

Critical weaknesses
  1. The contribution of pseudo-query supervision is not sufficiently isolated. The method uses corpus-specific synthetic queries for both codebook learning and depth allocation. While the ablations separate these components, it is unclear how much of the improvement comes from the proposed quantization scheme versus the additional retrieval supervision. This aspect needs to be isolated carefully to prove the paper's claims.

  2. Missing comparisons with stronger quantization methods. ColBERT-style retrieval has a substantial body of work on efficient quantization, including ColBERTv2, PLAID and WARP. These are not experimentally compared. A comparison with retrieval-aware quantization methods such as CrossQ seems important for establishing the novelty and effectiveness of the proposed approach.

Questions for authors
  1. Can the authors compare against existing retrieval-aware quantization methods under the same backbone, storage budget, and synthetic-query supervision? This would help clarify where the gains come from.

  2. How sensitive are the results to the number and quality of synthetic queries? Can the authors better isolate the effect of this supervision from the quantization method itself?

  3. The paper mentions that code will be released, but no implementation is currently provided. Can the authors make the code available to support reproducibility?

Overall, I like the paper and find the results promising. However, I would like to see these issues addressed before leaning toward acceptance.

7 comments
AuthorsOpenReview ↗

Response to Reviewer BxT2 (1/3)

Thank you for the helpful suggestions. We respond to each comment below, and the corresponding revisions are highlighted in the manuscript.

W1 and Q2: Isolating the Contribution and Evaluating the Robustness of Pseudo-Query Supervision.

We agree that the contribution of pseudo-query supervision needs to be carefully isolated from that of the proposed quantization scheme. We therefore revisit our existing ablation results to clarify their respective contributions, and conduct additional experiments to further investigate the sensitivity and robustness of pseudo-query supervision.

1. Revisiting Existing Ablations and Further Isolating Contributions

To clarify this point, we revisit the existing ablation results in Table 5 of the original submission. Specifically, variant (c) applies the same synthetic queries and joint retrieval loss to standard RQ codebook learning, while retaining a fixed depth for every token. Thus, it incorporates our retrieval supervision into a standard quantizer without introducing the proposed variable-depth quantization scheme. Compared with standard RQ, this additional supervision brings 2.2, 1.6 and 0.7 points at 2, 4 and 6 bytes per token. With the same retrieval-aware supervision, RetVDQ further improves over (c) by 6.7, 1.3 and 1.2 points, demonstrating the substantial contribution of the proposed quantization scheme, particularly under tight storage budgets.

To further isolate the effect of synthetic-query supervision on depth allocation, we compare Fixed-depth, shuffled, and matched allocation while keeping the same learned codebooks from the first stage. Fixed-depth allocation does not use queries, whereas shuffled allocation randomly assigns each page queries generated for another page. As shown in Table R1 below, our matched allocation outperforms both alternatives across all budgets, showing that depth allocation benefits from synthetic queries and, importantly, their correct correspondence to the target pages. (Table 10 in revised manuscript)

Table R1. Effect of page-query correspondence.

Code Budgets (B/token)246
Fixed-depth67.874.676.3
Shuffled70.474.375.8
Matched (Ours)73.575.976.8

2. Contribution of Dynamic Depth Allocation.

To further isolate the contribution of dynamic depth allocation, we compare it with static utility-guided allocation (the new variant g) using the same learned codebooks and synthetic queries. Our dynamic allocation outperforms static allocation by 1.3, 1.3 and 0.3 points, and outperforms random and reconstruction-based allocation by larger margins (Table 5). The relevant results are shown below.

VariantDepth allocation246
(e)Random70.273.775.5
(f)Reconstruction-guided70.674.375.8
(g)Static utility-guided72.274.676.5
OursDynamic utility-guided73.575.976.8

3. Sensitivity and Robustness of Pseudo-Query Supervision

To further assess the robustness of pseudo-query supervision, we conduct additional experiments on three ViDoRe V1 subsets, examining its sensitivity to query quantity and generation models, as well as its robustness to query intent coverage.

On Query Quantity. We vary the number of page-grounded queries used for depth allocation while keeping the codebooks fixed. As shown in Table R2, even a single page-grounded query provides useful retrieval signals for depth allocation, with further improvements as the number of queries increases to 8. Beyond 8 queries, the performance saturates, which may reflect overlap in the retrieval information provided by additional queries. We thus adopt 8 queries per page as the default setting. (Figure 4 in revised manuscript)

Table R2: Sensitivity to the Number of Queries.

B/token1248121624
271.672.072.773.573.773.774.0
474.774.875.375.976.076.076.0
676.276.276.476.876.776.776.8
AuthorsOpenReview ↗

Response to Reviewer BxT2 (2/3)

On Query Generator. To assess the impact of the query generator, we replace Qwen3-VL-8B-Instruct with Qwen2.5-VL-7B or InternVL3.5-4B. As shown in the table below, the two alternatives lower macro-averaged nDCG@5 by at most 1.7 and 0.9 points, respectively. Even with the smallest query-generation backbone, InternVL3.5-4B, the gains over the strongest baseline, RQ, are 8.0, 2.6, and 1.4 points at 2, 4, and 6 B/token, respectively. These results demonstrate the robustness of RetVDQ to the choice of query generator. (Table 7 in revised manuscript)

Table R3. Sensitivity to the Query Generator.

Method246
Qwen3-VL-8B (Ours)73.575.976.8
Qwen2.5-VL-7B71.875.275.8
InternVL3.5-4B72.675.676.3
RQ64.673.074.9

On Query Intent Coverage. Our synthetic queries cover four main intents: Extractive (retrieving specific information), Numerical (numerical reasoning), Comparative (comparing items), and Enumerative (listing multiple items), accounting for 63.14%, 10.10%, 18.14%, and 1.31% of queries on average, respectively. To evaluate the effect of incomplete intent coverage, we remove each of the four query intents (Extractive, Numerical, Comparative, and Enumerative) from training while keeping them in the test set. As shown in Table R4, excluding individual intents causes only small performance drops, which become smaller at higher storage budgets. These results suggest that RetVDQ can preserve retrieval-relevant information even when some query intents are absent from the synthetic supervision. (Table 9 in revised manuscript)

Table R4. Sensitivity to Query Intent.

Code Budgets (B/token)246
w/o Numerical72.575.776.4
w/o Comparative72.575.376.3
w/o Enumerative73.275.876.4
w/o Extractive73.175.776.3
Ours73.575.976.8

Changes in the paper: We have added the static allocation variant (g) to Table 5 and discussed it in Section 4.3. We have also added Section 4.5 with Tables 7 and Figure 4. Two experiments have been added to Appendix B.3.

W2 and Q1: Comparisons with retrieval-aware and efficient quantization baselines.

Thanks for suggesting stronger baselines. We have added CrossQ (ICML 2026), ColBERTv2 (ACL 2022), and MarginMerge (arXiv 2026) to our evaluation.

Comparison with CrossQ. CrossQ is the closest to our setting, since it also learns the quantizer with retrieval supervision. We adapt it to visual document retrieval and train it with the same backbones, code budgets, and synthetic queries as RetVDQ. As shown in Tables R5 and R6, RetVDQ outperforms CrossQ at all budgets on both ViDoRe V1 and V2 and with both backbones. For example, with ColPali at 2 bytes per token, RetVDQ is ahead by 45.9 points on V1 and 34.6 points on V2. CrossQ even performs below standard RQ in our setting. A likely reason is that evidence on a document page is spread across many local patches, and different queries rely on different patches. CrossQ encodes each token based on other tokens in the document, which can blur such local evidence. RetVDQ encodes each token separately and gives more codes to tokens that matter for retrieval.

AuthorsOpenReview ↗

Response to Reviewer BxT2 (3/3)

Comparison with ColBERTv2. For the ColBERT line, ColBERTv2, PLAID, and WARP address different aspects of efficiency. ColBERTv2 introduces residual compression to reduce index storage, which matches the goal of RetVDQ. PLAID and WARP primarily focus on retrieval-engine efficiency rather than developing new fixed-budget quantization methods. Specifically, PLAID directly adopts ColBERTv2's residual compression, making our ColBERTv2 baseline representative of its underlying compression scheme; its additional contributions lie in candidate pruning and optimized scoring. WARP similarly focuses on accelerating XTR-based retrieval through efficient candidate selection, implicit decompression, and score aggregation, rather than introducing a retrieval-aware quantization objective comparable to RetVDQ. We therefore focus our experiments on directly comparable compression methods including ColBERTv2 and discuss PLAID and WARP as complementary retrieval-engine optimizations. As shown in Tables R5 and R6, RetVDQ outperforms both baselines at all budgets, for example by 23.0 and 14.3 points on ViDoRe V1 and V2 with ColPali at 2 bytes per token.

Table R5: ViDoRe V1: Macro-averaged nDCG@5 across code budgets (B/token).

Method234567
ColBERTv251.163.669.172.274.375.9
CrossQ28.254.865.570.572.774.7
RetVDQ74.177.579.180.079.880.0

Table R6:ViDoRe V2: Macro-averaged nDCG@5 across code budgets (B/token).

Method234567
ColBERTv236.742.643.945.247.549.4
CrossQ16.433.741.142.644.243.6
RetVDQ51.052.054.254.154.354.5

Comparison with MarginMerge. MarginMerge reduces the number of embeddings and cannot be set to the same code budget, so we compare it with all methods by plotting retrieval quality against index size on six ViDoRe V1 subsets (Figure 3). On both ColPali and ColQwen2.5, RetVDQ achieves comparable or higher nDCG@5 with substantially less storage. Representative results are reported in Tables R7 and R8 for ColPali and ColQwen2.5, respectively.

Table R7. Storage–Accuracy Trade-off on ViDoRe V1 subsets with ColPali Backbone.

ComparisonMethodRemaining Storage (%)FP Retention (%)
Low storageRetVDQ2.094.8
MarginMerge2.186.5
Medium storageRetVDQ2.898.0
MarginMerge3.090.3
High compressionRetVDQ3.698.6
MarginMerge10.096.3

Table R8. Storage–Accuracy Trade-off on ViDoRe V1 subsets with ColQwen2.5 Backbone.

ComparisonMethodRemaining Storage (%)FP Retention (%)
Low storageRetVDQ1.895.3
MarginMerge2.186.9
Medium storageRetVDQ2.597.5
MarginMerge3.190.5
High compressionRetVDQ3.398.7
MarginMerge10.197.3

Changes in the paper: We have added CrossQ and ColBERTv2 to Tables 1–3, Figure 3 and to the Baselines paragraph in Section 4.1, and discussed their results in Section 4.2. We have also added MarginMerge to Figure 3 and its discussion in Section 4.2, and added PLAID and WARP to the related work in Appendix A.3.

Q3: Code availability and reproducibility.

The anonymous code is available at https://anonymous.4open.science/r/RetVDQ-A680. It includes the full implementation, training and evaluation scripts, configurations, and instructions for reproducing the main results.

We sincerely hope our responses and additional experiments have addressed all your concerns, and we would greatly appreciate it if you could reconsider your rating toward a positive score. If you have any further questions, please feel free to follow up, and we are glad to address them.

Reviewer_82avOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RetVDQ learns residual codebooks with retrieval supervision at every prefix, then allocates extra codes using synthetic page queries. It reports strong compression on two ViDoRe suites while keeping the retriever frozen. The results are promising, but the storage format and allocation objective need closer justification.

Critical strengths
  1. Training intermediate prefixes directly fits the variable-depth setting well.
  2. The component ablations show that codebook learning and utility-based allocation each help, especially at tight budgets.
  3. Preserving the frozen query encoder and MaxSim interface makes the method relatively easy to integrate.
Critical weaknesses
  1. Appendix B.1 says depths are implicit, but a stream of variable-length byte codes still needs token boundaries. Equation 10 counts neither lengths nor offsets. Even a packed three-bit depth adds nearly 19 percent at a two-byte budget. The compression claim needs a serialized, independently decodable index with all metadata counted.
  2. Equation 9 rewards increasing a positive page score, rather than reducing its error or preserving a ranking. It can favor an overshoot and penalize a refinement that corrects an overestimated match. With unnormalized reconstructions, norm inflation is another concern. Compare against allocation based on full-precision score error or positive-negative margins.
  3. Eight filtered synthetic queries determine each page's precision, but their coverage is not stress-tested. Rare table cells or unseen query intents may receive only the base code. Query-count, generator, and held-out intent controls would show whether the gains survive this dependency.
Questions for authors

What exact byte layout recovers every token boundary? Does the allocation advantage remain after metadata is counted and the utility is changed to preserve margins?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 82av (1/2)

Thank you for the constructive comments. All revisions are highlighted in the manuscript.

W1 and Q1: Index serialization and metadata accounting

Index serialization. We have revised Appendix B.1 to describe the construction and decoding of an independently decodable index. We group the embeddings of each page by their retained depths in ascending order and store the codes of each group contiguously. The number of embeddings in each group, together with its depth, uniquely determines the boundaries of all embeddings in the code stream; we refer to these group counts as the depth histogram. Since MaxSim is invariant to the ordering of document embeddings, this reordering does not affect retrieval scores and requires no restoration of the original order. To minimize metadata, we enumerate all valid histograms under the given page size and code budget, and store only the index of the histogram corresponding to RetVDQ's final depth allocation. Since the enumeration follows a predefined numerical ordering, it can be reproduced automatically during decoding and requires no additional storage. Given the recovered histogram, the decoder can identify the boundaries of every embedding in the code stream.

Metadata accounting. We explicitly include the histogram metadata in the storage accounting in Eq. (10). It requires only 5–6 bytes per page, corresponding to 0.243%, 0.146%, and 0.097% of the code storage at budgets of 2, 4, and 6 B/token, respectively. As reported in Table R9, after including this metadata, the total index storage is only 1.3%, 2.0%, and 2.8% of the full-precision index storage at these budgets (Table 4 in the revised manuscript). Thus, the metadata overhead is negligible and does not materially affect the compression advantage. Even with this overhead included, RetVDQ outperforms uniform RQ by 9.4 nDCG@5 points on ViDoRe V1 with ColPali at 2 B/token.

Table R9. Depth metadata storage and the resulting remaining document-index storage on ViDoRe.

Code Budget (B/token)234567
Metadata (Bytes/page)566665
Remaining Storage (%)1.2601.6512.0422.4332.8233.214

Changes in the paper: We have added the depth-grouped layout, the metadata encoding, and the updated storage accounting to Appendix B.1 (Eq. 10 and Table 8).

W2: Choice of allocation utility and retrieval performance.

Comparison with margin-preserving allocation. We agree that directly increasing positive-page scores does not explicitly control their deviations from full-precision scores or preserve positive-negative margins. To examine this concern, we compare our allocation utility with a margin-preserving alternative on three ViDoRe V1 subsets: TabFQuAD, TAT-DQA, and InfoVQA. Specifically, following Eq. (4), the alternative defines the allocation objective as the average Smooth L1 loss between quantized and full-precision positive-negative margins over synthetic triplets, and measures the utility of an additional code by the reduction in this loss.

As shown in Table R10, our utility outperforms margin-preserving allocation across all nine dataset-budget settings, with macro-averaged gains of 10.6, 9.0, and 1.9 nDCG@5 points at 2, 4, and 6 B/token, respectively. The advantage is particularly pronounced under tighter budgets. This difference stems from how the two utilities prioritize additional codes. Our utility favors codes that yield larger improvements in positive-page scores, whereas margin-preserving allocation favors those that reduce deviations from full-precision margins. Since hard negatives are the top-ranked non-relevant pages under full precision, their margins over positive pages can be small or even negative due to ranking errors in the full-precision retriever. Consequently, matching these margins may penalize additional codes that improve positive-page scores but increase deviations from the target margins. Our utility instead rewards these improvements directly.

Table R10. nDCG@5 of different allocation utilities.

DatasetB/tokenOursMargin-preserving
TabFQuAD2 / 4 / 683.4 / 85.0 / 85.479.8 / 82.6 / 83.8
TAT-DQA2 / 4 / 660.8 / 63.6 / 64.845.5 / 49.6 / 63.2
InfoVQA2 / 4 / 676.3 / 79.3 / 80.263.3 / 68.7 / 77.7
Average2 / 4 / 673.5 / 75.9 / 76.862.8 / 67.0 / 74.9

L2 Normalization and Norm Inflation. Regarding the concern about norm inflation, all reconstructed embeddings are L2-normalized before MaxSim computation in both allocation and retrieval. Hence, the norm of the reconstructed embedding cannot be increased to obtain a larger MaxSim score, since the actual scoring vector always has unit norm. Therefore, norm inflation cannot explain the observed gains.

AuthorsOpenReview ↗

Response to Reviewer 82av (2/2)

Changes in the paper: We have specified L2 normalization in Eq. (11) of Appendix B.1, and added this discussion to Section 4.4 (Table 6).

W3: Robustness to synthetic-query coverage.

We agree that a few synthetic queries may not cover all needs of a page, and test this dependency in Section 4.5 and Appendix B.3 on three ViDoRe V1 subsets.

Effect of Pseudo-Query Supervision. To isolate the effect of synthetic-query supervision on depth allocation, we compare Fixed-depth, shuffled, and matched allocation while keeping the same learned codebooks from the first stage. Fixed-depth allocation does not use queries, whereas shuffled allocation randomly assigns each page queries generated for another page. As shown in Table R1 below, our matched allocation outperforms both alternatives across all budgets, showing that depth allocation benefits from synthetic queries and, importantly, their correct correspondence to the target pages. (Table 10 in revised manuscript)

Table R1. Effect of page-query correspondence.

Code Budgets (B/token)246
Fixed-depth67.874.676.3
Shuffled70.474.375.8
Matched (Ours)73.575.976.8

Sensitivity on Query Quantity. We vary the number of page-grounded queries used for depth allocation while keeping the codebooks fixed. As shown in Table R2, even a single page-grounded query provides useful retrieval signals for depth allocation, with further improvements as the number of queries increases to 8. Beyond 8 queries, the performance saturates, which may reflect overlap in the retrieval information provided by additional queries. We thus adopt 8 queries per page as the default setting. (Figure 4 in revised manuscript)

Table R2. Sensitivity to the Number of Queries.

B/token1248121624
271.672.072.773.573.773.774.0
474.774.875.375.976.076.076.0
676.276.276.476.876.776.776.8

Sensitivity on Query Generator. To assess the impact of the query generator, we replace Qwen3-VL-8B-Instruct with Qwen2.5-VL-7B or InternVL3.5-4B. As shown in the table below, the two alternatives lower macro-averaged nDCG@5 by at most 1.7 and 0.9 points, respectively. Even with the smallest query-generation backbone, InternVL3.5-4B, the gains over the strongest baseline, RQ, are 8.0, 2.6, and 1.4 points at 2, 4, and 6 B/token, respectively. These results demonstrate the robustness of RetVDQ to the choice of query generator. (Table 7 in revised manuscript)

Table R3. Sensitivity to the Query Generator.

Method246
Qwen3-VL-8B(Ours)73.575.976.8
Qwen2.5-VL-7B71.875.275.8
InternVL3.5-4B72.675.676.3
RQ64.673.074.9

Sensitivity on Query Intent Coverage. Our synthetic queries cover four main intents: Extractive (retrieving specific information), Numerical (numerical reasoning), Comparative (comparing items), and Enumerative (listing multiple items), accounting for 63.14%, 10.10%, 18.14%, and 1.31% of queries on average, respectively. To evaluate the effect of incomplete intent coverage, we remove each of the four query intents (Extractive, Numerical, Comparative, and Enumerative) from training while keeping them in the test set. As shown in Table R4, excluding individual intents causes only small performance drops, which become smaller at higher storage budgets. These results suggest that RetVDQ can preserve retrieval-relevant information even when some query intents are absent from the synthetic supervision. (Table 9 in revised manuscript)

Table R4. Sensitivity to Query Intent.

Code Budgets (B/token)246
w/o Numerical72.575.776.4
w/o Comparative72.575.376.3
w/o Enumerative73.275.876.4
w/o Extractive73.175.776.3
Ours73.575.976.8

Changes in the paper: We have added Section 4.5 with Figure 4 and Tables 7, Appendix B.3 with Table 9-10.

We sincerely hope our responses and additional experiments have addressed all your concerns, and we would greatly appreciate it if you could reconsider your rating toward a positive score. If you have any further questions, please feel free to follow up, and we are glad to address them.

Dear Reviewer 82av,

With less than 24 hours remaining in the discussion period, we would greatly appreciate your feedback on our rebuttal and additional experiments. Please let us know whether our responses affect your assessment or if any concerns remain unresolved, so that we can provide further clarification before the deadline.

Thank you again for your time and consideration. Even a brief update would be very helpful.

Sincerely,

The Authors

Dear Reviewer,

With less than 6 hours remaining in the discussion period, we would like to follow up on our rebuttal. We have made every effort to respond thoroughly to your comments and questions, but have not yet received your feedback. We would greatly appreciate even a brief update on whether our responses affect your assessment or whether any concerns remain unresolved.

We sincerely appreciate your time and attention.

Sincerely,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the Rebuttal and Discussion

We sincerely thank the Area Chair for their time and all reviewers for their constructive feeadback. In response, we have addressed the concerns raised during the review process through additional experiments, baseline comparisons, and methodological clarifications. All corresponding revisions are highlighted in blue in the updated manuscript.

As of comment, we are encouraged by the positive follow-up assessments from Reviewer mU5r. We appreciate Reviewer BxT2’s positive initial impression of our work and his willingness to further consider our responses and additional experiments. We have not received follow-up feedback from Reviewer 82av yet.

The main additions and revisions are summarized below:

  • Reviewer 82av: Clarification of the independently decodable index format and complete metadata accounting; comparison with margin-preserving allocation and clarification of L2 normalization; additional analyses of the sensitivity on synthetic-query quantity, generator, intent coverage, and page-query correspondence.
  • Reviewer BxT2: Isolation of the contributions of retrieval supervision and dynamic depth allocation; additional comparisons with CrossQ (ICML 2026), ColBERTv2 (ACL 2022) and MarginMerge (arXiv 2026) under matched settings, and discussion of PLAID and WARP; release of the anonymous code implementation, training and evaluation scripts, and configurations.
  • Reviewer mU5r: Additional analyses of synthetic-query robustness and query distribution shifts; offline index construction cost analysis and clarification of corpus adaptation and incremental indexing; comparison with MarginMerge and clarification of the relationship to HPC-ColPali.

As contributions of RetVDQ:

  • To the best of our knowledge, RetVDQ is the first framework to vary quantization precision across embeddings in multi-vector VDR. By keeping every embedding, it naturally avoids the information loss of token pruning and merging.
  • We design retrieval-aware progressive codebook learning and residual-depth allocation, which ensure that every code prefix remains effective for retrieval and every bit is spent where it matters most for MaxSim ranking.
  • Extensive experiments on 14 subsets of ViDoRe V1 and V2 with two backbones show that RetVDQ consistently outperforms five quantization methods including ColBERTv2 (ACL 2022) and CrossQ (ICML 2026) and three token reduction methods including MarginMerge (arXiv 2026) under various storage budgets.

Point-to-point responses can be found below. We sincerely appreciate the reviewers' constructive feedback and are grateful to the Area Chair for their time and efforts in overseeing the review process. We hope our additional experiments and revisions help clarify the contributions and effectiveness of RetVDQ.

Sincerely,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.