- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
RetVDQ is a quantization framework for reducing the storage overhead of multi-vector indexes in ColPali-style visual document retrieval. Existing quantizers such as PQ, OPQ, and RQ assign the same precision to every document embedding. Under MaxSim, however, the retrieval contribution of each embedding varies with the query, so uniform precision is not necessarily efficient. The authors propose two main components. First, progressive codebook learning trains the residual codebooks so that each code prefix preserves the full-precision ranking as much as possible. Second, retrieval-aware depth allocation estimates the marginal retrieval utility of adding a code to each embedding and distributes the storage budget greedily. Both stages use offline synthetic queries as supervision, and the retriever backbone remains frozen. RetVDQ achieves higher nDCG@5 than PQ, OPQ, and RQ on ViDoRe V1 and V2, and with ColPali it retains 99.3% of full-precision quality across all 14 subsets while using only 3.2% of the original storage.
Critical strengths
- Starting from the observation that embedding importance is query-dependent, the paper replaces uniform precision with variable-depth allocation based on offline estimates of retrieval utility.
- A Compression Framework Applicable Without Retriever Fine-tuning.
- The ablation in Section 4.3 shows that each component improves performance on its own, and combining them works best. The method does more than add greedy allocation to standard RQ; learning the codebooks themselves in a retrieval-aware way is a positive aspect.
- The gains are especially clear under the tight 2B/token budget. On ViDoRe V1 with ColPali, RQ reaches 64.7 nDCG@5 while RetVDQ reaches 74.1. The improvements are also consistent across two backbones and two benchmarks.
Critical weaknesses
- Dependence on synthetic query quality needs analysis. Both codebook learning and depth allocation rely on synthetic queries. The limitations section mentions this, but a quantitative analysis is needed of how the number, diversity, and generator of synthetic queries, as well as their gap from the real query distribution, affect final retrieval quality.
- Offline index construction cost is not analyzed. Depth allocation recomputes the marginal retrieval utility after each code is added. As the number of embeddings per document and the number of synthetic queries grow, this may require substantial offline computation. The paper demonstrates storage efficiency, but a comparison of the total index construction time, including codebook training and greedy allocation, seems necessary.
- Generalization of corpus-specific optimization is unclear. RetVDQ optimizes the codebooks and depth allocation using synthetic queries from the target corpus. This corpus adaptation is a strength, but it is unclear how often the codebooks would need retraining when the corpus changes or new documents are added continuously. In addition, each newly added page requires VLM-generated pseudo queries before depth allocation can be performed.
Questions for authors
- Comparing with methods such as HPC-ColPali [1] and MarginMerge [2] in future work could further clarify the contribution of RetVDQ.
[1] Duong Bach. Hierarchical Patch Compression for ColPali: Efficient Multi-Vector Document Retrieval with Dynamic Pruning and Quantization. In Proceedings of KDIR, 2025. arXiv:2506.21601.
[2] Coverage Matters: MarginMerge for Compressing Multi-Vector Visual Document Retrievers. arXiv:2608.02969, 2026.
5 comments
Response to Reviewer mU5r (2/3)
Changes in the paper: We have added Section 4.5, including Figure 4 and Table 7. Furthermore, to present the additional sensitivity analyses, we have added two experiments in Appendix B.3, with the results reported in Tables 9 and 10.
W2: Offline index construction cost and computational efficiency.
We agree that offline construction cost is an important consideration. The original submission includes a component-wise runtime analysis in Appendix C.6 (Table 12 in the revised manuscript). All reported timings were measured using a single NVIDIA L40S GPU. We summarize the results here to clarify the main sources of computational cost. On 500 InfoVQA pages, each containing 1,030 embeddings, standard RQ takes 1.8 minutes in total. RetVDQ additionally requires 1.84 hours for synthetic query generation, 123.5 minutes for progressive codebook learning, 7.9 minutes for joint codebook learning, and 1.8 minutes for greedy depth allocation. Encoding takes 0.03 minutes for both methods.
Most of the additional cost comes from synthetic query generation and codebook learning. Codebooks are learned only once per corpus, while query generation and depth allocation are performed independently for each page and can therefore be parallelized across pages to accelerate offline index construction. Notably, depth allocation itself takes only approximately 0.2 seconds per page, despite recomputing utilities after each code. Importantly, all additional computations are performed during offline index construction and do not need to be repeated for individual queries. Online retrieval retains the original query encoder and MaxSim scoring, requiring only an additional code-decoding step. Moreover, with ColPali, RetVDQ consistently outperforms RQ under matched code budgets, achieving maximum nDCG@5 improvements of 9.4 and 5.8 points on ViDoRe V1 and V2, respectively. From another perspective, at 4 B/token, RetVDQ matches or surpasses RQ at 7 B/token on both datasets, reducing code storage by approximately 43%. Overall, RetVDQ’s additional one-time indexing cost translates into improved retrieval quality under the same storage budget, substantially reduced storage for comparable retrieval quality, and minimal additional online overhead.
Table 12 (in the revised manuscript). Offline index construction cost on a representative ViDoRe V1 workload. Times are measured from cached document embeddings and exclude document encoding by the retriever.
| Component | Time |
|---|---|
| Synthetic query generation | 1.84 h |
| RQ codebook learning | 1.76 min |
| RQ encoding | 0.03 min |
| RetVDQ progressive learning | 123.54 min |
| RetVDQ joint optimization | 7.88 min |
| RetVDQ depth allocation | 1.84 min |
| RetVDQ encoding | 0.03 min |
Reference in the paper: Appendix C.6 and Table 12 (in the revised manuscript).
W3: Corpus adaptation and incremental indexing.
Thank you for raising this important concern. We agree that corpus-specific optimization raises a practical consideration when the corpus is continuously updated. To address this, RetVDQ separates corpus-level codebook learning from page-level depth allocation. For newly added pages, the existing codebooks can be directly reused, and only synthetic query generation and depth allocation are required. This incremental process takes approximately 13 seconds per page in our setting, with query generation accounting for most of the time. Existing index entries remain unchanged.
Codebook retraining is therefore unnecessary for routine document additions and is expected only when the corpus distribution shifts substantially, for example, when a large number of documents from a substantially different domain or document type are introduced. Thus, the need for retraining depends on the extent of distribution shift rather than the frequency of document additions. This is consistent with other learned quantizers, such as QINCo2 (ICLR 2025) and CrossQ (ICML 2026), whose codebooks are also fitted to the target corpus.
Changes in the paper: We have revised the Limitations paragraph.
Response to Reviewer mU5r (3/3)
Q1: Comparisons with additional VDR compression methods.
Thank you for these references. We have added MarginMerge to our storage–quality comparison on both ColPali and ColQwen2.5 (Figure 3). RetVDQ achieves similar or higher nDCG@5 with much less storage. Selected results are shown in Tables R7 and R8.
As shown in Tables R7 and R8, RetVDQ consistently achieves higher FP retention than MarginMerge across all three compression levels and both backbones. The advantage is especially pronounced under high compression: RetVDQ retains 98.6% and 98.7% of full-precision performance using only 3.6% and 3.3% storage, respectively, whereas MarginMerge requires approximately 10% storage to retain 96.3% and 97.3%. These results demonstrate that RetVDQ provides a more favorable storage–quality trade-off across both backbones.
Table R7. Storage–Accuracy Trade-off on ViDoRe V1 subsets with ColPali Backbone.
| Comparison | Method | Remaining Storage (%) | FP Retention (%) |
|---|---|---|---|
| Low storage | RetVDQ | 2.0 | 94.8 |
| MarginMerge | 2.1 | 86.5 | |
| Medium storage | RetVDQ | 2.8 | 98.0 |
| MarginMerge | 3.0 | 90.3 | |
| High compression | RetVDQ | 3.6 | 98.6 |
| MarginMerge | 10.0 | 96.3 |
Table R8. Storage–Accuracy Trade-off on ViDoRe V1 subsets with ColQwen2.5 Backbone.
| Comparison | Method | Remaining Storage (%) | FP Retention (%) |
|---|---|---|---|
| Low storage | RetVDQ | 1.8 | 95.3 |
| MarginMerge | 2.1 | 86.9 | |
| Medium storage | RetVDQ | 2.5 | 97.5 |
| MarginMerge | 3.1 | 90.5 | |
| High compression | RetVDQ | 3.3 | 98.7 |
| MarginMerge | 10.1 | 97.3 |
HPC-ColPali. Since HPC-ColPali combines two distinct modules, quantization and attention-guided token pruning, a fair comparison should isolate its quantization component. This component assigns a single code to each embedding, essentially corresponding to single-level RQ, which is already included in our baselines. Therefore, its quantization component has already been covered in our comparisons. Its pruning component further reduces the number of embeddings, which is complementary to our adaptive depth allocation. Combining token pruning with adaptive depth allocation is a promising direction for future work, as noted in our Limitations.
Changes in the paper: We have added MarginMerge to Figure 3 and its discussion in Section 4.2, and added both MarginMerge and HPC-ColPali to the related work in Appendix A.2.
We sincerely hope our responses have addressed all your concerns, and we would be grateful if you would consider reflecting this in your final rating. If you have any further questions, please feel free to follow up, and we are glad to address them.
Response to Reviewer mU5r (1/3)
Thank you for the constructive comments. All revisions are highlighted in the manuscript.
We agree that the dependence on synthetic queries needs quantitative analysis. We have added Section 4.5 and Appendix B.3, with experiments on three ViDoRe V1 subsets.
Effect of Pseudo-Query Supervision. To isolate the effect of synthetic-query supervision on depth allocation, we compare Fixed-depth, shuffled, and matched allocation while keeping the same learned codebooks from the first stage. Fixed-depth allocation does not use queries, whereas shuffled allocation randomly assigns each page queries generated for another page. As shown in Table R1 below, our matched allocation outperforms both alternatives across all budgets, showing that depth allocation benefits from synthetic queries and, importantly, their correct correspondence to the target pages. (Table 10 in revised manuscript)
Table R1. Effect of page-query correspondence.
Sensitivity on Query Quantity. We vary the number of page-grounded queries used for depth allocation while keeping the codebooks fixed. As shown in Table R2, even a single page-grounded query provides useful retrieval signals for depth allocation, with further improvements as the number of queries increases to 8. Beyond 8 queries, the performance saturates, which may reflect overlap in the retrieval information provided by additional queries. We thus adopt 8 queries per page as the default setting. (Figure 4 in revised manuscript)
Table R2. Sensitivity to the Number of Queries.
Sensitivity on Query Generator. To assess the impact of the query generator, we replace Qwen3-VL-8B-Instruct with Qwen2.5-VL-7B or InternVL3.5-4B. As shown in the table below, the two alternatives lower macro-averaged nDCG@5 by at most 1.7 and 0.9 points, respectively. Even with the smallest query-generation backbone, InternVL3.5-4B, the gains over the strongest baseline, RQ, are 8.0, 2.6, and 1.4 points at 2, 4, and 6 B/token, respectively. These results demonstrate the robustness of RetVDQ to the choice of query generator. (Table 7 in revised manuscript)
Table R3. Sensitivity to the Query Generator.
Sensitivity on Query Intent Coverage. Our synthetic queries cover four main intents: Extractive (retrieving specific information), Numerical (numerical reasoning), Comparative (comparing items), and Enumerative (listing multiple items), accounting for 63.14%, 10.10%, 18.14%, and 1.31% of queries on average, respectively. To evaluate the effect of incomplete intent coverage, we remove each of the four query intents (Extractive, Numerical, Comparative, and Enumerative) from training while keeping them in the test set. As shown in Table R4, excluding individual intents causes only small performance drops, which become smaller at higher storage budgets. These results suggest that RetVDQ can preserve retrieval-relevant information even when some query intents are absent from the synthetic supervision. (Table 9 in revised manuscript)
Table R4. Sensitivity to Query Intent.
Query Distribution Gap. The official benchmark queries are independently provided, with most derived from existing QA datasets and a small portion generated by LLMs. None of them are used by our query generator or during compression. Thus, all reported retrieval results are evaluated on real queries that are independent of the synthetic supervision. Furthermore, the intent-removal experiments above deliberately introduce distribution shifts in query intents between synthetic training queries and real test queries. The relatively small performance drops suggest that RetVDQ is robust to such shifts in query intent distributions.