acceptodds
Under review as a conference paper at ICLR 2027

Budget-Aware Multi-Vector Index Compression for Visual Document Retrieval

Abstract

Visual Document Retrieval (VDR) has become an essential capability for document intelligence systems, yet the growing storage cost of multi-vector indexes has become a critical barrier to large-scale deployment. Compressing VDR indexes introduces two fundamental challenges: 1) how to allocate a limited storage budget between preserving document-side matching opportunities and reducing representation cost; 2) how to maintain reliable late-interaction scoring when aggressive compression distorts the geometry of compressed representations. In this work, we propose BaMIC (Budget-aware Multi-Vector Index Compression), a coverage-first compression method. We identify a structural asymmetry between compression dimensions: reducing vector cardinality directly removes potential matching opportunities, whereas numerical compression preserves matching coverage but may introduce geometric distortions in the extreme compression regime. Based on this observation, BaMIC prioritizes document-side vector coverage over per-vector precision and adapts representation rates to the available storage budget, and applies geometry-calibrated scoring, which requires no additional training or storage, to mitigate compression-induced radial bias. Extensive experiments across multiple VDR benchmarks, vision-language backbones, and compression budgets demonstrate that BaMIC achieves favorable storage-quality trade-offs, retaining 94.1% of full-precision Hit@1 on average with ColQwen3.5 at 64 compression and improving Hit@1 by up to 34.5 points over the strongest vector-count baseline under equal storage budgets.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Visual Document Retrieval, Multi-Vector Index Compression, storage efficiency

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_oGeJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes BaMIC, a budget-aware compression method for multi-vector visual document retrieval. The key insight is that preserving document-side vector coverage is more important than maintaining high numerical precision under limited storage budgets. Accordingly, BaMIC retains all patch embeddings, applies rate-adaptive quantization, and introduces geometry-calibrated scoring (nADC) to mitigate norm distortion under aggressive compression. Experiments across multiple benchmarks and vision-language backbones demonstrate improved storage–accuracy trade-offs over pruning, merging, and conventional quantization methods, retaining 94.1% of full-precision Hit@1 with ColQwen3.5 at 64× compression.

Critical strengths
  1. The paper provides a useful perspective on multi-vector index compression by highlighting the importance of preserving matching opportunities over numerical precision. The decomposition of compression errors into support, directional, and radial distortion offers a clear explanation of why different compression strategies affect retrieval performance differently.
  2. The paper evaluates BaMIC across multiple benchmarks, vision-language backbones, and compression ratios, with comparisons against pruning, merging, and quantization methods. The results demonstrate strong storage–accuracy trade-offs, particularly under aggressive compression. The ablation studies further support the effectiveness of geometry-calibrated scoring.
Critical weaknesses
  1. Limited methodological novelty. BaMIC primarily combines existing quantization techniques (SQ, PQ, and OPQ) with vector normalization. Although the empirical observations are interesting, the proposed nADC is relatively straightforward, and its distinction from existing normalized similarity computation methods is not sufficiently established. This raises concerns about the novelty of the technical contribution.

  2. Limited generality and practical efficiency. The coverage-first strategy is not consistently optimal, as the joint allocation experiments in Appendix C show that retaining fewer vectors with higher precision can outperform full-support compression under extreme storage constraints. Moreover, BaMIC preserves all document vectors and provides little improvement in query latency, limiting its practical benefits beyond storage reduction.

Questions for authors

See Critical Weaknesses.

Reviewer_QFhoOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

BaMIC is motivated by the question of how to use a limited storage budget effectively when compressing multi-vector indexes for visual document retrieval. The authors divide index compression into two directions. The first is cardinality reduction, which reduces the number of document vectors through pruning or merging, and the second is numerical compression, which reduces the precision of each vector. The main hypothesis of the paper is that cardinality reduction can remove the matching opportunities that MaxSim needs, whereas quantization retains all matching candidates. Based on this, BaMIC proposes a coverage-first compression policy. It keeps every document vector by default and applies scalar quantization or PQ/OPQ depending on the compression rate. In addition, noting that the norm variation of decoded vectors in sub-bit compression biases MaxSim scoring, BaMIC corrects it with normalized asymmetric distance computation (nADC). With ColQwen3.5 at 64× compression, BaMIC retains 94.1% of full-precision Hit@1 on average.

Critical strengths
  1. BaMIC analyzes the difference between two compression dimensions, cardinality and precision. Approaching the problem from the importance of preserving matching opportunities in MaxSim is an intuitive yet practical insight.

  2. The paper observes that norm variation of decoded vectors in sub-bit quantization affects MaxSim winner selection, and verifies this through controlled interventions.

  3. Unlike prior work that evaluates compression quality mainly through reconstruction error or retrieval performance, this paper separates and analyzes the mechanisms that cause performance degradation.

  4. BaMIC uses scalar quantization above 1 bit/dim and PQ/OPQ below it, and applies geometry calibration to the decoded vectors. This design reflects that the dominant error source and the appropriate compression mechanism change with the compression budget. In other words, it proposes a way to select and combine existing compression techniques according to the properties of MaxSim.

Critical weaknesses
  1. The novelty of the proposed method seems limited. The components of BaMIC are well-known scalar quantization, PQ/OPQ, and vector normalization. Rather than a new compression approach, the work is closer to combining an analysis of compression mechanisms with an appropriate quantization policy. It would therefore be better to emphasize the analysis and design principles, rather than a new approach, as the contribution.

  2. Does the coverage-first policy generalize? The results are generally solid, but it is unclear whether the policy applies to all multi-vector retrieval settings. Proposing such an experimental design with an analytical approach would make the contribution clearer.

  3. Trade-off between storage efficiency and retrieval latency. BaMIC greatly reduces index storage, but since it keeps every document vector, the MaxSim computation itself is not reduced, as Table 3 also shows. The work would have been more complete if latency reduction had also been part of the contribution.

Questions for authors
  • What document or query characteristics make the coverage-first policy especially effective? An analysis by embedding redundancy or query complexity would be interesting.

  • The paper explains well why nADC is effective in sub-bit compression. How effective is the calibration at higher precision?

1 comment

Thank you for the thorough rebuttal and the additional experiments. I will keep my original rating.

Reviewer_VnGKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

BaMIC keeps document vectors while reducing their precision, then normalizes decoded vectors to correct MaxSim bias. Experiments across three backbones show substantial storage savings. The empirical findings are useful, although the blanket preference for keeping every vector needs narrowing.

Critical strengths
  1. The comparisons at matched storage budgets make a convincing case that aggressive pruning can discard useful matches.
  2. The controlled interventions isolate radial bias, and the correction needs no additional per-vector metadata.
  3. Several backbones, paired significance tests, and a detailed failure analysis give the empirical findings weight.
Critical weaknesses
  1. Table 5 weakens the central allocation claim. At roughly 64 times compression, keeping half the vectors and increasing precision beats full-support BaMIC-PQ by 6.5 Hit@1 points on MMLong-34K. This deserves a proper hybrid frontier, including OPQ and informed pruning, rather than being treated as a future extension.
  2. The calibration is exactly cosine scoring of decoded PQ or OPQ vectors. Its practical value is clear, but the algorithmic contribution beyond standard quantization plus normalization is small. The presentation currently gives this correction more conceptual weight than the method supports.
  3. Storage savings do not resolve the search bottleneck. Table 3 still takes about 39 ms to score 34K pages, versus about 0.5 ms after vector reduction. Without an approximate search experiment, the coverage-first recommendation is hard to assess when both memory and response time matter.
Questions for authors

How does the best calibrated hybrid compare with full support at the same bytes and scoring latency? Which allocation claims still hold after that comparison?

Reviewer_XyBbOpenReview ↗
Rating
2
Confidence
1
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes BaMIC, a budget-aware multi-vector index compression method for Visual Document Retrieval (VDR). The authors show that reducing vector precision generally preserves retrieval performance better than reducing vector count under the same storage budget. Based on this observation, BaMIC preserves all document vectors and applies rate-adaptive quantization with geometry-calibrated scoring to mitigate compression-induced distortions. Experiments across multiple benchmarks and vision-language backbones show that BaMIC achieves favorable storage–retrieval trade-offs, retaining 94.1% of full-precision Hit@1 at 64× compression and outperforming vector-count reduction baselines by up to 34.5 percentage points.

Critical strengths
  1. Insightful analysis of compression strategies. The paper provides a useful theoretical and empirical analysis of the trade-off between vector cardinality and numerical precision under limited storage budgets. In particular, the observation that preserving matching opportunities is generally more important than maintaining high vector precision offers valuable insights for designing storage-efficient VDR systems.

  2. Simple yet effective geometry calibration. The proposed geometry-calibrated scoring method addresses compression-induced radial bias without requiring additional training or per-vector storage. Its effectiveness is supported by controlled experiments that isolate the impact of radial distortion, demonstrating consistent improvements under aggressive compression.

  3. Comprehensive experimental evaluation. The paper evaluates the proposed method across multiple datasets, vision-language backbones, and compression budgets. The experiments include comparisons under equal storage budgets, ablation studies, statistical significance tests, and efficiency analyses. These evaluations provide convincing empirical evidence for the effectiveness of the proposed approach.

Critical weaknesses
  1. Limited methodological novelty. BaMIC primarily combines existing quantization techniques (scalar quantization, PQ, and OPQ) with vector normalization. In particular, the proposed geometry calibration essentially normalizes reconstructed vectors before computing similarity scores, which is a relatively straightforward operation. Moreover, the budget-aware strategy follows a fixed rule rather than explicitly optimizing the allocation of storage between vector cardinality and precision. Therefore, the technical novelty of the proposed framework appears somewhat limited.

  2. The coverage-first strategy is not always optimal. The central argument of the paper is that preserving all document vectors is preferable to reducing vector cardinality. However, Table 5 in Appendix C shows that combining pruning with quantization can outperform the full-support BaMIC approach under extreme compression budgets. For example, pruning 50% of the vectors and applying BaMIC-PQ improves Hit@1 by 6.5 percentage points on MMLong-34K at approximately 64× compression. This raises questions about the generality of the proposed coverage-first strategy and suggests that jointly optimizing vector cardinality and precision could be more effective.

  3. Limited improvement in retrieval efficiency. Although BaMIC substantially reduces index storage, it preserves all document vectors and therefore does not significantly reduce query-time computation. As shown in Table 3, its query latency remains close to that of the full-precision approach (approximately 39 ms), whereas vector-count reduction methods achieve approximately 0.5 ms. Furthermore, the experiments rely on exhaustive MaxSim scoring without approximate nearest-neighbor indexing. The practical scalability of BaMIC for large-scale retrieval systems therefore remains insufficiently demonstrated.

Questions for authors
  1. Regarding the coverage-first strategy (W2): Table 5 shows that combining pruning with quantization can outperform the full-support BaMIC approach, particularly under extreme compression budgets. For example, pruning 50% of the vectors improves Hit@1 by 6.5 percentage points on MMLong-34K at approximately 64× compression. Could the authors further explain why preserving all vectors is preferred over jointly optimizing vector cardinality and precision? Would an adaptive allocation strategy provide better performance across different compression budgets?

  2. Regarding retrieval efficiency (W3): Although BaMIC significantly reduces index storage, Table 3 shows that its query latency remains comparable to full-precision retrieval, while vector-count reduction methods achieve substantially lower latency. Given that practical deployment is a key motivation, could the authors discuss how BaMIC could be integrated with efficient retrieval mechanisms, such as approximate nearest-neighbor search? It would also be helpful to evaluate whether the storage savings translate into practical retrieval efficiency improvements at larger scales.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.