acceptodds
Under review as a conference paper at ICLR 2027

When Does Query-Aware Quantization Help? From Redundancy In-Distribution to Gains in Multimodal Retrieval

16 views

Abstract

Embedding-based retrieval at scale depends on vector quantization to hold large databases in memory. A natural idea is to compress vectors so that similarity to queries, rather than the vectors themselves, is preserved. Methods built on this idea, however, often yield only modest gains over strong query-agnostic baselines on standard benchmarks, and the field has lacked a principled account of when query awareness should help. In this paper, we find that on standard benchmarks the query and database distributions are so closely aligned that a rotation optimized on the data alone already captures the query structure, leaving query-aware objectives little to exploit. The redundancy persists even locally, where genuine query structure does exist: we show it adds nothing beyond local data statistics. On multimodal text-to-image retrieval (Yandex Text2Image), where the two distributions differ by construction, the situation reverses: at moderate compression rates, our proposed spectral method delivers 7.1 percentage points more Recall@10 than the standard OPQ+IVFPQ pipeline (a 26% relative improvement) and 1.7 points more than the strongest baseline we test, at equal memory and search cost; every query-aware and locally adaptive variant we test also improves on OPQ+IVFPQ in this regime. We distill both findings into a diagnostic that, from a sample of calibration queries alone, determines in minutes whether a given workload will benefit from query-aware quantization before any method is built.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
RAG, Quantization, Multimodal RAG, Query-Aware Quantization

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_5wdaOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates when query-aware quantization can improve vector retrieval. Firstly, the expected inner product error is expressed as the weighted reconstruction error of the query's second-order moment. Then, through the alignment of the query and database spectra and principal subspaces, random shuffling control, and split-half reliability tests, the potential value of global and local query information is analyzed, and a global method TAQ combining query spectral transformation and bit allocation is proposed, as well as LQ-PQ using local query statistics and contraction estimation. The main contribution lies in attempting to explain the limited benefits of query-aware quantization and distinguishing the benefits brought by local codebook adaptation and query information through controlled comparisons of local query statistics and local data statistics. The study validates on MS-MARCO, SIFT, GloVe, and Yandex Text2Image, and conducts tests by combining ablation experiments, paired bootstrap, and no-routing exhaustive search. The results show that in the tested distribution configurations, query statistics do not provide stable additional benefits; while in the higher bit budget of cross-modal retrieval, TAQ outperforms OPQ + IVFPQ by approximately 7.1 percentage points, and LQ-PQ outperforms local data statistics control by approximately 1.5 percentage points.

Critical strengths
  1. This paper focuses on "When query-aware quantization is worthwhile to use", addressing the method selection issue in actual retrieval systems: whether introducing additional query data and training complexity can yield sufficient benefits.
  2. The authors examine the statistical differences between queries and databases, local structures, and coding budgets, attempting to explain the reasons for the different performance of the methods under different workloads. This analytical perspective helps to understand the applicable conditions of query-aware quantization.
  3. LQ-PQ is implemented by comparing with local data statistics and sharing, mainly changing the source of the weighting matrix. At the same time, a non-weighted local codebook is set up to distinguish the roles of "local adaptation itself" and "query statistics weighting".
Critical weaknesses
  1. There is no main figure, making it difficult to understand the author's intention, and the format is messy.
  2. The results are not sufficient to prove the universal applicability rule.
  3. QUIP (2016) has utilized query non-central second-order moment training for quantization codebooks. Query-aware Quantization (2023) also employs locally grouped shared query matrices. Therefore, expressions such as "precise query-aware target" and "first local query-aware PQ" need to be redefined to highlight the specific differences in routing, contraction estimation, and controlled analysis presented in this paper.
  4. The authors first perform query spectral transformation and then allocate bits according to the original query eigenvalues, acknowledging that the actual effect is approximately equivalent to repeatedly emphasizing the query spectrum. The resulting performance improvement cannot directly serve as evidence for the validity of the original HH weighted target.
  5. This paper lacks sufficient ablation studies and cannot determine whether the benefits come from metric transformation, bit allocation, or a combination of both.
  6. The same number of encoded bits does not imply the same total memory. Local methods require additional storage of multiple sets of codebooks. The same number of candidate lookup tables does not mean the same search time, multiple lookup tables and variable-bit codebooks will increase the cost.
Questions for authors

Please see "Critical Weaknesses".

Reviewer_U1ZZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies when query-aware quantization improves ANN retrieval. The authors propose three methods. TAQ combines query-based transformation and bit allocation, OPQ-Q uses query-based bit allocation with an OPQ rotation, and LQ-PQ learns codebooks from local query statistics. Experiments show little additional benefit from query information on the tested in-distribution datasets, and improved T2I retrieval at 200 bits per vector. The authors also introduce diagnostics to assess when query awareness may help.

Critical strengths
  • The paper clearly presents both the benefits and limitations of query-aware quantization rather than reporting only positive results. Its comparison of in-distribution and out-of-distribution settings offers useful guidance on when query information may help and when its benefits are limited.

  • TAQ achieves the highest R@10 among the compared methods on Yandex T2I at 200 bits per vector, reaching 34.27% compared with 34.05% for LQ-PQ. It outperform the LOPQ-style (32.57%) at this bit budget.

Critical weaknesses
  • My main concern is that the conclusions extend beyond the tested methods. Established alternatives are cited but not evaluated, particularly APQ/AOPQ in OOD-DiskANN (Jaiswal et al., 2023), which also studies text-to-image retrieval. Comparisons under matched compression and search budgets are needed, or the conclusions should be limited to the authors' formulations.

  • The OOD evidence is limited to T2I, where the query and database modalities differ. This does not establish whether the same findings hold under domain shifts within a single modality. Additional same-modality, cross-domain tasks are needed to validate the conclusions and diagnostic.

  • Section 2 (lines 174–176) states that the “two mechanisms compound,” giving weights roughly proportional to the square of lambda. This effectively applies query importance twice. TAQ’s poor in-distribution results could therefore reflect this design rather than query information being unhelpful. Testing each step separately would help distinguish these explanations

  • The comparison would benefit from measurements of actual memory usage and search cost. Matching bits per vector does not necessarily mean equal memory usage or search time in practice.

Questions for authors
  • Could you compare TAQ and LQ-PQ with established query-aware methods, particularly APQ/AOPQ, under matched calibration data and compression budgets?

  • Do the conclusions hold on additional OOD datasets, including domain shifts within the same modality?

  • What happens when TAQ is implemented consistently with the eigenvalue-weighted objective? Please separate the effects of the space transformation and query-based bit allocation.

  • Please report recall–latency curves and total index memory, including regional codebooks, on the same hardware.

Reviewer_SPFnOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper aims to explore the scenarios in which query-aware quantization is beneficial for vector quantization. The authors first formalize task-aware reconstruction loss through the query-covariance weighted reconstruction error. The authors then utilize this query-covariance matrix to derive a PQ-like quantization method, TAQ, that is aware of the prominent query directions. The authors then empirically compare then query and data covariance matrices on MS-MARCO bi-encoder embeddings, SIFT, GloVe and Yandex Text2Image. For the first three datasets the data covariance and query covariance matrices share a similar structure, so in contrast to Text2Image, the authors conclude that TAQ will not lead to substantial performance gains on these datasets. The authors then explore whether utilizing local-query structure (coarse query regions) may be beneficial for the cases in which the covariance matrices share the same structure. The authors then propose another variant of PQ, LQ-PQ. The authors evaluate both methods on MS-MARCO, SIFT, GloVe and Yandex Text2Image and compare them to notable benchmarks.

Critical strengths

The paper asks a useful, under-studied question. Query-aware quantization has produced inconsistent gains, and the paper asks when it should help instead of proposing yet another loss. Framing the answer as a measurable property is an interesting direction

Critical weaknesses
  1. The theory does not support the paper's central claim, and the proposed methods aren't directly derived from it. The main hypothesis is that query-aware quantization helps when the query distribution differs from the data distribution. However, the theory in section 2 talks about the H-weighted objective which depends solely on the query distribution, and not the data distribution.
  2. The papers discusses many concepts (H-weighted objective, H vs Σ\Sigma analysis, a global method TAQ which fails in distribution, a local-structure analysis, another method LQ-PQ). Each concept is studied very narrowly; each one of the methods should be accompanied with more appropriate theory. In addition, more OOD datasets (of different kinds of tasks involving vector search) should be tested to verify that TAQ actually helps for OOD in general, and not just for this specific dataset.
Questions for authors

see weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.