acceptodds
Under review as a conference paper at ICLR 2027

Query-Agnostic Visual KV Compression at Matched Memory Budgets

Abstract

Prefix caching allows vision-language models (VLMs) to reuse the key-value (KV) cache of an image across questions, which requires compression to be completed before the question is known. Most existing visual KV compression methods are query-aware and therefore cannot be applied in this scenario; for the query-agnostic setting, there is a lack of a systematic and fair comparison under the same memory allocation. In this paper, we investigate query-agnostic visual KV compression under matched memory budgets, across two VLM families and four visual question answering benchmarks. On our fixed paired evaluation manifests, retaining deterministically spaced evicted tokens has higher mean scores than merged tokens in all eight model–task comparisons and matches or exceeds tiled moment summaries in all comparisons. Besides, pre-registered falsification tests and an exploratory mixture study do not establish a summary advantage. Meanwhile, we provide an attention-mass bound on the maximum contribution of an eviction summary and analyze three mechanisms that limit summary methods: dilution of individual value outliers, instability of the dispersion correction, and per-block RoPE interactions in merged keys. Finally, quantization behavior depends on the grouping axis and on metadata cost. Together, these controlled comparisons clarify how byte allocation, retained-token fidelity, and task structure affect query-agnostic visual KV compression.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Vision-Language Model, KV cache

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_xEc1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies query-agnostic compression of visual KV prefixes for reuse across questions. It compares quantization, pruning, retention of deterministically spaced evicted tokens, spatially merged tokens, and moment summaries under approximately matched memory budgets. Experiments cover LLaVA-1.5-7B and Qwen2.5-VL-7B on four VQA benchmarks. With bf16 retained cores, sampled tokens have higher mean scores than merged tokens in all eight model–task cells and non-negative margins over moment summaries. The paper also derives an attention-mass bound and discusses outlier dilution, dispersion instability, and conditional RoPE cancellation as possible explanations. Additional experiments show that quantization results depend strongly on grouping and metadata accounting.

Critical strengths
  1. Relevant problem formulation. Compressing an image prefix before future questions are available is a meaningful serving constraint. Separating token selection from the representation of evicted information is useful, and the paper appropriately identifies original-token retention as a baseline that more elaborate representations should justify outperforming.

  2. Useful negative results and disclosures. The study reports failed summary configurations, paired uncertainty estimates, the scope restriction to bf16 retained cores, and the strong effect of quantization grouping. These disclosures provide useful evidence within the tested setting and avoid presenting every experiment as a success.

Critical weaknesses
  1. The central comparison does not yet establish a general disadvantage of merging or summaries. The merged-token baseline directly averages post-RoPE K/V within fixed spatial tiles, without counts or dispersion correction (Section 3). This can change both positional structure and softmax mass. Even for identical keys, replacing (n) entries with one changes their unnormalized mass from (n\exp(x)) to (\exp(x)) unless multiplicity is compensated. The paper discusses RoPE cancellation, but discussion of a failure mechanism is not a demonstration that a position-aware, count-corrected merging baseline still loses to sampling. Moment summaries address some mass effects, but do not resolve the representativeness of the plain merging comparison. The conditional RoPE lemma and cross-model performance differences also do not isolate RoPE as the cause of the observed collapse. Internal inconsistencies further weaken confidence in the comparisons. In Table 21, Qwen–TextVQA is 0.880 for “K row / V groups” and 0.000 for “K/channel / V row,” whereas the text attributes the rescue to K grouping. A labeling error could explain this, but the configuration-to-result mapping needs correction. Likewise, the 6.2% metadata claim for K per-channel / V per-token-row appears inconsistent with Appendix C.1’s 32-token grouping formula: for (n=576) and (d=128), it gives 15.625%. Finally, the 17-token sampling attachment uses approximately 5.5% more bytes than the summary attachment. The whole-cache gap is small, but this favors sampling, contrary to Appendix C.3’s stated bias direction.

  2. The contribution rests on empirical conclusions whose strength and scope remain limited. Only one of eight sampling-versus-moment-summary comparisons has a confidence interval excluding zero; the quantized-core robustness criterion is not met. The study does include text-rich tasks such as TextVQA, ChartQA, and DocVQA, but remains limited to two model families, single-image short-answer VQA, greedy decoding, and fixed source-order subsets of 150 examples, with some comparisons using 48. It does not adequately separate task requirements such as localized retrieval versus distributed aggregation, or test answer quality across multiple real questions sharing an image. The theory does not close this gap. Proposition 1 bounds single-query attention-output differences, not task-level gains. Proposition 2 concerns value-only permutation-invariant moments and does not establish impossibility for joint K–V summaries. The assertion that sampled retention lies outside Equation (2)’s estimator class also needs justification, since its retained numerator and mass can be written in that form. Moreover, the headline 6.5–8.3× latency reduction includes prefix reuse; compression’s incremental speedup over an already cached full prefix is approximately 1.24–1.38×. Compression construction cost and finite-reuse amortization are needed to assess the practical contribution.

Questions for authors
  1. Configuration and accounting clarification. Can you reconcile Tables 20–22 with the claim that K grouping drives the rescue? Please identify the exact grouping of each column and whether the issue is limited to labels or affects the reported contrasts. Can you also reconcile the metadata fractions with Appendix C.1, including the claimed 6.2% overhead for K per-channel / V per-token-row?

  2. Merging controls. Have you evaluated count/logit-mass compensation and a principled position-aware merging strategy under the same effective byte budget? If so, please provide the results. If not, what evidence distinguishes an inherent limitation of merging from limitations of the fixed-tile post-RoPE averaging baseline? Targeted ablations of multiplicity correction and positional handling would materially affect my assessment.

  3. Statistical support and scope. What is the strongest conclusion supported by the paired intervals, particularly given that only one bf16 sampling-versus-moment-summary interval excludes zero and the quantized-core criterion is not met? Do available results show whether the ordering depends on localized detail retrieval versus distributed aggregation, or on answering multiple real questions from one compressed image prefix? I acknowledge the existing OCR/document benchmarks; my concern is generalization across information structure and task requirements.

  4. Theory and deployment interpretation. Why does sampled retention fall outside Equation (2)’s class when its extra tokens supply a numerator and denominator contribution in exactly that form? Which parts of the outlier argument apply to the statistics actually stored by your implementation, including joint K–V information? Please also clarify the one-time compression cost and the number of prefix reuses required for an end-to-end benefit.

If the rebuttal convincingly clarifies the inconsistencies and provides credible evidence supporting the central empirical contribution, I would reconsider my assessment and consider raising my score.

Reviewer_eBfiOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies query-agnostic compression of the visual KV cache in VLMs, motivated by prefix caching: an image's KV cache is prefilled once and reused across many questions, so compression must happen before any question is known. The authors compare five families of training-free methods under a per-sample, byte-matched memory ledger: quantization (INT8/INT4/INT2, rotated, asymmetric and KIVI-style grouping), query-agnostic pruning (grid-uniform and "FastV-blind"), appending evenly spaced evicted tokens ("sampled tokens"), spatially tiled moment summaries with a log-normal readout, and merged tile-mean tokens. Experiments cover LLaVA-1.5-7B and Qwen2.5-VL-7B on TextVQA, ChartQA, DocVQA and VRSBench, using 150 paired examples per cell (48 in some rounds), with decision rules described as pre-registered. The main finding is that, at matched bytes with bf16 retained cores, retaining sampled tokens has higher mean scores than merged tokens in 8/8 cells and matches or exceeds moment summaries in 8/8 cells (only one interval clears zero), while three falsification rounds find no advantage for summaries or mixtures. A theory section bounds the possible benefit of any eviction summary by the attention mass on evicted tokens. It notes that query-agnostic selection leaves about 3.1× more evicted mass than query-aware selection (7.4% vs. 2.3%), and attributes the failure of summaries to value-outlier dilution, divergence of the dispersion correction, and per-block RoPE cancellation in merged keys. The paper also presents a compression-regime map (INT4 at 4×, pruning at ~5–8×, sampled tokens at ≥10×) and appendix studies showing that per-channel grouping of K rescues INT4/INT2 quantization on Qwen. Finally, it reports 6.5–8.3× lower time-to-first-token and 2.75–3.4× lower peak KV memory relative to per-question full prefill.

Critical strengths
  1. A relevant setting with a careful, honest comparison protocol. Prefix caching makes query-agnostic compression a real deployment constraint, and most visual KV compression work ignores it. The per-sample byte ledger (including quantization metadata), paired bootstrap comparisons, explicit decision rules, and transparent reporting of null results and corrections are more rigorous than is typical in this area.

  2. Useful practical findings. Spending evicted bytes on a few raw tokens is at least as good as spending them on moment summaries or merged tokens, and query-agnostic selection leaves substantially more attention mass on evicted tokens than query-aware selection. The 2×2 grouping factorial isolates per-channel grouping of K as what rescues Qwen's KV quantization. These are actionable observations for practitioners.

Critical weaknesses
  1. The comparison omits the most relevant query-agnostic baselines and uses weak versions of some families. Vision-side pruning methods such as VisionZip and DivPrune are already query-agnostic and fit prefix caching naturally, yet they are not evaluated. KVzip, which the paper names as the closest work, is not adapted either. The merged-token baseline averages post-RoPE keys without any position handling, so its collapse to 0.000–0.026 on LLaVA is largely expected, and Lemma 1 essentially explains a design choice rather than a limitation of merging. The pruning selectors (grid-uniform, FastV-blind) are also simple. As a result, the "map" of query-agnostic visual KV compression does not reflect the state of the art.

  2. The quantization results in the main map look unreliable. The main text (Table 1, Figure 4, the 4×/8× regimes) uses per-row symmetric quantization. Appendix F shows that per-channel K grouping recovers full-cache quality in 8/8 cells, but the main figure is explicitly not redrawn. Several results suggest implementation problems that need a sanity check:

    • On Qwen, per-row asymmetric INT4 collapses to 0.000 while symmetric INT4 with the same groups reaches 0.536, even though the asymmetric variant is strictly more expressive and has lower measured error.
    • Hadamard-rotated INT4, described as faithfully implementing QuaRot, also collapses to near 0 on Qwen despite a lower error norm.

    Attributing these to "error direction" without checking against a reference implementation is not convincing, and the main-text regime map rests partly on them.

  3. The central verdict is essentially a null result with limited power. The claim that sampled tokens "match or exceed" moment summaries rests on 7/8 intervals that straddle zero, at nn=150 per cell (and nn=48 in the deep-regime and question-conditioned runs, where the pre-registered robustness check comes out "VERDICT CHALLENGED"). The byte budget also slightly favors the sampled method (8,704 B vs. 8,196–8,256 B on the evicted slot, about 5.5%). Two cells have very low ceilings (LLaVA ChartQA 0.200, DocVQA 0.172) and little discriminating power. "Matches" would need an equivalence test, and larger or full evaluation splits would make the conclusion more informative.

  4. The theory is elementary, and the mechanism evidence is anecdotal. Proposition 1 follows from the triangle inequality, Proposition 2 from permutation invariance, Proposition 3 from the well-known blow-up of log-normal moment matching, and Lemma 1 is an informal geometric-series argument. The empirical support for the mechanisms comes largely from single examples: Figure 3a uses one sample, and the dispersion probe uses one sample (id 1196) and one KV head. The claim that "one mechanism, rather than a difference in implementation quality" explains the cross-family merging gap is therefore overstated.

  5. The practical benefit is mostly attributable to prefix caching, not to compression. The 6.5–8.3× TTFT reduction is relative to per-question full prefill. Compared with an uncompressed reused cache, compression adds only 1.4× (LLaVA) and 1.2× (Qwen), and the absolute peak KV memory on Qwen is only 20.9 MiB. The latency benchmark uses 48 queries over 6 images. Because the setting is limited to single, moderate-resolution images with short answers, the regimes where visual KV compression matters most (high-resolution documents, multi-image, video, many concurrent requests) are not tested.

  6. The presentation makes the paper hard to read and verify. The appendix reads like an experiment log:

    • Repeated errata, "frozen wording," checker fixes and amendments.
    • A leftover template placeholder ("Frozen wording ... {k} = 4").
    • References to nonexistent sections (L.3–L.5, J.5, J.8.2), and a missing row 10 in Table 6.
    • Tables where different configurations share the same label ("per-channel quantization" in Tables 19, 23 and 25, "retained-token method" for several conditions).

    Key definitions needed to read the main results (e.g., the exact moment-summary readout, the FastV-blind selector) are scattered across the appendix.

Questions for authors
  1. Missing baselines. How do query-agnostic vision-side pruning methods (e.g., VisionZip, DivPrune), a KVzip-style query-agnostic scorer, and a merging baseline that merges pre-RoPE keys (or re-applies positions after merging) compare under your byte ledger? These comparisons would most directly affect my assessment of whether the "retain tokens rather than summarize" conclusion holds against strong alternatives.

  2. Quantization sanity checks. Why does per-row asymmetric INT4 collapse to 0.000 on Qwen when symmetric INT4 with the same groups reaches 0.536? Why does Hadamard-rotated INT4 collapse despite lower error? Have these been checked against the official KIVI/QuaRot implementations? Will the main-text map and Figure 4 be updated to use per-channel K grouping?

  3. Statistical support for "matches." Can you report equivalence tests (e.g., TOST with a stated margin) for sampled tokens vs. moment summaries, results on full evaluation splits, and a comparison in which the sampled method is given no more bytes than the summary? Is the pre-registration available in an anonymized, timestamped form?

  4. Population-level mechanism evidence. Can you quantify outlier dilution, dispersion-driven tile flips and merged-key norm attenuation across all samples, layers and heads, rather than on single examples, and relate these statistics to per-sample accuracy differences?

  5. Settings where compression matters. How do the conclusions and the TTFT/memory benefits change for high-resolution documents, multi-image or video prefixes, and serving under many concurrent requests, where KV memory is the actual bottleneck?

Reviewer_8ph5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(−1)
Read review
Paper summary

This paper studies query-agnostic visual KV compression for VLMs. It compares quantization, pruning, retained tokens, moment summaries, and merged tokens under matched memory budgets across two model families and four VQA benchmarks. The main finding of this work is that, with bf16 retained cores, sampled retained tokens are a more reliable usage of evicted-cache bytes than summaries. The paper also provides an attention-mass bound and mechanism analyses, and sketches a compression-regime map.

Critical strengths

The research problem is relevant, and the byte-matched setup is also reasonable. The analysis of negative result on moment summaries is useful and insightful. The two-model, four-benchmark matrix and ledger accounting are also solid.

Critical weaknesses

(1) The key claim of this work is weak and should be supported by more sufficient experimental result and analysis.

(2) Conclusions are tied to bf16 retained cores and per-row symmetric quantization; the appendices show that changing grouping can reshape the map, which weakens the generality of this approach.

(3) The summary baselines may also be too weak to support a broad “tokens beat summaries” conclusion.

Questions for authors

Please include more experiments by increasing sample size and foreground confidence intervals/effect sizes and adding another model family and multi-image or video prefixes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.