acceptodds
Under review as a conference paper at ICLR 2027

HeadGuard: Selective Head Protection for Low-Bit VLM KV-Cache Quantization

Abstract

Low-bit key–value (KV) cache quantization saves storage but can sharply degrade vision–language model (VLM) accuracy. We introduce HeadGuard, a composable head-protection method that augments a base KV-cache quantizer with a fixed high-precision mask. Image-sensitivity and output-sensitivity scores select physical KV heads offline, with approximately 1/8 protected in the main experiments; their image keys and optionally values remain in bfloat16 (BF16), while the base quantizes unprotected image entries. Across eight VLMs, three base quantizers, and eight benchmarks (six discriminative and two generative), HeadGuard recovers a substantial fraction of lost accuracy on weaker quantizers, with the strongest gains for Qwen and InternVL. At 2 bits, the six-task discriminative mean over eight models rises from 0.436 to 0.580 on the weakest base; protection can also improve generated answers and caption fidelity to BF16 outputs. Mean accuracy gains persist across all three quantizers with both tested calibration datasets. Keys-only protection retains substantial recovery at lower modeled storage cost. Evaluated through simulated quantization, HeadGuard offers a composable way to improve low-bit VLM accuracy without replacing the underlying quantizer.

Primary area:
foundation or frontier models, including LLMs
Keywords:
vision-language models, KV-cache quantization, mixed-precision quantization, attention-head selection, sensitivity analysis, multimodal inference

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_6aZ7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

HeadGuard is a mixed-precision KV-cache quantization method for VLMs, where image token keys (and optionally values) are kept in BF16 for selected attention heads. The heads are ranked by combining image- and output-sensitivity scores computed offline from calibration samples. This produces a static protection mask that does not require gradient computation at test-time. Eight VLMs, three base quantizers, and eight benchmarks are broadly covered in experiments. Notably, substantial accuracy recovery over uniform per-token minmax quantization is achieved, and keys-only protection can perform similar to protecting both keys and values at lower modeled storage cost.

Critical strengths
  1. Evaluation broadly covers 8 VLMs, 3 base quantizers, and 8 benchmarks. Held-out calibration splits are also clearly mentioned.
  2. The protection mask is simple, static, and computed offline from only 100 calibration samples, which can work with any base quantizer that allows per-head bypass. It requires no gradient computations at inference-time.
  3. K-only protection (HG-K) achieves accuracy similar to K+V protection at about 22% lower modeled image-cache storage at 2-bit. This also supports the useful observation that image keys (not values) are the main quantization bottleneck.
  4. HeadGuard can recover a lot where uniform min-max quantization fails. Specifically in Table 1, Qwen2.5-VL goes from 0.360 to 0.747 and InternVL3 from 0.362 to 0.688 at 2-bit.
  5. Comparison study on protected head selectors covers six reasonable alternatives. The authors also openly report where the method falls short.
Critical weaknesses

My main concern is that the manuscript lacks experimental evidence crucial for supporting the paper's key motivation and claims.

Roughly in priority order:

  1. The evaluation is limited to benchmarks that do not require long answers, reasoning, or multi-turn interaction. In such tasks, the assumption that image tokens dominate a VLM's KV cache might not hold.
  2. Efficiency results are severely limited; only a modeled bit-width of the image KV cache is reported, while text KV remains in BF16. The authors should report the end-to-end latency with and without the method, along with the peak VRAM consumption.
  3. In Table 2, the proposed image-sensitivity score adds little gain beyond the existing output-sensitivity score adapted from Michel et al., (2019). Furthermore, the gain is particularly marginal on the KIVI-style partial reimplementation.
  4. The fraction of protected heads is chosen empirically, and the budget sweep (Fig. 1) shows it does not transfer across different models. This requires users to run a per-model sweep, degrading the deployability of the proposed method.
  5. Statistical support is provided only on POPE (Tables 7-9). On 2-bit, the HeadGuard gain over min-max uniform is not reliable on Gemma models. On 3-bit, the HeadGuard gain over output-only is not significantly positive for any model.
  6. Many implementation details (e.g., what KIVI- and AKVQ-style reimplementations include, whether quantization is applied before or after RoPE, whether image KV cache is quantized during prefill, and exact quantizer settings) are missing.
  7. No analysis of which heads are selected and protected is reported. Such analysis might help explain why the Gemma models do not recover at 2-bit even with 1/4 of the heads protected (Fig. 1).
Questions for authors

I have a few questions for the authors:

  1. Can the authors provide end-to-end measured peak VRAM and latency, with and without KV cache compression?
  2. Can the authors analyze the fraction of image tokens in KV cache after the full text generation, and the resulting whole-KV cache bit-width?
  3. At 2-bit min–max, K+V protection raises the modeled image KV cache cost from 2.25 to 3.97 bits per element (Sec. 5.4) yet reaches only 0.580 six-task accuracy (Sec. 5.1), while uniform 3-bit KIVI-style reaches 0.741 (Table 2) at a likely lower bit-width. Can the authors report the measured KV cache storage for the main experiments (each base, bit-width, and K+V/K-only setting), so that accuracy can be compared at matched storage?
  4. How much do the top-K heads chosen by HeadGuard and the output-sensitivity alone overlap? If they differ, what is the qualitative/quantitative impact on the VLM's behavior?
  5. For the AKVQ-style, how are the 2% per-head BF16 tokens treated inside protected heads selected by HeadGuard, and are they included in the modeled storage?
  6. For the AKVQ-style, are key norms measured before or after rotation?
2 comments

Thank you for the effort to address my concerns. Clarifications on implementation details and additional head-selection analysis are informative and helpful. However, I think the following are still open and critical.

  1. W3: The image-sensitivity score does not reliably improve upon the output-only selection, which weakens one of the paper's core contributions.
  2. Q1 & Q3: Measured end-to-end peak VRAM, latency, and actual KV-cache storage are not provided. This leaves HeadGuard’s efficiency advantage unverified.

I maintain my score.

Thank you for the follow-up. We are glad the implementation details and head analysis helped.

Image term (W3). We agree, and the revision says so: the output score is the recommended default, and the image term is optional. The paper's gains do not depend on it: output-only selection is within about one point of HeadGuard's mean gain on every base and bit width (Table 2). What remains is the main finding: in the Qwen and InternVL models, protecting a few KV heads recovers most of the low-bit loss, and the label-free output score finds them (on Qwen2-VL-2B, recovery hinges on three heads it ranks first to third with POPE calibration; Appendix J.2).

Efficiency (Q1, Q3). HeadGuard claims no efficiency advantage over its base: it spends extra storage on a few heads to recover accuracy. That cost follows from the cache format; Appendix H gives the image-cache bits for every base and bit width and the whole-cache bits per model. Peak memory and latency depend on the base's packed kernel, which only needs to skip quantization for the selected heads. Section 6 lists end-to-end measurement as future work.

HeadGuard's case is therefore repair, not efficiency: when the base is fixed, for example by the kernels a deployment has, and it fails on a model, protecting 1/8 of the heads recovers much of the loss (Qwen2.5-VL 0.360 → 0.747 at 2 bits, Table 1).

Reviewer_fHdeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

HeadGuard addresses accuracy degradation from low-bit KV-cache quantization in vision-language models. It combines image and output sensitivity scores computed during offline calibration to select KV heads for BF16 protection, while quantizing the remaining image KV entries. Text KV and model weights remain in BF16. The paper also studies keys-only protection. Experiments across eight models, three base quantizers, and eight benchmarks show substantial recovery in some settings, depending on the model, quantizer, and calibration data.

Critical strengths

S1: A simple and useful protection strategy.

The method uses a fixed mask selected offline, requiring neither weight updates nor inference-time gradients. Selecting physical KV heads also makes the protection unit consistent with the cache structure under grouped-query attention.ention.

Critical weaknesses

W1: The reasons for simplifying the base quantizers and the effects of these changes are insufficiently examined.

Why does the KIVI-style implementation omit the original grouping and high-precision residual-cache policy [1], and why does the AKVQ-style implementation replace attention-based token saliency with key norms [2]? These changes may affect both baseline accuracy and the additional gains from HeadGuard. Although the paper discloses the modifications, it does not adequately explain these choices or isolate their effects through controlled comparisons.

W2: Simulated quantization leaves practical memory and inference benefits unverified.

Low-bit entries are quantized and reconstructed as floating-point tensors rather than stored in a packed low-bit cache. The experiments therefore assess accuracy under simulated quantization, while storage savings are estimated analytically. Actual GPU memory savings, decoding latency, and throughput remain unverified.

References

[1] Liu et al. KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache, ICML 2024.

[2] Su et al. AKVQ-VL: Attention-Aware KV Cache Adaptive 2-Bit Quantization for Vision-Language Models, 2025.

Questions for authors

Q1: Why does image sensitivity help in some settings but cause large regressions in others, and when should it be included?

The paper acknowledges that its incremental benefit is conditional. However, with MMBench calibration, Qwen2.5-VL, and 2-bit AKVQ-style quantization, six-task accuracy falls from 77.4% with output-only selection to 62.7% with HeadGuard (Table 17), and the TextVQA score falls from 0.748 to 0.380 (Table 20). What determines whether image sensitivity helps or hurts? Can independent calibration or validation data guide the choice between hybrid and output-only selection?

Q2: Do the conclusions hold with complete task inputs and official scoring?

MMMU currently receives only the first image, while TextVQA uses a custom scorer with a token-containment fallback. Could the main results be checked using all required images for multi-image questions and the official TextVQA scorer, to establish whether the gains and rankings depend on these evaluation choices?

Q3: What determines the large differences in protection effectiveness across models?

Under 2-bit min-max quantization, protection changes Gemma-3 accuracy from 37.18% to 36.70% and Gemma-4 from 36.15% to 36.33%, both far below BF16, whereas Qwen and InternVL recover substantially. At 3 bits, LLaVA-NeXT changes slightly from 66.72% to 66.60%.

Q4: How do image preprocessing choices affect quantization damage and protection benefits?

The evaluation caps image resolution, disables Gemma's additional cropping mode, and restricts InternVL to one image tile. These settings change the visual information and token composition. Do the results remain consistent under standard model preprocessing and different resolutions, cropping, or tiling settings? Would the existing protection mask transfer, or would recalibration be required?

Q5: How sensitive are the results to the particular calibration examples selected?

Tables 16 and 17 compare 100 POPE examples with 100 MMBench examples, but do not establish stability across repeated draws from the same dataset. With an independent evaluation set held fixed, how much do the selected heads, final accuracy, and hybrid-minus-output-only gains vary across calibration subsets and random seeds?

Reviewer_C7wYOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes HeadGuard, a method to reduce accuracy loss from KV-cache quantization in vision–language models. It uses gradients on a small calibration set to rank attention heads by their importance. About one eighth of the most 'important' KV heads keep their image keys, and optionally values, in high precision BF16. The remaining image KV entries use low-bit quantization using off the shelf methods, while text KV stays in BF16. Across eight models and eight benchmarks, the method recovers substantial accuracy with weaker quantizers, especially on Qwen and InternVL. Output sensitivity alone accounts for most of the gains. Protecting only keys gives similar accuracy at a lower estimated storage cost. The experiments simulate quantization and do not measure actual memory savings or inference speed.

Critical strengths

S1 Broad range of MLLMs evaluated in simple setting: The paper reports results over a wide class of MLLMs, including failures such as both Gemma models (Fig. 1).

Critical weaknesses

W1 Recent baselines missing: HeadGuard is compared only against its own, realtively older base quantizers (RTN, simplified KIVI, simplified AKVQ-VL). Recent strong mixed-precision KV allocators, including ones that allocate different precision among different heads and channels as is done by the paper are not compared against , e.g. (i) head level TurboAttention [A] and channel level MixKVQ [B] and (ii) layer-level KVTuner [C] and KVmix [D]

Without these, HeadGaurd's method: that head-level protection, and keeping those heads in full BF16 rather than just higher bits, is the right granularity is untested, and it may well not hold once the base allocates precision by importance.

The paper needs to :

(a)compare against these baselines and show a better memory-accuracy tradeoff at the same Average bit per token budget.

(b) It also needs to show that keeping the protected heads in full BF16 makes sense over just giving them more bits.

(c) Compare its The gradient-based head scores to cheaper head/channel ranking statistics, such as those proposed by [A] and [B] under the same 1/8 budget.

W2 Limited novelty, the image component does not help: The output-sensitivity score is Michel et al.'s (L98-99) head importance . The only VLM-specific addition is the image-sensitivity score, and the paper's own ablation (Table 2) shows combining it with output sensitivity beats output-only by almost nothing in most settings using non RTN quantizers.

W3 Weak to no gains on decent base quantizer: The main gains are on per-token min-max (RTN), which is extremely weak in terms of absolute performance at 2 bits. On KIVI-style the gain is tiny at 2 bits and ~0 at 3 bits, possibly within variance (W4). No actual recent strong base like KVQuant [E], RotateKV [F] or TurboQuant [G] is run, where the gain might well be 0.

W4 No std deviations and small gains: The main tables have no variance, so it is hard to judge the significance of the improvements especially on several critical resutls. e.g. On the KIVI-style base the gain is under one point at 2 bits and near zero at 3 bits. These small gains might be within the variance on these benchmarks.

  • [A] TurboAttention: Efficient Attention Approximation for High Throughputs LLMs. Kang et al., MLSys 2025.
  • [B] MixKVQ: Query-Aware Mixed-Precision KV Cache Quantization for Long-Context Reasoning. Zhang et al., ACL 2026.
  • [C] KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inference. Li et al., ICML 2025.
  • [D] KVmix: Gradient-Based Layer Importance-Aware Mixed-Precision Quantization for KV Cache. Li et al., AAAI 2026.
  • [E] KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization. Hooper et al., NeurIPS 2024.
  • [F] RotateKV: Accurate and Robust 2-Bit KV Cache Quantization for LLMs via Outlier-Aware Adaptive Rotations. Su et al., IJCAI 2025.
  • [G] TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate. Zandieh et al., ICLR 2026.
Questions for authors

See W1

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.