- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
HeadGuard is a mixed-precision KV-cache quantization method for VLMs, where image token keys (and optionally values) are kept in BF16 for selected attention heads. The heads are ranked by combining image- and output-sensitivity scores computed offline from calibration samples. This produces a static protection mask that does not require gradient computation at test-time. Eight VLMs, three base quantizers, and eight benchmarks are broadly covered in experiments. Notably, substantial accuracy recovery over uniform per-token minmax quantization is achieved, and keys-only protection can perform similar to protecting both keys and values at lower modeled storage cost.
Critical strengths
- Evaluation broadly covers 8 VLMs, 3 base quantizers, and 8 benchmarks. Held-out calibration splits are also clearly mentioned.
- The protection mask is simple, static, and computed offline from only 100 calibration samples, which can work with any base quantizer that allows per-head bypass. It requires no gradient computations at inference-time.
- K-only protection (HG-K) achieves accuracy similar to K+V protection at about 22% lower modeled image-cache storage at 2-bit. This also supports the useful observation that image keys (not values) are the main quantization bottleneck.
- HeadGuard can recover a lot where uniform min-max quantization fails. Specifically in Table 1, Qwen2.5-VL goes from 0.360 to 0.747 and InternVL3 from 0.362 to 0.688 at 2-bit.
- Comparison study on protected head selectors covers six reasonable alternatives. The authors also openly report where the method falls short.
Critical weaknesses
My main concern is that the manuscript lacks experimental evidence crucial for supporting the paper's key motivation and claims.
Roughly in priority order:
- The evaluation is limited to benchmarks that do not require long answers, reasoning, or multi-turn interaction. In such tasks, the assumption that image tokens dominate a VLM's KV cache might not hold.
- Efficiency results are severely limited; only a modeled bit-width of the image KV cache is reported, while text KV remains in BF16. The authors should report the end-to-end latency with and without the method, along with the peak VRAM consumption.
- In Table 2, the proposed image-sensitivity score adds little gain beyond the existing output-sensitivity score adapted from Michel et al., (2019). Furthermore, the gain is particularly marginal on the KIVI-style partial reimplementation.
- The fraction of protected heads is chosen empirically, and the budget sweep (Fig. 1) shows it does not transfer across different models. This requires users to run a per-model sweep, degrading the deployability of the proposed method.
- Statistical support is provided only on POPE (Tables 7-9). On 2-bit, the HeadGuard gain over min-max uniform is not reliable on Gemma models. On 3-bit, the HeadGuard gain over output-only is not significantly positive for any model.
- Many implementation details (e.g., what KIVI- and AKVQ-style reimplementations include, whether quantization is applied before or after RoPE, whether image KV cache is quantized during prefill, and exact quantizer settings) are missing.
- No analysis of which heads are selected and protected is reported. Such analysis might help explain why the Gemma models do not recover at 2-bit even with 1/4 of the heads protected (Fig. 1).
Questions for authors
I have a few questions for the authors:
- Can the authors provide end-to-end measured peak VRAM and latency, with and without KV cache compression?
- Can the authors analyze the fraction of image tokens in KV cache after the full text generation, and the resulting whole-KV cache bit-width?
- At 2-bit min–max, K+V protection raises the modeled image KV cache cost from 2.25 to 3.97 bits per element (Sec. 5.4) yet reaches only 0.580 six-task accuracy (Sec. 5.1), while uniform 3-bit KIVI-style reaches 0.741 (Table 2) at a likely lower bit-width. Can the authors report the measured KV cache storage for the main experiments (each base, bit-width, and K+V/K-only setting), so that accuracy can be compared at matched storage?
- How much do the top-K heads chosen by HeadGuard and the output-sensitivity alone overlap? If they differ, what is the qualitative/quantitative impact on the VLM's behavior?
- For the AKVQ-style, how are the 2% per-head BF16 tokens treated inside protected heads selected by HeadGuard, and are they included in the modeled storage?
- For the AKVQ-style, are key norms measured before or after rotation?
2 comments
Thank you for the follow-up. We are glad the implementation details and head analysis helped.
Image term (W3). We agree, and the revision says so: the output score is the recommended default, and the image term is optional. The paper's gains do not depend on it: output-only selection is within about one point of HeadGuard's mean gain on every base and bit width (Table 2). What remains is the main finding: in the Qwen and InternVL models, protecting a few KV heads recovers most of the low-bit loss, and the label-free output score finds them (on Qwen2-VL-2B, recovery hinges on three heads it ranks first to third with POPE calibration; Appendix J.2).
Efficiency (Q1, Q3). HeadGuard claims no efficiency advantage over its base: it spends extra storage on a few heads to recover accuracy. That cost follows from the cache format; Appendix H gives the image-cache bits for every base and bit width and the whole-cache bits per model. Peak memory and latency depend on the base's packed kernel, which only needs to skip quantization for the selected heads. Section 6 lists end-to-end measurement as future work.
HeadGuard's case is therefore repair, not efficiency: when the base is fixed, for example by the kernels a deployment has, and it fails on a model, protecting 1/8 of the heads recovers much of the loss (Qwen2.5-VL 0.360 → 0.747 at 2 bits, Table 1).
Thank you for the effort to address my concerns. Clarifications on implementation details and additional head-selection analysis are informative and helpful. However, I think the following are still open and critical.
I maintain my score.