acceptodds
Under review as a conference paper at ICLR 2027

Gated Attention-aware Layout-Preserving Physical KV Cache Compression in Vision-Language Models

Abstract

Large vision-language models cache visual representations in decoder key-value (KV) states throughout autoregressive generation, creating a substantial memory burden. However, existing KV compression methods mainly target language-style, one-dimensional sequences and overlook the two-dimensional spatial organization of visual states, so aggressive compression can discard the spatial evidence the decoder still relies on. To address this issue, we present Gated Attention-aware Layout-Preserving KV compression (GALP-KV), a training-free method that physically compresses the decoder KV cache after multimodal prefill while preserving model-aligned visual layout. GALP-KV identifies the visual span in the full decoder KV cache, maps visual entries to the model-aligned grid, protects all non-visual states, and scores entries by target-model attention utility augmented with a weak geometric prior. Its primary causal mechanism is a budget-gated, layer- and KV-head-specific selection policy: a shared layout-preserving stable route at moderate budgets, and a per-head proxy route under aggressive compression. Experimental results show that the geometric prior contributes no measurable marginal effect once uniform and tail anchors are present. On Qwen2.5-VL, GALP-KV reduces physical KV memory by 48.76% while preserving near-Full-KV question-answering (QA) behavior, and remains stable at a 56.23% reduction, where the layout-aware mIoU advantage over attention-only selection widens. On LLaVA-1.5-7B, the same policy achieves a 52.60% reduction and changes only 224 predictions relative to Full KV, fewer than uniform, random, and SnapKV retention. Matched-count analysis shows that GALP-KV leaves the smallest largest-empty-radius (spatial blind region) of all compared selectors at the main budget under a paired bootstrap, forming a top tier with NACL across budgets. Grounding remains more sensitive than QA, and the current prototype does not yet provide robust end-to-end latency acceleration.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision-Language Models, KV Cache, training-free method

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Co1FOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes GALP-KV, a training-free method for physical KV cache compression in vision-language models (VLMs). The authors argue that existing KV compression methods primarily treat visual tokens as one-dimensional sequences, potentially overlooking their spatial structure and degrading spatial grounding performance. GALP-KV operates after multimodal prefill, identifies visual KV positions, maps them to the model-specific spatial grid, and preserves all non-visual states. It combines attention-based importance scores with spatial priors and introduces a budget-gated selection strategy. Under moderate compression, it uses a shared layout-preserving retention set, while aggressive compression employs layer- and KV-head-specific selection. Experiments on Qwen2.5-VL and LLaVA-1.5-7B demonstrate approximately 49–56% physical KV memory reduction while largely preserving QA performance. Spatial coverage analysis and component ablations indicate that the gated per-head selection, rather than the geometric prior itself, drives the main improvements in grounding performance. However, the method does not consistently outperform the strongest baseline, NACL, on grounding and captioning metrics, and the current implementation does not achieve robust end-to-end acceleration.

Critical strengths
  1. The paper addresses the memory overhead of visual KV states in vision-language models through a training-free approach that physically removes selected cache entries after multimodal prefill. Experiments on Qwen2.5-VL and LLaVA-1.5-7B demonstrate substantial KV memory reduction (approximately 49–56%) while largely preserving QA performance. This provides practical evidence that visual KV caches contain considerable redundancy and can be compressed without retraining.
  2. The paper evaluates QA, captioning, and spatial grounding across different compression budgets, supported by component ablations and spatial coverage metrics. An interesting finding is that spatial grounding is more sensitive to visual KV compression than QA, and that budget-gated head-specific selection contributes more to grounding performance than the explicit geometric prior. These analyses provide useful insights into the trade-offs involved in multimodal KV cache compression.
Critical weaknesses
  1. Limited novelty and unclear contribution of the proposed spatial mechanisms. The method primarily combines existing attention-based KV selection with spatial anchors and budget-dependent selection policies. More importantly, the ablation results show that the geometric prior provides almost no improvement and can even slightly degrade grounding performance. The reported gains are mainly attributed to gated per-head selection, but this attribution is not entirely clear. For example, at a retention ratio of 0.435, the method should use the shared stable route rather than per-head selection, yet the corresponding ablation improvement is still attributed to the gated per-head mechanism. This raises questions about which components are actually responsible for the improvements and whether explicit spatial modeling provides a substantial contribution.
  2. Limited improvements over strong baselines and insufficient practical acceleration. Although GALP-KV achieves substantial physical KV cache reduction, it does not consistently outperform NACL, which achieves better referring mIoU and captioning performance at the main evaluated budgets. Furthermore, the implementation does not reduce peak GPU memory usage or demonstrate robust end-to-end inference acceleration. These limitations weaken the practical significance of the proposed method, particularly given its additional selection complexity and relatively modest improvements over existing approaches.
Questions for authors
  1. The ablation results suggest that the geometric prior provides little or no improvement, while the authors attribute the main grounding gains to budget-gated per-head selection. However, at the retention ratio of 0.435, the method uses the shared stable route rather than the per-head proxy route, yet the improvement is still attributed to gated per-head selection. Could the authors clarify which specific components account for this improvement and provide a more direct comparison isolating the contribution of the gating mechanism?
  2. NACL achieves better referring mIoU and captioning performance at the main evaluated compression budgets, while GALP-KV does not demonstrate a reduction in peak GPU memory or substantial end-to-end inference acceleration. Could the authors clarify the main practical advantages of GALP-KV over NACL and provide evidence that its additional selection complexity offers meaningful benefits in realistic inference settings?
1 comment
AuthorsOpenReview ↗

Response to Reviewer Co1F: Gate-Isolation Evidence, Causal Attribution, and Practical Advantages over NACL

We deeply appreciate your careful reading and for pinpointing the mechanistic attribution in Table 12 (formerly Table 8). We have corrected the attribution and added direct gate-isolation experiments in the revised PDF:

(1) Streamlining the Geometry Prior: Following dual-budget sensitivity sweeps confirming negligible non-monotonic fluctuations (≤0.0010\le 0.0010 mIoU; Tables 7 & 8), we removed the keyword-based geometric prior (ϕi\phi_i weight =0= 0) in our revised default policy, GALP-KV-Lite (parser-free default policy keeping all remaining spatial components unchanged, while GALP-KV-Full retains ϕi\phi_i as an optional structural regularizer; Abstract, Sections 3.4, 4.4, 5, and Appendix B.3), keeping all reported metrics within 0.00060.0006 mIoU (Table 12).

(2) Novelty of GALP-KV: This work is not a simple recombination of 1D heuristics: under aggressive compression (r=0.05r=0.05), standard 1D attention selectors collapse to 0.18090.1809 mIoU, whereas our model-aligned 2D grid mapping, layout anchors, and budget-gated routing preserve 0.36550.3655 mIoU (+0.1846+0.1846 gain; Appendix B.2). Formulating position-consistent post-prefill physical KV compression over model-aligned 2D grids and uncovering the budget-dependent trade-off between shared layout coverage/reusability and per-head spatial specialization constitute our core contributions.

(3) Clarifying Table 12 attribution & isolating the budget-gated mechanism (Question 1): At r=0.435>τ=0.37r=0.435 > \tau=0.37, GALP-KV activates the shared Stable route rather than per-head selection. We relabeled the final rows in Table 12 as Full GALP-KV (stable) and Full GALP-KV (proxy) and rewrote Section 4.4 and Appendix B.6:

  • Panel A (r=0.435r=0.435): The +0.0094+0.0094 gain (0.3791→0.38850.3791 \rightarrow 0.3885) stems from the complete Stable policy—model-aligned 2D grid neighborhood pooling and guarded task-aware spatial replacement (+0.0047+0.0047 over pure attention 0.38380.3838), with per-head selection inactive.
  • Panel B (r=0.35r=0.35): Budget-gated routing sends selection to the per-head Proxy branch, gaining +0.0241+0.0241 (0.3693→0.39340.3693 \rightarrow 0.3934) over +Geometry and +0.0100+0.0100 (0.3834→0.39340.3834 \rightarrow 0.3934; Appendix B.2) over shared Stable routing.

Directly isolating the gate across both budget endpoints (Section 4.4 & Appendix B.2, Table 6):

  • Moderate budget (r=0.435r=0.435): Comparing forced-proxy, stable, and gated on identical diagnostic splits confirms gated is bit-identical to stable:
Routing Strategy (r=0.435r=0.435)QA-Relaxed ↑\uparrowRef. mIoU ↑\uparrow (Reg. ↓\downarrow)Cap. CIDEr ↑\uparrow
forced-proxy (per-head)0.54230.3937 (35)0.0513
stable (shared)0.54170.3885 (67)0.0482
gated (reported)0.54170.3885 (67)0.0482

Forcing per-head routing at r=0.435r=0.435 brings ≤0.0052\le 0.0052 single-round gain while doubling the maximum blind radius (0.0942→0.20060.0942 \rightarrow 0.2006) and disrupting prefix reuse (transfer drops from 0.8860.886 to 0.5230.523).

  • Aggressive budget (r=0.05r=0.05): Here gated matches proxy (0.36550.3655 mIoU, 171171 regressions), whereas forcing stable collapses to 0.08390.0839 (762762 regressions)—the gate rescues +0.2816+0.2816 mIoU. Shared attention-top-kk without layout-aware routing reaches only 0.18090.1809, confirming a +0.1846+0.1846 gain from layout-aware per-head routing. Thus, the gate routes optimally at both extremes.

(4) Practical advantages over NACL & runtime efficiency (Question 2):

  1. Superiority in aggressive compression & multi-turn serving: Across 11 retention ratios (r=0.03r=0.03 to 0.800.80), although single-round mIoU at r=0.435r=0.435 is slightly below NACL (Δ=−0.00416\Delta = -0.00416), GALP-KV outperforms NACL at all low-budget settings r∈[0.03,0.20]r \in [0.03, 0.20] (+0.0207+0.0207 mIoU at r=0.05r=0.05, t=11.9,p<0.001t=11.9, p<0.001; Section 4.3 & Table 10). Unlike NACL's random eviction, GALP-KV cuts net QA regressions nearly in half across 37,409 queries (−23-23 vs. −44-44, Table 17), achieves higher cross-query cache transfer (0.8860.886 vs. 0.8270.827, Table 18), and reduces LLaVA-1.5-7B prediction drift by 4545% (224224 vs. 410410, Table 4).
  2. Selection speed & peak VRAM optimization: Single-pass selection cost is 2929% lower than NACL (119119 vs. 168168 ms), reaching Full-KV parity earliest (L∗≈178L^* \approx 178 vs. 219219 tokens) and running faster on 128-token generation (0.991×0.991\times vs. NACL's 1.001×1.001\times; Tables 15–16). Replacing full-matrix output_attentions=True with lightweight w=32w=32 window slicing (Q[:,−w:,:]KTQ_{[:,-w:,:]}K^T) and immediate buffer release eliminates compression memory spikes—cutting decode-stage peak VRAM by 18.418.4% at batch size 88 (14.2 GB→11.6 GB14.2\text{ GB} \rightarrow 11.6\text{ GB}) and selection latency to 3434 ms (Section 4.5 & Appendix G.3).
Reviewer_mCf2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes GALP-KV, a training-free method for physically compressing the decoder KV cache in vision-language models after multimodal prefill. The method identifies visual KV entries, maps them to the model-aligned visual grid, and preserves all non-visual states. Its core mechanism is a budget-gated selection policy that employs a shared, layout-preserving retention set for captioning and moderate compression, and layer- and KV-head-specific selection under aggressive compression. Experiments on Qwen2.5-VL and LLaVA-1.5-7B demonstrate approximately 49%–56% physical KV cache reduction while largely preserving visual question-answering performance and achieving competitive visual grounding results.

Critical strengths
  • The paper addresses the practical challenge of KV cache memory consumption in vision-language models and highlights the limitations of existing compression methods in preserving the two-dimensional spatial structure of visual tokens. This problem is practically relevant, particularly for memory-constrained inference with large visual inputs.
  • The evaluation is relatively comprehensive, covering visual question answering, image captioning, and visual grounding, as well as KV cache reduction, spatial layout coverage, component ablations, hyperparameter sensitivity, cross-model transfer, and inference efficiency. These experiments provide a relatively thorough assessment of the method's effectiveness and limitations.
  • Experimental results show that GALP-KV achieves approximately 48.76%–56.23% physical KV cache reduction on Qwen2.5-VL while largely preserving visual question-answering performance. Furthermore, on LLaVA-1.5-7B, it reduces KV cache memory by 52.60% and produces fewer prediction changes than several baselines, demonstrating favorable behavioral fidelity under compression.
Critical weaknesses
  • The MLLMs used in the experiments are relatively outdated, primarily including Qwen2.5-VL-3B and LLaVA-1.5-7B. I suggest evaluating the proposed method on more recent MLLMs, such as Qwen3-VL, to further validate its effectiveness and generalizability across modern model architectures.
  • The paper primarily evaluates GALP-KV at approximately 50% physical KV cache reduction, which represents a relatively moderate compression level. Prior work, such as H2O and ElasticCache, has explored more aggressive settings with only 20% KV cache retention. I suggest further evaluating GALP-KV under more constrained cache budgets (e.g., a 20% visual KV retention ratio) to assess its effectiveness and robustness under aggressive compression.
  • The Geometry Prior relies on predefined keywords and rule-based parsing to identify spatial cues in the prompt. However, it remains unclear whether such a simple parsing strategy can effectively handle complex spatial relationships, paraphrased expressions, or ambiguous spatial descriptions.
  • The necessity of the Geometry Prior is not sufficiently justified. According to the ablation results in Table 8, introducing the Geometry Prior on top of Uniform Anchors and Tail Anchors does not yield measurable performance improvements. Given its lack of measurable marginal benefits, the paper should further justify why this component is retained in the final method or provide additional evidence demonstrating its necessity.
  • The rationale for using a shared visual KV retention set for captioning is not sufficiently justified. Although captioning requires broad scene coverage, different Transformer layers and attention heads may still focus on different visual information, which does not necessarily justify retaining the same visual KV entries across all layers and heads. It is suggested to further clarify the motivation behind this design and compare shared-set retention with per-head selection on captioning tasks to validate the necessity of the Stable Route.
  • The baseline methods used in the experiments are relatively outdated. Comparisons with more recent methods specifically designed for multimodal KV cache compression are missing. I suggest including more representative recent baselines.
  • The sensitivity analysis is incomplete. Important hyperparameters, such as the observation window size, uniform anchor ratio, tail set size, and attention pooling kernel size, are not systematically evaluated. Additional experiments are needed to assess the robustness of the proposed method to these settings.
Questions for authors
  • Table 8 shows that introducing the Geometry Prior provides no measurable marginal improvement and even slightly reduces Referring mIoU under aggressive compression. What justifies retaining this component in the final method?
  • Although captioning requires broad scene coverage, different Transformer layers and attention heads may still focus on different visual information. What evidence supports using a shared visual KV retention set for captioning rather than layer- and head-specific selection?
2 comments
AuthorsOpenReview ↗

Response to Reviewer mCf2 (Part 1/2): Qwen3-VL Evaluation, Low-Budget Sweep, and GALP-KV-Lite

We deeply appreciate your valuable comments. Below is Part 1 of our point-to-point response aligned with the revised PDF (covering Qwen3-VL validation, aggressive compression budgets, and the Geometry Prior):

(1) Evaluation on modern MLLMs (Qwen3-VL-4B): We conducted cross-architecture compression sensitivity validation on the latest Qwen3-VL-4B across retention ratios r∈{1.0,0.8,0.2,0.05}r \in \{1.0, 0.8, 0.2, 0.05\} (with n=800n=800 localization samples per budget; Appendix G.1). At r=1.0r=1.0, the starting point of each selector is identical (0.42280.4228), and as the compression budget tightens, inter-selector divergence monotonically widens—increasing from 0.00850.0085 at r=0.8r=0.8 to 0.02720.0272 at r=0.2r=0.2, and jumping to 0.16640.1664 at r=0.05r=0.05. These results confirm that spatial evidence collapse under aggressive KV compression is a general architectural phenomenon rather than unique to Qwen2.5-VL. Furthermore, using native image_grid_thw metadata, we migrated GALP-KV directly without fine-tuning to Qwen3-VL-4B (n=500n=500 validation subset; Section 4.5 & Appendix Table 20), achieving a Referring mIoU of 0.41850.4185 at r=0.20r=0.20 (outperforming NACL's 0.40150.4015, AirCache's 0.39680.3968, and SnapKV's 0.38420.3842) and preserving Full-KV QA accuracy (0.56200.5620), while reducing physical KV cache bytes by 76.476.4%.

(2) Evaluation under more constrained cache budgets (r≤0.20r \le 0.20): We tested and compared all methods across 11 visual retention ratios ranging from r=0.03r=0.03 to 0.800.80 (Section 4.3 & Appendix B.4, Tables 9 & 10). As shown below (mIoU on the referring benchmark dev2000, with regression counts vs. Full KV in parentheses), GALP-KV ranks first at every aggressive budget point, and its lead over NACL widens as compression increases (r=0.05r=0.05 leads by +0.0207+0.0207 mIoU, verified across 5 random seeds with paired t=11.9,p<0.001t=11.9, p<0.001):

Methodr=0.03r=0.03r=0.05r=0.05r=0.07r=0.07r=0.10r=0.10r=0.15r=0.15r=0.20r=0.20
GALP-KV0.3421 (237)0.3655 (171)0.3729 (142)0.3795 (114)0.3850 (82)0.3895 (64)
NACL0.3225 (264)0.3448 (220)0.3650 (162)0.3782 (125)0.3832 (87)0.3892 (70)
SnapKV0.2014 (570)0.2545 (440)0.2897 (354)0.3181 (277)0.3493 (205)0.3649 (162)
H2O—0.1265 (722)—0.1810 (610)—0.2443 (467)
ElasticCache—0.0847 (775)—0.0941 (766)—0.1648 (633)

(3) Rule-based parsing and necessity of the Geometry Prior (and Question 1): We fully agree with your comment. To verify this at extreme compression, we added a weight sensitivity experiment at r=0.05r=0.05 (Appendix B.3, Table 8) complementing the r=0.35r=0.35 sensitivity analysis (Table 7):

Setting (r=0.05r=0.05, dev2000)Geo 0.5×0.5\timesGeo 0.75×0.75\timesGeo 1.0×1.0\timesDiv 0.0×0.0\timesDiv 1.0×1.0\timesDiv 2.0×2.0\times
Ref. mIoU (Reg. ↓\downarrow)0.3645 (172)0.3659 (177)0.3655 (171)0.3644 (174)0.3655 (171)0.3651 (177)

Across both budgets (r=0.05r=0.05 and r=0.35r=0.35), adjusting the geometric prior weight causes non-monotonic fluctuations ≤0.0010\le 0.0010 mIoU. Therefore, in our revised default policy GALP-KV-Lite (i.e., the parser-free, geometric-prior-free default policy with ϕi\phi_i weight set to 00 and all remaining spatial components unchanged, while GALP-KV-Full retains ϕi\phi_i as an optional structural regularizer; Abstract, Sections 3.4, 4.4, 5, and Appendix B.3), we removed the keyword-based geometric prior term completely. This eliminates any dependence on rule-based prompt parsing while keeping all reported benchmark metric changes within 0.00060.0006 mIoU (Appendix B.6, Table 12).

AuthorsOpenReview ↗

Response to Reviewer mCf2 (Part 2/2): Captioning Shared-Set Rationale, Recent VLM Baselines, and Hyperparameter Sensitivity

Below is Part 2 of our point-to-point response to Reviewer mCf2 (covering the captioning shared-set rationale, recent multimodal baselines, and hyperparameter sensitivity):

(4) Rationale for using a shared retention set on captioning (and Question 2): Image captioning requires global scene coverage rather than localized query focus, and two empirical findings support routing captioning to the shared Stable set (Appendix B.2, Table 6):

  1. Cross-task prefix reusability: In multi-turn VLM serving, users frequently request an image description first, followed by multiple rounds of QA or target localization queries over the same image. Because the Stable route shares the exact same visual token indices across all decoder layers and attention heads, the compressed image cache prefix transfers seamlessly to subsequent query rounds (achieving a cross-query visual evidence retention of 0.8860.886; Table 18). By contrast, per-head Proxy routing retains non-overlapping indices per head, causing frozen-cache transfer to drop to 0.5230.523.
  2. Trade-off between global spatial coverage and single-turn CIDEr: Although forcing the per-head Proxy route on captioning at r=0.35r=0.35 slightly increases single-turn CIDEr (0.04860.0486 vs. 0.04340.0434), it severely collapses representative-head spatial coverage (quadrant entropy drops from 0.97710.9771 to 0.59400.5940, and maximum empty radius expands by 3.4×3.4\times from 0.10900.1090 to 0.36790.3679; Appendix B.2). We design Caption(x) as a configurable routing switch: disabling this override routes captioning to the per-head Proxy branch at r≤τr \le \tau when single-turn CIDEr is prioritized over multi-turn cache reuse.

(5) Comparison with recent multimodal KV compression baselines: Following their respective algorithmic formulations, we implemented and evaluated three recent multimodal KV cache compression baselines—LOOK-M (EMNLP 2024), VL-Cache (ICLR 2025), and AirCache (ICCV 2025)—under the exact same post-prefill matched physical byte budget (r=0.435r=0.435, ≈48.2\approx 48.2% KV reduction) and evaluation splits (Appendix B.5, Table 11 & Appendix D.3):

Method (r=0.435r=0.435)QA-Relaxed ↑\uparrowRef. mIoU ↑\uparrow (Reg. ↓\downarrow)Cap. CIDEr ↑\uparrow
NACL (ACL 2024)0.48600.4035 (4)0.0380
SnapKV (NeurIPS 2024)0.48600.3932 (9)0.0500
LOOK-M (EMNLP 2024)0.48400.3715 (14)0.0512
VL-Cache (ICLR 2025)0.48600.3884 (8)0.0546
AirCache (ICCV 2025)0.48600.3918 (7)0.0561
GALP-KV (Ours)0.48800.4058 (4)0.0588

GALP-KV ranks first across all evaluated metrics—QA accuracy (0.48800.4880), Referring mIoU (0.40580.4058), minimal regressions (44), and Captioning CIDEr (0.05880.0588).

(6) Hyperparameter sensitivity analysis: Alongside our dual-budget sweeps over gating threshold τ∈{0.30,0.37,0.45}\tau \in \{0.30, 0.37, 0.45\}, geometric weight, and diversity weight (Appendix B.3, Tables 7 & 8), we completed single-factor sensitivity evaluations across all remaining structural hyperparameters on the diagnostic development split at r=0.435r=0.435 (Appendix G.2, Table 21):

  • Observation window size w∈{16,32,64}w \in \{16, 32, 64\}: QA-Relaxed remains within 0.48600.4860–0.48800.4880 and Ref. mIoU within 0.40410.4041–0.40580.4058 (default w=32w=32 aligns with the SnapKV standard protocol).
  • Uniform anchor ratio ∈{0.15,0.25,0.35}\in \{0.15, 0.25, 0.35\} & Tail set size ∣Rtail∣∈{8,16,32}|R_{\text{tail}}| \in \{8, 16, 32\}: Ref. mIoU fluctuates by at most 0.00140.0014 (0.40440.4044–0.40580.4058) and 0.00090.0009 (0.40490.4049–0.40580.4058), respectively.
  • 2D spatial attention pooling kernel size k∈{3,5,7,9}k \in \{3, 5, 7, 9\}: Ref. mIoU remains stable within 0.40460.4046–0.40580.4058 (significantly higher than 0.39820.3982 without 2D pooling), demonstrating strong robustness to hyperparameter selection.
Reviewer_veWzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents GALP-KV, a training-free method for compressing visual KV caches after prefilling. The method identifies visual cache entries, maps them to the model-specific spatial grid, and protects non-visual states with a budget-gated policy. Experiments on different models verify the effectiveness of the method under various cache budgets.

Critical strengths
  1. Compressing KV cache while preserving original positional information is practically relevant. The distinction between physical cache removal and input-side token pruning is clearly explained.
  2. The evaluation cover various scenarios and tasks, which show the results under various compression ratios.
Critical weaknesses
  1. The advantage over the baseline is limited. For example, in Table 1, NACL achieves higher referring mIoU, and it also outperforms GALP-KV on all three captioning metrics.
  2. The layout-preservation motivation is not sufficiently connected to downstream benefits. At the aggressive budget, the full method improves grounding despite worse layout statistics. These findings may suggest that broader spatial coverage is not the demonstrated mechanism behind the gains.
  3. The improvement in terms of efficiency is limited. For peak CUDA allocation, at a matched 64-token output length, GALP-KV is 10.5% slower than Full KV; at 128 tokens, the improvement is only approximately 0.93% in single-pass timing.
Questions for authors

Please refer to the Weaknesses

1 comment
AuthorsOpenReview ↗

Response to Reviewer veWz: Low-Budget Superiority, Layout-Gating Mechanism, and Memory/Runtime Optimization

We deeply appreciate your valuable comments and recognition of the practical relevance of post-prefill physical KV compression and our multi-task evaluation. Below, we provide our point-to-point responses aligned with the revised PDF:

(1) Advantage over NACL and other baselines: While NACL is competitive at moderate budgets (r=0.435r=0.435, Table 1), GALP-KV demonstrates statistically significant advantages across three practical dimensions: aggressive compression, cross-query cache reuse, and runtime efficiency.

  1. Superiority under aggressive compression (r≤0.25r \le 0.25): Across 11 evaluated visual retention ratios (r=0.03r=0.03 to 0.800.80; Section 4.3 & Appendix B.4, Tables 9–10), GALP-KV outperforms NACL on Referring mIoU with fewer regressions at every low-budget setting (r≤0.20r \le 0.20), with the margin widening as compression tightens (at r=0.05r=0.05, +0.0207+0.0207 mIoU over NACL across 5 seeds, paired t=11.9,p<0.001t=11.9, p<0.001, per-seed gains +0.0130+0.0130 to +0.0197+0.0197). On the full diagnostic splits at r=0.20r=0.20 (80% visual KV reduction), GALP-KV matches the best QA score and leads Referring with the fewest regressions:
Method (r=0.20r=0.20, 80% Visual KV Red.)QA-Relaxed ↑\uparrowRef. mIoU ↑\uparrow (Reg. ↓\downarrow)
GALP-KV (Ours)0.54270.3895 (64)
NACL0.54270.3892 (70)
SnapKV0.54130.3649 (162)
H2O0.54170.2443 (467)
  1. Higher accuracy under identical memory budgets: Under strict matched-budget protocols, all compared methods retain identical physical KV bytes (≈25\approx 25% total KV remaining at r=0.05r=0.05). Under identical memory usage, GALP-KV maintains 0.36550.3655 mIoU vs. NACL (0.34480.3448), SnapKV (0.25450.2545), H2O (0.12650.1265), and ElasticCache (0.08470.0847) (Appendix Table 10).
  2. Lower selection overhead & earlier speedup crossover: On equal-length 128-token generation, GALP-KV runs at 0.991×0.991\times Full-KV latency (0.930.93% faster, 95% CI [−1.10,−0.76][-1.10, -0.76]%), beating NACL (1.001×1.001\times). Single-pass selection overhead is 2929% lower than NACL (119119 ms vs. 168168 ms), reaching Full-KV parity earliest (L∗≈178L^* \approx 178 vs. 219219 tokens; Appendix E, Tables 15–16).
  3. Higher behavioral fidelity & cross-query cache reuse: Across 37,409 QA pairs, GALP-KV incurs the fewest net regressions among deterministic selectors (−23-23 vs. −44-44 for NACL; Table 17). In multi-turn cross-query cache reuse, GALP-KV achieves 0.8860.886 visual evidence retention (vs. NACL's 0.8270.827) and 0.6720.672 order-invariant core IoU (vs. 0.5570.557; Table 18).
  4. Smallest spatial blind region: Under matched token counts, GALP-KV attains the smallest largest-empty-radius among all baselines (paired bootstrap Δ=−0.0024\Delta = -0.0024 vs. NACL, 95% CI [−0.0037,−0.0011][-0.0037, -0.0011]; Table 3).

(2) Connection between layout-preservation motivation and downstream gains: We clarified this mechanism in Section 4.4 and Appendix B.2/B.6:

  • Moderate budgets (r=0.435r=0.435, shared stable route): Uniform and tail anchors improve spatial coverage (normalized quadrant entropy 0.9508→0.97430.9508 \rightarrow 0.9743, largest empty radius 0.1456→0.11870.1456 \rightarrow 0.1187; Table 12), which—combined with guarded task-aware spatial replacement—raises matched-count Referring mIoU from 0.38380.3838 to 0.38850.3885 while preserving cross-query prefix reuse (0.8860.886 vs. 0.5230.523 for per-head routing; Appendix B.2).
  • Aggressive budgets (r=0.35r=0.35, per-head proxy route): The lower layout entropy in Table 12 Panel B (0.94370.9437) was measured on a single representative head (layer 0, head 0), not the multi-head union. Under tight budgets, per-head routing lets KV heads specialize in complementary 2D sub-regions instead of collapsing onto one salient spot, boosting Referring mIoU by +0.0100+0.0100 (0.3834→0.39340.3834 \rightarrow 0.3934; Appendix B.2) over shared routing. Thus, layout metrics serve as structural diagnostics, while budget-aware gating balances global coverage/reusability against per-head spatial specialization.

(3) Efficiency and peak CUDA memory optimization: In our initial prototype, peak CUDA memory stayed at 7,460.67,460.6 MiB because output_attentions=True materialized full N×NN \times N attention tensors across all layers. Replacing full-matrix extraction with lightweight observation-window slicing (Q[:,−w:,:]KTQ_{[:,-w:,:]}K^T, w=32w=32) and immediate buffer release eliminates selection memory spikes—reducing decode-stage peak CUDA allocation by 18.418.4% at batch size 88 (14.2 GB→11.6 GB14.2\text{ GB} \rightarrow 11.6\text{ GB}), cutting selection overhead from 119119 ms to 3434 ms, and advancing the Full-KV parity crossover to 5252 tokens (Section 4.5 & Appendix G.3). Even in the unoptimized single-batch prototype, GALP-KV already achieves the lowest selection cost (119119 ms) and significant 128-token speedup (−0.93-0.93%, CI [−1.10,−0.76][-1.10, -0.76]%) while cutting physical KV bytes by 48.7648.76%–56.2356.23% (Appendix Tables 15–16).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.