acceptodds
Withdrawn from ICLR 2027

GeoKV: Geometry-Aware KV Cache Compression via Global Structural Contribution and Key Rectification

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-Language Models (VLMs) have demonstrated remarkable capabilities, yet the rapidly growing memory consumption of Key-Value (KV) caches poses a major challenge for efficient inference. Existing KV cache compression methods predominantly characterize token importance from a query-dependent functional perspective, such as attention or local relevance, while overlooking the global structural role of cached representations. Consequently, tokens with low current-query relevance may still be important for preserving underrepresented directions and the global geometry of the cached-key space. To address this limitation, we propose GeoKV, a global-structure-aware framework for KV cache compression. Specifically, we introduce Global Structural Contribution (GSC), a query-independent criterion derived from the marginal gain of a regularized log-determinant objective, and perform greedy selection to preserve structurally valuable tokens. We further introduce a lightweight geometry-aware key rectification module based on a low-rank feature-space transformation to mitigate compression-induced geometric distortion. Extensive experiments on VLMs demonstrate that GeoKV effectively reduces KV cache size while maintaining model performance. Comprehensive analyses further show that structural contribution captures complementary information beyond query-dependent functional relevance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV Cache Compression, Vision-Language Models, Geometric Representation, Token Selection, Key Rectification

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_MkmrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes GeoKV, a geometry-aware KV cache compression method for vision-language models. The authors argue that existing methods relying on attention scores or other query-dependent signals may overlook tokens that are important for preserving the global structure of cached keys. To address this, GeoKV introduces Global Structural Contribution (GSC), which measures the marginal gain of each token under a regularized log-determinant objective and uses greedy selection to retain structurally important tokens. In addition, a lightweight low-rank key rectification module is calibrated offline to reduce the geometric distortion caused by compression. Experiments on several VLMs and question-answering benchmarks show that GeoKV generally outperforms existing KV cache compression methods, particularly at low retention ratios. The authors also provide ablation studies and efficiency measurements to evaluate the two components.

Critical strengths
  1. Consistent improvements at lower KV cache compression rates. GeoKV outperforms the compared methods across multiple VLMs and benchmarks, with more noticeable gains at low KV retention ratios. For example, at 5% retention, it improves the average task accuracy over VL-Cache by 5.3 and 3.7 percentage points on LLaVA-OneVision-7B and InternVL2-8B, respectively. These results are encouraging, as maintaining model performance under such limited cache budgets is an important goal of KV cache compression.
Critical weaknesses
  1. Insufficient discussion and comparison with structure-aware KV cache compression methods. The paper argues that existing methods mainly focus on functional relevance while overlooking the structural contribution of cached keys. However, prior works such as KVCrush (cited in Sec. I Introduction) and KEYDIFF (KEYDIFF: Key Similarity-Based KV Cache Eviction for Long-Context LLM Inference in Resource-Constrained Environments, NeurIPS 2025) have already explored redundancy and diversity in KV cache selection. For example, KEYDIFF uses key similarity to identify redundant representations without relying on query-dependent attention. Although GeoKV adopts a different log-determinant-based selection criterion, the paper does not sufficiently explain its advantages over these existing approaches or provide direct experimental comparisons. This makes it difficult to assess the novelty and practical benefits of the proposed GSC mechanism.

  2. Limited evaluation of computational scalability. GSC relies on iterative greedy selection, which can become expensive as the context length increases. However, the efficiency experiments only consider inputs of 740 tokens, making it difficult to assess the method's practicality for long-context inference. Since KV compression methods are more valuable in long-context scenarios, more comprehensive efficiency experiments with varying context lengths are needed to support the practicality of the method.

Questions for authors
  1. The proposed method does not seem to rely on any VLM-specific properties and could potentially be applied to text-only LLMs. Could the authors clarify why the method is primarily designed and evaluated for VLMs, and whether similar benefits can be expected for conventional LLMs?
  2. In Table 5, GSC-only has higher geometric distortion than Random (0.320 vs. 0.311), despite better task accuracy (76.8% vs. 71.2%). This raises questions about whether the performance improvement can be attributed to the proposed geometric preservation objective. Could the authors explain this result?
  3. The rectifier is calibrated on MSCOCO but applied to different visual tasks and input distributions. Could the authors provide further evidence that the calibrated rectifier generalizes reliably across these settings?
  4. The rectifier applies a feature-space transformation to post-RoPE keys, which may alter the relative positional relationships encoded by RoPE. Have the authors examined its potential impact on RoPE's relative positional properties and the resulting attention logits?
  5. The supplementary code does not appear to include an end-to-end pipeline for reproducing the key results in Table 2 and Figure 3. Moreover, its H2O and SnapKV implementations seem to differ from those described in the original papers. Could the authors provide the complete model-integrated evaluation code and configurations used to obtain these results?
Reviewer_XQZnOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

GeoKV selects tokens that contribute new directions to the retained key space, then adjusts their keys using an offline-calibrated low-rank transformation. It reports improved VQA accuracy under cache compression and argues that preserving key geometry explains these gains.

Critical strengths
  • The ablations suggest that selecting complementary key directions improves on attention-based selection even when both methods receive independently calibrated key correction. This supports retaining useful information that attention scores alone may miss.
Critical weaknesses
  • Query independent geometric selection is already explored by KNorm (Devoto et al., 2024), KeyDiff (Park et al., 2025), and ManifoldKV (Datta et al., 2026), so the paper’s contrast with prior attention-based methods is too broad. It does not establish whether its more expensive greedy selection and calibrated correction offer a meaningful advantage over simpler geometric scorers.

  • The AirCache comparison needs reconciliation: Table 2 reports 74.2 on LLaVA TextVQA at 20% retention, whereas AirCache’s published Table 1 reports 75.3 at 10%

Questions for authors
  • Can you clarify the geometry normalization and explain why the AirCache baseline is weaker than its published results?
Reviewer_q5t5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

GeoKV compresses VLM KV caches by selecting a shared token subset across KV heads using the marginal gain of a regularized log-determinant objective. A low-rank, offline-calibrated transformation then adjusts retained post-RoPE keys to match the original second-moment geometry. The paper reports near-full-cache VQA performance at 20% retention on several VLMs, and uses geometric distortion and future attention coverage to motivate the approach.

Critical strengths
  1. The paper addresses a useful distinction between immediate query relevance and the information retained for future queries. Combining discrete token selection with a small offline-calibrated rectifier is a concrete design worth investigating.

  2. The marginal-gain calculation and the submodularity argument are clear. The appendix also independently calibrates a rectifier for SnapKV under matched settings, which is a sensible control for separating selection from rectification.

Critical weaknesses
  1. The supplied FAC implementation cannot measure the claimed coverage. It normalizes attention over retained keys only, then sums over queries and averages over retained tokens, yielding T/N′ regardless of the selected keys. This cannot support the method-dependent results in Tables 4 and 11. Separately, the D_F normalization needs reconciliation: for an unrectified, unweighted uniform fixed-size subset, Eq. (16) implies E[D_F]≥1−ρ, whereas Table 10 reports Random D_F=0.376 at ρ=0.20. This expectation bound does not prohibit every individual draw or finite-sample mean; it calls for the exact normalization, number of samples, and repeated-selection statistics.

  2. The structural LOO diagnostic in the supplement reduces exactly to ||kᵢ||₂²/||KᵀK||_F per head. It measures the magnitude of a token's second-moment perturbation, but cannot distinguish equal-norm redundant and complementary directions. This weakens the interpretation of Figure 1 as evidence for global structural complementarity and the classification of Top-LOO as a 7200 ms non-deployable oracle in Table 5: the same score can be computed with vectorized squared norms, preserving the original normalization when aggregating heads.

  3. The efficiency claim is not adequately reconciled with the algorithm. Section 3.3.1 gives O(H_KV N N′d²) selection work, but Table 13 reports the same 2.0 ms compression overhead from 5% to 50% retention; executing selection once does not remove its dependence on N′. The only reported setup has 740 input tokens, and the batch-32 throughput column is approximately the reciprocal of per-step latency without clarifying whether it is per-sequence or aggregate throughput.

  4. The novelty is insufficiently positioned against existing geometric and query-agnostic eviction methods. Section 2.1 characterizes prior methods as universally local or query-dependent, yet MixKV already incorporates key diversity in VLM compression, Compactor uses geometry-based leverage scores in a query-agnostic setting, and KVzip explicitly targets reuse across future queries. CapKV also develops a log-determinant capacity objective, albeit with different query/value-dependent scoring. GSC's determinant-lemma marginal gain and standard submodular guarantee do not by themselves establish a new selection principle.

Questions for authors
  1. Can you compare GSC with the exactly equivalent vectorized squared-norm version of the supplied Top-LOO score, using the same retention budget and rectification setting? Reporting attention-output error and task accuracy would clarify whether the benefit comes from directional coverage rather than key magnitude.

  2. Do the gains of GeoKV persist on more recently open-sourced models such as Qwen3.5-9B?

  3. What implementation or approximation explains the flat selection cost across retention ratios?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.