- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes GeoKV, a geometry-aware KV cache compression method for vision-language models. The authors argue that existing methods relying on attention scores or other query-dependent signals may overlook tokens that are important for preserving the global structure of cached keys. To address this, GeoKV introduces Global Structural Contribution (GSC), which measures the marginal gain of each token under a regularized log-determinant objective and uses greedy selection to retain structurally important tokens. In addition, a lightweight low-rank key rectification module is calibrated offline to reduce the geometric distortion caused by compression. Experiments on several VLMs and question-answering benchmarks show that GeoKV generally outperforms existing KV cache compression methods, particularly at low retention ratios. The authors also provide ablation studies and efficiency measurements to evaluate the two components.
Critical strengths
- Consistent improvements at lower KV cache compression rates. GeoKV outperforms the compared methods across multiple VLMs and benchmarks, with more noticeable gains at low KV retention ratios. For example, at 5% retention, it improves the average task accuracy over VL-Cache by 5.3 and 3.7 percentage points on LLaVA-OneVision-7B and InternVL2-8B, respectively. These results are encouraging, as maintaining model performance under such limited cache budgets is an important goal of KV cache compression.
Critical weaknesses
-
Insufficient discussion and comparison with structure-aware KV cache compression methods. The paper argues that existing methods mainly focus on functional relevance while overlooking the structural contribution of cached keys. However, prior works such as KVCrush (cited in Sec. I Introduction) and KEYDIFF (KEYDIFF: Key Similarity-Based KV Cache Eviction for Long-Context LLM Inference in Resource-Constrained Environments, NeurIPS 2025) have already explored redundancy and diversity in KV cache selection. For example, KEYDIFF uses key similarity to identify redundant representations without relying on query-dependent attention. Although GeoKV adopts a different log-determinant-based selection criterion, the paper does not sufficiently explain its advantages over these existing approaches or provide direct experimental comparisons. This makes it difficult to assess the novelty and practical benefits of the proposed GSC mechanism.
-
Limited evaluation of computational scalability. GSC relies on iterative greedy selection, which can become expensive as the context length increases. However, the efficiency experiments only consider inputs of 740 tokens, making it difficult to assess the method's practicality for long-context inference. Since KV compression methods are more valuable in long-context scenarios, more comprehensive efficiency experiments with varying context lengths are needed to support the practicality of the method.
Questions for authors
- The proposed method does not seem to rely on any VLM-specific properties and could potentially be applied to text-only LLMs. Could the authors clarify why the method is primarily designed and evaluated for VLMs, and whether similar benefits can be expected for conventional LLMs?
- In Table 5, GSC-only has higher geometric distortion than Random (0.320 vs. 0.311), despite better task accuracy (76.8% vs. 71.2%). This raises questions about whether the performance improvement can be attributed to the proposed geometric preservation objective. Could the authors explain this result?
- The rectifier is calibrated on MSCOCO but applied to different visual tasks and input distributions. Could the authors provide further evidence that the calibrated rectifier generalizes reliably across these settings?
- The rectifier applies a feature-space transformation to post-RoPE keys, which may alter the relative positional relationships encoded by RoPE. Have the authors examined its potential impact on RoPE's relative positional properties and the resulting attention logits?
- The supplementary code does not appear to include an end-to-end pipeline for reproducing the key results in Table 2 and Figure 3. Moreover, its H2O and SnapKV implementations seem to differ from those described in the original papers. Could the authors provide the complete model-integrated evaluation code and configurations used to obtain these results?