acceptodds
Under review as a conference paper at ICLR 2027

CoCoKV: Consensus- and Completeness-Aware KV Cache Compression for LLM Inference

Abstract

Efficient long-context inference in Large Language Models (LLMs) is severely constrained by the memory footprint of the Key-Value (KV) cache. Recent prompt-stage methods reduce this overhead by estimating historical-token importance from observation-window attention and retaining only a compact subset of the KV cache. However, their reliance on head-specific token importance gives rise to two potential limitations: i) Head-wise Attention Partiality, where individual attention heads can capture specialized yet partial views of token importance, and ii) Token-wise Information Omission, where selecting highly attended tokens leads to semantic omission while overlooking other crucial information. To address these limitations, we present CoCoKV, a novel KV cache compression framework designed to retain both reliable and comprehensive evidence under constrained cache budgets. Specifically, CoCoKV introduces a soft cross-head consensus mechanism that uses collective signals to refine head-specific attention while preserving head distinctiveness. Building on the calibrated importance scores, a cross-token completeness selection strategy is applied to promote broader semantic coverage. Evaluations across Needle-in-a-Haystack and LongBench datasets on three LLMs demonstrate that CoCoKV outperforms competitive baselines, validating its effectiveness and robustness.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV Cache Compression, LLM Inference, Consensus, Completeness

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_YZvoOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work finds that the reliance on head-specific token importance gives rise to two potential limitations: 1) individual attention heads capture part of the token importance; 2) close attention tokens lead to semantic omission while overlooking other crucial information. This work proposes CoCoKV, which is a novel KV cache compression framework. The experiment evaluates the method with NIAH benchmarks and 16 LongBench datasets.

Critical strengths
  1. This work proposes a cross-head consensus module that calibrates head-specific attention with collective evidence to improve importance reliability. This considers cross-head information.
  2. This work proposes a cross-token completeness module that selects informative tokens with broader semantic coverage. This could expand the semantic coverage.
  3. The method is easy to understand, and the paper is well-written.
  4. This work conducts experiments to support that the proposed method is useful.
Critical weaknesses
  1. The author could provide code to help reproduce the results.
  2. For Figure 6, the author should provide the full attention method to provide the baselines performance
  3. What is performance on math dataset, such as MATH datasets?
Questions for authors

N/A

3 comments

Thank you again for your constructive comments and positive assessment of our work.

As the discussion period is approaching its deadline, we would like to kindly follow up and ask whether our rebuttal has adequately addressed your concerns. We would be happy to provide any further clarifications or engage in additional discussion if needed.

I will keep the positive score for this work. The author resonse addresses the major concerns.

We are glad that our response has addressed your questions. Thanks again for your valuable suggestions.

Reviewer_Syf8OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CoCoKV combines a soft cross-head consensus mechanism with a cross-token completeness selection strategy that keeps broader semantic coverage under tight cache budgets. On Needle-in-a-Haystack and LongBench, it beats competitive baselines.

Critical strengths

This paper is easy to follow.

Critical weaknesses

My main concern is how the method is positioned and evaluated. KV cache compression methods are now commonly evaluated in both query-aware and query-agnostic settings. The paper does not say whether CoCoKV only works in the query-aware setting, and all experiments seem like query-aware setting. This should be clarified.

Recent works currently cares more about robustness, and the query-agnostic setting is widely accepted as the harder and more realistic benchmark. If the paper only evaluates the query-aware setting, the authors should explain why it still matters in practice. For example, compressing LongBench to a 64-token budget is extreme, and in the query-aware setting it also makes the task overly easy.

The paper does not appear to evaluate the efficiency of the compression method itself.

Questions for authors

See weaknesses.

Reviewer_7vh6OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CoCoKV, a prompt-stage KV cache compression method combining cross-head consensus with cross-token completeness. It adjusts head-specific importance using cross-head attention statistics, then selects tokens using both importance and distance from previously selected keys. Experiments on three language models show improvements over the evaluated compression baselines on LongBench.

Critical strengths
  1. A clear and reasonable motivation.

Importance-based selection can retain redundant tokens, while individual heads may provide incomplete importance estimates. Combining cross-head information with diversity addresses these two limitations in a coherent way.

  1. Encouraging results across multiple models.

The evaluation covers three language models and 16 LongBench datasets. CoCoKV improves average performance over the evaluated compression baselines on all three models. The budget sweeps and component ablations provide additional evidence for the method's effectiveness.

Critical weaknesses
  1. The connection between Lemma 1 and Eq. (11) needs clarification.

    Lemma 1 gives a valid coverage bound for pure farthest-point selection. However, Eq. (11) also includes an importance score, so it need not select the farthest point at each step. The proof therefore does not establish the same bound for Eq. (11). Please clarify whether the lemma is intended only to motivate the completeness term or whether a coverage guarantee is also claimed for the combined rule. The latter would require additional analysis.

  2. A direct comparison with MixKV is missing.

    MixKV (Liu et al., 2025b), which the paper cites, also combines importance and diversity and includes text-only LongBench experiments on Mistral-7B-Instruct-v0.2. Figure 4b shows that both modules help within CoCoKV, but does not show whether CoCoKV improves over MixKV. Please compare the two under the same model, cache budget, and evaluation settings.

  3. The runtime comparison needs more detailed measurements.

    Section 4.1.4 mainly supports the low-overhead claim by reporting approximately four GPU hours for complete evaluations of several methods. This does not sufficiently characterize inference overhead, especially since the greedy selection cost grows approximately quadratically with the retained-token budget. Please report compression time, time to first token, decoding latency, and actual memory usage across representative input lengths and cache budgets. Performing compression only once during the prompt stage does not by itself establish that its overhead is negligible.

Questions for authors

Please refer to the weaknesses above.

1 comment

Dear reviewer, thank you again for your time and consideration.

We would like to kindly follow up on our rebuttal, as the discussion period is approaching its deadline. If you have a chance, we would greatly appreciate it if you could review our responses and let us know whether our clarifications have addressed your concerns.

Thanks very much for your time and consideration.

Comments on the paper

2 comments

Dear Reviewers,

The authors have posted their rebuttals and updated their manuscript/responses. Please carefully read the authors' responses and engage in the discussion to clarify any remaining concerns.

In light of the rebuttal, please let the authors know whether your concerns have been adequately addressed, and update your review comments and scores if appropriate. Active participation is essential for us to reach a well-calibrated decision.

Please note that the author-reviewer discussion phase ends on November 18 at 11:59 PM AoE, after which no further responses can be posted. Thank you for your continued time and effort.

Best regards,

Area Chair

AuthorsOpenReview ↗

Summarization of Rebuttal

Dear Area Chair and reviewers,

Thanks for your time and effort in reviewing CoCoKV.

In the last period, we would like to summarize our rebuttal in the following three points:

  1. Novelty. CoCoKV introduces Cross-head Consensus to estimate token importance from agreement across heads, and Cross-token Completeness to capture information missing from the retained set. We additionally compared CoCoKV with MixKV, a closely related baseline, and observed substantial gains in average LongBench performance on all three models.
  2. Generality. We extended our evaluation to two additional settings. For query-agnostic compression, CoCoKV achieved comparable performance on LongBench. For mathematical reasoning, it outperformed both existing baselines on MATH and GSM8K.
  3. Efficiency. We analyzed the overhead of CoCoKV’s greedy selection, which is restricted to a small set of candidate tokens. We further fuse the entire loop into a single GPU kernel to accelerate it. On LongBench, CoCoKV achieves inference efficiency comparable to SnapKV and PyramidKV.

Following the rebuttal, all three reviewers confirmed that our responses addressed their concerns and recommended acceptance.

We appreciate your consideration of these results and will incorporate them into the revised manuscript.

Best regards,

The Authors of Submission 17513

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.