acceptodds
Under review as a conference paper at ICLR 2027

Position-Aware Cooperative Token-Channel Compression with Key-Value Decoupling

Abstract

The linearly growing key-value (KV) cache imposes a critical storage bottleneck for large language models (LLMs) in long-context inference scenarios, hindering their practical deployment. Existing methods compress the KV cache by evicting or merging tokens and reducing channel redundancy, thereby reducing its memory footprint. However, these merging methods overlook that after Rotary Position Embedding (RoPE), semantically similar tokens can yield inconsistent attention responses due to positional differences, whereas tokens with similar attention responses have higher positional-function similarity and are more mergeable. Moreover, these methods handle the token and channel dimensions in isolation, and apply a uniform compression strategy to both K and V, overlooking token-channel cooperative potential and distinct roles of K and V in the attention mechanism. In this paper, we propose a Position-Aware Cooperative Token-Channel Compression with Key-Value Decoupling method (PACT-KV). It designs positional-function similarity to measure the mergeability of tokens within the SimHash cluster for K-side compression, which jointly considers semantics and attentions, and avoids positional interference brought by physical proximity. Meanwhile we propose a cluster-conditioned K-side channel pruning to capture consistent channel-energy patterns in the same cluster, which naturally cooperates with the K-side token merging to reduce the overhead. Due to the semantic aggregation role of V, which contrasts with the query-matching role of K, PACT-KV further performs an independent V-side token merging and executes V-side channel pruning when V tokens still exceed the budget. Experimental results on LongBench and Needle-in-a-Haystack (NIAH) show that PACT-KV achieves strong performance while significantly reducing both memory footprint and inference latency.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV Cache Compression, KV-Asymmetric Compression, SimHash-Guided K-Side Token Merging and Channel Pruning, Long-Context Inference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ZQrEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PACT-KV combines position-aware token merging with channel pruning and independent key/value compression. Pre-RoPE SimHash groups constrain key merging, recent-query attention responses estimate positional compatibility, and cluster-specific channel masks compress keys. Values use separate merging and fallback channel pruning. LongBench, retrieval, and runtime experiments evaluate quality under equivalent cache budgets.

Critical strengths
  1. The asymmetric K/V design is technically sensible: it distinguishes query matching from value aggregation and connects token grouping to a structured channel mask instead of simply stacking two compression methods.
  2. Equivalent-cache-budget comparisons across three models and combined token/channel baselines provide useful support for the quality benefit. The measured runtime also makes the prefill cost visible rather than reporting decode speed alone.
Critical weaknesses
  1. The core positional proxy is based on a short recent-query window, while the merge bias is estimated from aggregated importance ratios. Figure 2 provides motivating correlation, but it does not establish that these approximations remain faithful when future queries change substantially; that limitation directly affects the reliability of irreversible prompt compression.
  2. The efficiency table includes H2O, but omits SnapKV and the token/channel eviction combinations used in the quality comparison. PACT-KV also has higher TTFT than the listed methods. Thus, the reported throughput advantage does not yet establish a general quality–latency advantage over those additional alternatives.
Questions for authors
  1. How large is attention/output error for query shifts outside the probing window, and how sensitive are downstream results to the importance-ratio bias?
  2. What is the quality–TTFT–throughput comparison against SnapKV and combined channel-pruning eviction baselines under the same measured memory budget?
2 comments

Thank you very much for your valuable and positive comments.

Weakness1 & Question1

Impact of Query Shift and Error Stability

First, we would like to clarify the scope of the problem studied in this paper. The core objective of KV Cache compression is to preserve as much information as possible that is most valuable for subsequent generation under a limited storage budget, thereby achieving a balance between the compression ratio and model performance. However, during the Prefill stage, future Queries in the Decode stage are not yet available. Therefore, existing KV Cache compression methods can typically only evaluate Token importance based on the information available in the Prompt, retaining important information while compressing or evicting less important information.

Accordingly, we investigate a more specific practical question: During natural autoregressive Decode following a single Prompt, does the shift of subsequent Queries relative to the Prefill reference Query grow with generation length, causing approximation errors to accumulate and significantly affect model performance?

To address this question, we analyze the relationship between Query Shift and Logit Approximation Error, and derive an upper bound on the Attention Distribution Error between the original Decode process and the reconstruction after token merging. Detailed derivations are provided in the newly added Appendix E. The three results are:

\leq \frac{\|q_t-q_r\|_2\|d_j\|_2}{\sqrt D}$$ $$\Delta(q_t) \leq \Delta(q_r)+\Delta_{\mathrm{shift}}(q_t,q_r) \leq \Delta(q_r)+\frac{\lVert q_t-q_r\rVert_2}{\sqrt{D}} \max_{j,k}\lVert d_j-d_k\rVert_2$$ $$D_{\mathrm{KL}}(P\|\hat P) \leq \frac{1}{8} \left[ \Delta(q_r) + \Delta_{\mathrm{shift}}(q_t,q_r) \right]^2$$ On GovReport (512 Tokens) and LongCtx30 (1,500 Tokens), we track Query Shift, Query-induced Logit Drift, and Logit Error Range during long Decode. Figure 4 (Appendix E) shows that all three metrics rise within the first few tokens, then stabilize with local fluctuations and no sustained growth. Together, the analysis and experiments show that, under the evaluated natural autoregressive settings, Query Shift-induced Logit Approximation Error does not continuously accumulate with generation length. This supports determining a fixed Token Merging mapping during Prefill and retaining it throughout subsequent Decode in PACT-KV. ## Downstream Performance Sensitivity of Importance-ratio Bias Following your suggestion, we evaluate Bias selection on PassageCount, NarrativeQA, LCC, and Multi-News, reporting the average score across the four tasks. We examine three aspects: **(1) Compensation coefficient sensitivity.** Scaling the original $\beta$ by 0.5 and 1.5 yields average scores of 25.37 and 27.96, respectively, both below the original score of 28.66. The coefficient therefore affects downstream performance, with the original setting performing best among the tested configurations. **(2) Importance measurement.** To compare integrated attention importance with Key-only similarity, we test a cosine-similarity-based alternative: $$\boxed{\beta_j=\operatorname{CosSim}(k_i,k_j)-1}$$ Its compensated attention-weight ratio is: $$\frac{\hat a_j}{\hat a_i}=\exp\left(\operatorname{CosSim}(k_i,k_j)-1\right)$$ where $\hat a_j$ and $\hat a_i$ are the attention weights of the merged Token and Anchor Token, respectively. Key cosine similarity estimates their relative contribution: higher similarity gives a ratio closer to 1, while lower similarity gives a lower relative weight. This variant scores 25.97, below the original configuration. This indicates that Key similarity alone does not fully capture relative attention contributions, and that integrated importance ratios are more suitable for our Token Merging mechanism. **(3) Alternative compensation forms.** We further evaluate two alternative forms: $\beta=0$ and $\beta=\log(1+s_j/s_i)$, which achieve average scores of 24.00 and 22.90, respectively, both substantially lower than the original configuration. The results are summarized below: | Bias Configuration | Average Score Across Four Tasks | | :------------------------ | ------------------------------: | | Original: $\log(s_j/s_i)$ | **28.66** | | $1.5\log(s_j/s_i)$ | 27.96 | | Key-based Cosine Similarity| 25.97| | $0.5\log(s_j/s_i)$ | 25.37 | | $\beta=0$ | 24.00 | | $\beta=\log(1+s_j/s_i)$ | 22.90 | The original Importance-ratio Bias achieves the highest average score (28.66), exceeding $\beta=0$ by 4.66 points and $\beta=\log(1+s_j/s_i)$ by 5.76 points. These results further confirm the importance of preserving the original relative importance differences between tokens.

Weakness2 & Question2

Rationale for Selecting the Original Experimental Baselines

First, the original efficiency evaluation focused on Token Merging methods and selected H2O as a representative baseline because its attention-importance estimation mechanism is similar to ours. We acknowledge that SnapKV is a Token-level Eviction method with low overhead. It selectively retains Tokens, incurring less Prefill overhead than Token Merging methods. Therefore, we did not select SnapKV as a primary efficiency baseline.

Second, for combining Channel Pruning and Token Eviction, the original paper evaluates the practical inference efficiency of the overall PACT-KV framework. Comparisons with representative Token-level compression methods had demonstrated PACT-KV's practical feasibility. Considering that channel-importance estimation and pruning typically introduce additional computational overhead, we did not exhaustively evaluate all Channel Pruning and Token Eviction combinations.

Furthermore, we inspected SPARK's publicly available implementation and found that its Channel Pruning modifies K/V elements through masking or restoration but still writes dense tensors of the original size back to the cache. Pruned channels are not physically removed at the storage level, and conventional dense Attention computation still processes the full channel dimension. Therefore, the nominal pruning ratio of this implementation does not directly translate into reductions in actual KV Cache capacity or Attention computation, making it difficult to use as a runtime-efficiency comparison under the same physical cache budget.

Additional Experimental Results (As Suggested)

Following your suggestion, we add SnapKV, ThinK+H2O, and ThinK+SnapKV to the runtime-efficiency evaluation. All methods are tested using LLaMA-3-8B-Instruct on a single RTX 3090 GPU with a KV Cache Budget of 256. We test 4K/8K Prompts with 128/1K Decode Tokens, reporting TTFT, TPOT, end-to-end throughput, and actual GPU memory consumption.

First, we acknowledge SnapKV's clear Prefill advantage: it achieves lower TTFT than PACT-KV in all four configurations and higher end-to-end throughput in shorter Decode scenarios. This demonstrates the efficiency advantage of lightweight Token Eviction methods for short-generation scenarios. On the other hand, PACT-KV achieves the lowest TPOT in all four configurations and higher end-to-end throughput than H2O, ThinK+H2O, and ThinK+SnapKV. At 1K Decode Tokens, PACT-KV also achieves higher end-to-end throughput than all tested methods, including SnapKV.

Furthermore, the additional experiments show that simply combining Token-level Eviction and Channel Pruning does not necessarily lead to better inference efficiency. Both ThinK+H2O and ThinK+SnapKV achieve lower end-to-end throughput than PACT-KV in our experiments. In contrast, PACT-KV adopts a cooperative design tailored to the distinct K/V representation spaces rather than simply applying two compression operations sequentially. It achieves better compression quality, lower Decode latency, and higher end-to-end throughput in longer-generation scenarios, highlighting the need for co-design across compression dimensions.

The runtime-efficiency comparison of different KV Cache compression methods is presented below.

PromptDecodeMethodTTFT (s)TPOT (ms)E2E (tok/s)Memory (GiB)
4K128SnapKV1.35139.2420.1216.54
4K128ThinK+H2O2.29452.6214.1615.78
4K128ThinK+SnapKV2.03645.2016.3115.93
4K128PACT-KV3.12432.2417.0915.93
4K1024SnapKV1.34239.8824.0316.54
4K1024ThinK+H2O2.32654.6417.3315.78
4K1024ThinK+SnapKV2.07445.1920.8216.63
4K1024PACT-KV3.17434.9925.7716.04
8K128SnapKV3.28242.8014.6419.72
8K128ThinK+H2O5.81453.0510.1516.33
8K128ThinK+SnapKV5.06748.1811.3716.42
8K128PACT-KV6.39632.9811.7816.39
8K1024SnapKV3.22540.9722.4519.76
8K1024ThinK+H2O5.82755.1616.2216.34
8K1024ThinK+SnapKV5.09046.4019.1716.68
8K1024PACT-KV6.39235.5423.5316.50
Reviewer_R9wUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PACT-KV, a training-free KV cache compression method that considers the impact of RoPE, adopts asymmetric compression strategies for keys and values, and jointly exploits token- and channel-level redundancy.

On the K side, the authors propose a positional-function-aware token merging strategy based on SimHash-guided semantic clusters and recent-query attention-response similarity. They further introduce cluster-conditioned channel pruning to exploit within-cluster channel-saliency consistency while preserving differences across clusters. On the V side, the method independently performs token merging and fallback channel pruning under a unified memory budget.

Experiments on LongBench and Needle-in-a-Haystack demonstrate improved accuracy compared with baselines such as StreamingLLM and H2O. The paper also reports higher inference throughput than the evaluated baselines.

Critical strengths
  • Clear motivation: The authors identify that semantic similarity does not necessarily imply similar attention behavior in Transformers with RoPE. This motivates incorporating positional effects into K-side token merging rather than relying solely on semantic similarity.
  • Joint token-channel compression: The method jointly considers token merging and channel pruning while adopting different compression strategies for K and V, allowing it to exploit redundancy across multiple dimensions.
  • Promising experimental results: PACT-KV achieves higher accuracy than several baselines on LongBench and Needle-in-a-Haystack. The efficiency experiments also demonstrate improved decoding latency and throughput.
  • Training-free design: The proposed method does not require additional training or fine-tuning, making it easier to apply to existing pretrained models.
Critical weaknesses
  • Substantial prefill overhead and insufficient characterization of the end-to-end efficiency trade-off: Although PACT-KV improves decoding efficiency, it introduces considerable prefill overhead. The paper does not sufficiently analyze when the decoding benefits can amortize this additional cost.
  • Limited theoretical justification for the proposed compression heuristics: Several key components, including the merging criteria, importance-based logit compensation, and channel pruning strategies, are primarily heuristic. More theoretical analysis or error characterization is needed to justify these design choices.
  • Insufficient analysis of token importance estimation overhead: Global Attention Importance theoretically involves (O(L^2)) attention statistics. It is unclear whether these statistics are collected by reusing the original prefill attention computation or through an additional attention pass. If the former, how are these statistics collected without compromising the memory-efficient computation of FlashAttention, which avoids materializing the full attention matrix? The authors should clarify the implementation and report the corresponding latency and memory overhead.
  • Missing latency breakdown and analysis of acceleration sources: Although the paper reports improvements in decoding latency and throughput, it does not provide a component-level latency breakdown. Consequently, it is unclear how much acceleration comes from token merging, channel pruning, and compact attention computation, respectively.
Questions for authors
  1. Token eviction before compression: Why does PACT-KV first evict low-importance tokens before performing compression? If the eviction stage is removed while keeping the remaining components unchanged, would the performance degrade?
  2. Logical-to-physical token mapping: PACT-KV requires mappings from logical tokens to physical K/V tokens during attention computation. How are these mappings implemented in the GPU kernels, and what is their runtime overhead?
  3. Sources of acceleration: What are the main sources of the reported inference speedup? Could the authors provide a component-level latency breakdown and quantify the contribution of each optimization?
  4. Channel pruning compensation: What is the runtime overhead of the compensation operations introduced by K-side and V-side channel pruning? Are these operations fused into the attention kernels?
  5. Practical benefits of K/V decoupling: Does independent K/V compression directly reduce the computational cost of GPU attention, or is its primary benefit improved accuracy preservation under a fixed memory budget? Could the authors provide experimental evidence distinguishing these benefits?
7 comments

Thanks very much for your valuable and detailed comments. We respond to each concern below.

Weakness 1: Substantial prefill overhead and insufficient end-to-end efficiency trade-off

Regarding Prefill Overhead

PACT-KV introduces additional computational overhead during the Prefill stage, but this overhead mainly occurs during KV Cache compression and is incurred only once for each request. In contrast, the compressed KV Cache continuously reduces the computational overhead during subsequent Decode stages. Therefore, evaluating the practical efficiency of PACT-KV requires considering the entire generation process rather than judging it solely based on Prefill latency.

Regarding the End-to-End Efficiency Trade-off

The original paper has already conducted a practical evaluation of this efficiency trade-off. To investigate whether the additional overhead during the Prefill stage would offset the subsequent benefits from decoding in short-generation scenarios, we specifically included an experiment with a short Decode length of 128 tokens. Table 5 reports TTFT, TPOT, and E2E throughput, with two different Decode lengths of 128 and 1K tokens. Notably, even with the shorter 128-token Decode setting, PACT-KV still achieves the highest end-to-end throughput. This demonstrates that the additional Prefill overhead does not offset the benefits brought by the Decode stage.

Although the original paper has already evaluated this efficiency trade-off through end-to-end throughput experiments, following your suggestion, we further conduct a quantitative analysis. We establish an end-to-end latency model based on the TTFT and TPOT reported in Table 5. Since TTFT already includes Prefill and the first Decode step, while TPOT represents the average latency of subsequent Decode tokens, the end-to-end latency for generating NN tokens can be expressed as: T(N)=Tfirst+(N−1)Tdecode.T(N)=T_{\mathrm{first}}+(N-1)T_{\mathrm{decode}}.

Here, TfirstT_{\mathrm{first}} denotes TTFT, and TdecodeT_{\mathrm{decode}} denotes TPOT. For PACT-KV (denoted by PP) and a baseline method (denoted by BB), when PACT-KV has higher TTFT but lower TPOT, we equate their end-to-end latencies: TfirstP+(N∗−1)TdecodeP=TfirstB+(N∗−1)TdecodeB.T_{\mathrm{first}}^{P}+(N^{*}-1)T_{\mathrm{decode}}^{P}=T_{\mathrm{first}}^{B}+(N^{*}-1)T_{\mathrm{decode}}^{B}.

Rearranging this equation gives the break-even point:

{T_{\mathrm{decode}}^{B}-T_{\mathrm{decode}}^{P}}}$$ When the actual generation length satisfies $N>N^*$, PACT-KV achieves lower end-to-end latency than the corresponding baseline method. Based on the measured results in Table 5, we obtain the following break-even points: | Prompt Length | Compared Method | Break-even Decode Length $N^*$ | | ------------- | --------------- | ------------------------------ | | 4K | H2O | 71 Tokens | | 4K | CaM | 66 Tokens | | 4K | KeepKV | 36 Tokens | | 8K | H2O | 94 Tokens | | 8K | CaM | 46 Tokens | | 8K | KeepKV | 75 Tokens | These results show that under both 4K and 8K Prompt settings, the break-even points of PACT-KV relative to all compared methods are below 128 tokens. In other words, the additional Prefill overhead has already been amortized at the shortest Decode length evaluated in our paper, allowing PACT-KV to achieve an end-to-end efficiency advantage. As the Decode length increases further, its lower TPOT continues to provide additional benefits. In summary, the end-to-end throughput experiments in the original paper have already demonstrated the practical efficiency of PACT-KV under both short and long Decode settings, while the above break-even analysis further quantifies the generation length required for the Decode benefits to offset the Prefill overhead. Together, these results demonstrate that the additional Prefill overhead of PACT-KV can be amortized within a relatively short generation process, with its Decode acceleration advantage becoming even more pronounced for longer generations. We will incorporate this break-even analysis into the revised manuscript to explicitly illustrate this efficiency trade-off.

Weakness 2: Limited theoretical justification for the proposed compression heuristics.

We address the three components using the original definitions and rationales, supplemented by additional theoretical analyses and ablations.

(1) Theoretical Justification for Token Merging

K-side Token Merging combines pre-RoPE Key clustering, recent-query attention-response similarity, and V-space Cosine Similarity to capture Key semantics. Figure 2(b) supports the criterion of recent-query attention-response similarity.

In the newly added Appendix E, we analyze Query Shift's impact on Logit Approximation Error and derive a KL Divergence upper bound between the original Attention Distribution PP and post-merging reconstructed Attention Distribution P^\hat P for queries in the decode stage:

DKL(P∣P^)≤[Δ(qr)+Δshift(qt,qr)]28.D_{\mathrm{KL}}(P|\hat P)\leq\frac{\left[\Delta(q_r)+\Delta_{\mathrm{shift}}(q_t,q_r)\right]^2}{8}.

Detailed derivations are provided in Appendix E.

We also conduct a decode-stage stability experiment. Query Shift, Query-induced Logit Drift Δshift(qt,qr)\Delta_{\mathrm{shift}}(q_t,q_r), and Logit Error Range Δ(qt)\Delta(q_t) initially increase and then stabilize, without continuously accumulating or growing with generation length. Experimental details are provided in Appendix E.4 and Figure 4.

For V-side Token Merging, Eq. 10 in the original paper bounds the merging error, showing that, given attention weights, V-side Fusion Error is constrained by Value representation differences within each merging group. This provides mathematical support for using Value Content Similarity as the merging criterion.

(2) Theoretical Justification for Importance-based Logit Compensation

As described in Section 3.2.2, PACT-KV uses the Integrated Importance Score ratio to approximate the relative contributions of an absorbed Token and its Anchor Token in the unnormalized attention-score space, converting this ratio into an additive Softmax Logit compensation through a logarithmic transformation.

This design follows the exponential transformation property of Softmax. For an absorbed Token jj and its Anchor Token ii, we introduce an additive bias βj\beta_j into the merged Token's Attention Logit. Since both share the same Key, their post-Softmax attention-weight ratio is: exp⁡(ℓi+βj)exp⁡(ℓi)=exp⁡(βj).\frac{\exp(\ell_i+\beta_j)}{\exp(\ell_i)}=\exp(\beta_j). Here, ℓi\ell_i denotes the Attention Logit between the current Query and the Key of Anchor Token ii.

Matching this ratio to the estimated importance ratio gives: exp⁡(βj)≈sjsi,\exp(\beta_j)\approx\frac{s_j}{s_i}, which yields: βj=log⁡sjsi\beta_j=\log\frac{s_j}{s_i}

Thus, sjsi\frac{s_j}{s_i} becomes additive compensation βj=log⁡sjsi\beta_j=\log\frac{s_j}{s_i} in the Softmax Logit space.

To further validate this design, we compare alternative compensation formulations:

β ConfigurationAverage Score across Four Tasks
Original: log⁡(sj/si)\log(s_j/s_i)28.66
1.5log⁡(sj/si)1.5\log(s_j/s_i)27.96
Key-based Cosine Similarity25.97
0.5log⁡(sj/si)0.5\log(s_j/s_i)25.37
β=0\beta=024.00
β=log⁡(1+sj/si)\beta=\log(1+s_j/s_i)22.90

Across PassageCount, NarrativeQA, LCC, and Multi-News, the original formulation achieves the highest average score of 28.66. All alternatives perform worse, validating Logit Compensation based on the Integrated Importance Ratio.

(3) Theoretical Justification for Channel Pruning

For K-side Channel Pruning, PACT-KV measures channel importance through weighted absolute interactions between recent Queries and retained Keys (Eq. 8 of original paper), rather than global activation magnitudes: EcK,g=∑i∈PK(g)si∣Ki,c∣∑t=L−R+1L∣Qt,c∣.E_{c}^{K,g}=\sum_{i\in P_K^{(g)}}s_i|K_{i,c}|\sum_{t=L-R+1}^{L}|Q_{t,c}|.

This metric jointly considers token importance, Key channel activations, and recent Query responses. Sharing a channel mask within each SimHash cluster exploits within-cluster consistency while preserving differences across clusters. For pruned K channels, Mean Compensation approximates their energy contributions instead of directly zeroing them.

For V-side Channel Pruning, since Value does not participate in Attention Logit computation, channel contributions depend on activation magnitudes. The importance of retained physical V tokens is also considered through globally weighted absolute magnitudes (Eq. 12 of original paper): EcV=∑i∈PVsi∣Vi,c∣.E_c^V=\sum_{i\in P_V}s_i|V_{i,c}|.

Weighted mean compensation further approximates the aggregated contributions of pruned Value channels.

To validate the compensation mechanisms, we add independent Mean Compensation ablations. Removing K-side compensation, V-side compensation, and both reduces the average scores on PassageCount, NarrativeQA, LCC, and Multi-News to 27.16, 26.99, and 26.87, respectively, compared with 28.66 for the full method. These results confirm the contributions of both mechanisms to information preservation.

Weakness 3: Insufficient analysis of token importance estimation overhead.

First, it is worth clarifying that cumulative statistics for Global Attention Importance constitute an auxiliary computational component of our overall method rather than its core innovation. Therefore, the original paper did not provide a detailed analysis of the overhead of Global Attention Importance estimation.

Following your suggestion, we analyze the overhead of Global Attention Importance estimation. These attention statistics are collected through an additional attention pass. Specifically, we perform an additional attention-score computation and statistical aggregation using the same set of RoPE-transformed Queries and Keys. After Softmax, we accumulate the attention weights along the Query dimension to obtain Global Attention Importance. This process does not require rerunning the model Forward pass or performing Q/K Projection again. The additional overhead mainly comes from recomputing QK scores, their normalization and statistical aggregation.

Regarding the compatibility between cumulative attention score statistics and FlashAttention, previous work, MiniKV [1], has proposed a corresponding two-pass tiled computation scheme, namely the two-pass Triton-based selective FlashAttention kernel. In the first pass, it adopts FlashAttention's online Softmax computation to calculate the Attention Output and save the intermediate Log-Sum-Exp (LSE). In the second pass, it recomputes the tiled QK scores, uses the saved LSE for normalization, and accumulates attention weights along the Key dimension, thereby obtaining cumulative attention scores without materializing the full attention matrix.

We adopt the two-pass tiled computation strategy proposed by MiniKV and apply it to the Global Attention Importance estimation process in PACT-KV, avoiding the direct materialization of the full L×LL\times L Attention Probability Matrix. This implementation performs cumulative statistics by retaining the LSE and intermediate tiled results. Its additional workspace consists of linear-scale statistical results and tiled computation buffers, without requiring storage of the full quadratic-scale attention matrix.

To evaluate the computational overhead of Global Attention Importance statistics, we compare the performance before and after optimization. The results under the configuration of an 8K Prompt and 128 Decode Tokens are shown below:

MetricBefore OptimizationAfter Two-Pass OptimizationLatency Reduction
TTFT6.396 s5.326 s16.73%
End-to-End Latency11.78 s9.836 s16.50%

Meanwhile, the latency of the Decode stage remains almost unchanged. These results verify that PACT-KV can be integrated with FlashAttention, achieving reductions in both TTFT and End-to-End Latency.

Regarding GPU memory usage, the original standalone statistics implementation requires generating intermediate tensors related to attention scores. After adopting the two-pass tiled computation, there is no longer a need to materialize the full L×LL\times L Attention Probability Matrix. Instead, only the LSE, cumulative statistics, and temporary buffers required for tiled computation are retained. Therefore, this approach avoids intermediate storage requirements of the same order as the full attention matrix. We will clarify the distinction between the original statistics implementation and the optimized tiled implementation in the revised manuscript, and provide additional details regarding the implementation and GPU memory overhead.

[1] Akshat Sharma, Hangliang Ding, Jianping Li, Neel Dani, Minjia Zhang. MiniKV: Pushing the Limits of 2-Bit KV Cache via Compression and System Co-Design for Efficient Long Context Inference. ACL (Findings) 2025: 18506-18523.

Reviewer_rAM6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PACT-KV, a training-free KV cache compression method. On the K side, it merges tokens with similar attention responses within SimHash buckets, uses a bias term β to adjust the weights of the merged tokens, and prunes channels per bucket. On the V side, it merges tokens independently based on content similarity. Experiments on LongBench and NIAH with three 7–8B models show that it achieves the highest average scores among the compared methods. The motivation is reasonable and the K/V-decoupled framework is clear. However, the core β term lacks justification, the ablations are insufficient, and the method is complex while its gains over strong baselines are limited.

Critical strengths

1. A valuable observation on token merging under RoPE.

The authors observe that, after positional encoding, semantically similar tokens may receive inconsistent attention from queries, so merging should consider attention responses rather than semantics alone. The empirical analysis in Fig. 2(b) supports this observation.

2. A clear K/V-decoupled compression framework.

Using logical tokens and two index maps, the K and V sides can be compressed independently under their own criteria without constraining each other. Decoding operates directly on the compressed states without reconstructing the full KV cache, and this framework could be reused by future work.

3. Strong long-range retrieval under tight budgets.

On NIAH with a budget of 128, PACT-KV achieves 80.34, clearly higher than SnapKV (67.07) and KeepKV (56.27). This suggests that under extreme compression, merging preserves more long-range key information than direct eviction.

Critical weaknesses

1. The β bias lacks justification and ablation.

β = log(s_j/s_i) assumes that the attention ratio stays fixed during decoding, but the paper gives no derivation or error analysis. Setting τw = 1.00, which essentially disables K-side merging, reduces the scores by only 0.28/0.64. Please compare alternative forms of β (e.g., β = 0, log(1 + s_j/s_i)) and ablate them to show that the current form is effective.

2. Key hyperparameters are unjustified and insufficiently ablated.

The choices of R = 24 and the 10%/20% partition ratios are neither explained nor covered by any sensitivity analysis. The existing analysis uses only one model and two evaluation tasks, and the default C = 7 is not optimal (C = 5 performs better). Mean compensation and star absorption are not ablated individually.

3. The method is complex, but the gains are limited and key comparisons are missing.

Despite its many components, PACT-KV improves over ThinK+SnapKV by only 0.33. It is also not compared with LeanK or AdaKV. Under extreme compression the information loss is large, so seemingly orthogonal methods may require co-design. Please provide results for PACT-KV combined with AdaKV to support the orthogonality claim.

4. Incompatible with FlashAttention, which slows down inference.

Eq. 2 requires the full attention probability matrix, so the method cannot use FlashAttention. TTFT at 4K is 3.12s, versus 1.94s for H2O. We suggest the authors also release a FlashAttention-compatible kernel for the algorithm.

Questions for authors

see weakness

10 comments

Thanks very much for your valuable and detailed comments.

Weakness 1: Theoretical justification and ablation of the β Bias; Performance at τw\tau_w = 1.00

(1) Results for τw\tau_w = 1.00

For the results at τw=1.00\tau_w = 1.00, we would like to clarify that Figure 3 reports results on only two tasks, Qasper and GovReport. It is primarily intended to illustrate performance trends under different τw\tau_w settings and reflect parameter sensitivity, rather than measure the overall contribution of K-side Token Merging. Although disabling K-side Merging (τw=1.00\tau_w=1.00) causes limited degradation on these tasks, Table 4 already evaluates Token Merging through a component-level ablation. Across four LongBench tasks (Multi-News, LCC, NarrativeQA, and Passage Count), removing Token Merging reduces the average score from 28.66 to 25.81, a decrease of 2.85 points. This further demonstrates the important role of Token Merging in overall performance. Hence, its contribution cannot be judged solely from Figure 3.

(2) β Bias

The purpose of introducing β is to preserve, as much as possible, the original relative importance difference between a merged Token and its Anchor after K-side Token Merging.

Specifically, the merged Token jj and Anchor Token ii share the same physical Key after compression. Without compensation, their Attention Logits would be identical, and consequently, they would receive identical attention weights after Softmax, resulting in the loss of their original importance difference.

To address this issue, we introduce an additive bias βj\beta_j into the Attention Logit of the merged Token. Since both tokens use the same Key, their attention-weight ratio after compensation is: a^ja^i=exp⁡(ℓi+βj)exp⁡(ℓi)=exp⁡(βj).\frac{\hat a_j}{\hat a_i}=\frac{\exp(\ell_i+\beta_j)}{\exp(\ell_i)}=\exp(\beta_j). where a^j\hat a_j and a^i\hat a_i denote the attention weights obtained by the merged Token jj and Anchor Token ii, respectively, after Softmax with the Bias applied in the compressed representation. ℓi\ell_i denotes the Attention Logit computed between the current Query and the Key corresponding to Anchor Token ii.

Therefore, to make the ratio of the compensated attention weights consistent with the estimated importance ratio, we set: exp⁡(βj)≈sjsi,\exp(\beta_j)\approx\frac{s_j}{s_i}, which gives: βj=log⁡sjsi\boxed{\beta_j=\log\frac{s_j}{s_i}} This design is consistent with Star Absorption: Anchors are selected in descending order of integrated importance, so an absorbed Token satisfies sj≤sis_j\leq s_i. Setting β=log⁡(sj/si)\beta=\log(s_j/s_i) ensures that the compensated attention-weight ratio is no greater than 1, meaning that the merged Token's attention weight does not exceed the Anchor's, consistent with their importance ranking. In contrast, β=0\beta=0 yields a ratio of 1 and fails to preserve their original importance difference, while β=log⁡(1+sj/si)\beta=\log(1+s_j/s_i) yields a ratio greater than 1, causing the merged Token to receive a higher attention weight than the Anchor, contrary to their importance ranking.

As stated in Section 3.2.2 of our paper:

We use the ratio between integrated importance scores to approximate their relative contributions in the unnormalized attention-score space, and directly transform it into an additive logarithmic bias in the Softmax logit space.

(3) Ablation Study

To further verify the effects of the compensation coefficient and importance measurement on downstream performance, we conduct β ablation on PassageCount, NarrativeQA, LCC, and Multi-News, and calculate the average score. Specifically, we conduct comparisons from three aspects.

1) Compensation Coefficient Sensitivity

To verify the choice of compensation coefficient, we multiply the original β by 0.5 and 1.5, obtaining average scores of 25.37 and 27.96, respectively, both lower than the original score of 28.66.

2) Importance Measurement

To determine whether the integrated importance measure based on Q-K attention information is superior to one relying solely on Key similarity, we evaluate an alternative compensation formulation based on cosine similarity: βj=CosSim⁡(ki,kj)−1\boxed{\beta_j=\operatorname{CosSim}(k_i,k_j)-1} This formulation achieves an average score of 25.97, below the original configuration, indicating that compensation based on the integrated attention importance ratio sjsi\frac{s_j}{s_i} is more effective in preserving the relative contributions of different Tokens.

3) Alternative Compensation Formulations

Following your suggested alternatives, we set β=0\beta=0 and β=log⁡(1+sj/si)\beta=\log(1+s_j/s_i), obtaining average scores of 24.00 and 22.90, respectively, both substantially lower than the original score of 28.66.

β ConfigurationAverage Score across Four Tasks
Original: log⁡(sj/si)\log(s_j/s_i)28.66
1.5log⁡(sj/si)1.5\log(s_j/s_i)27.96
Cosine Similarity25.97
0.5log⁡(sj/si)0.5\log(s_j/s_i)25.37
β=0\beta=024.00
β=log⁡(1+sj/si)\beta=\log(1+s_j/s_i)22.90

These results validate the effectiveness of logarithmic compensation based on the integrated importance ratio in our method.

Weakness 2: Key hyperparameters are unjustified and insufficiently ablated.

(1) Selection of the Recent Query Window RR

We evaluate five configurations, R=8,16,24,32,64R=8,16,24,32,64, on Qasper and GovReport. Their corresponding average scores are 31.81, 31.72, 31.91, 31.21, and 30.31, respectively. The results show that R=24R=24 achieves the highest average score among the tested configurations, while R=8R=8 and R=16R=16 also exhibit comparable performance. As the window size increases further, model performance declines. Therefore, R=24R=24 is a reasonable choice that balances local Query information and downstream task performance.

(2) Sensitivity Analysis of Token Partition Ratios

First, with the most important class ratio fixed at 10%, we evaluate the performance with low-importance class ratios of 0%, 10%, and 30%. The corresponding average scores on Qasper and GovReport are 31.78, 31.42, and 31.72, respectively. The default configuration (most important class ratio = 10%, low-importance class ratio = 20%) achieves an average score of 31.91.

Second, with the low-importance class ratio fixed at 20%, we evaluate the performance with most important class ratios of 0% and 20%, obtaining average scores of 31.82 and 31.89, respectively.

The results show that PACT-KV maintains relatively stable performance within the tested parameter ranges, with relatively small score differences across different Partition Ratios. The default configuration achieves an average score of 31.91, ranking among the better-performing configurations. This indicates that our Token Partition strategy exhibits a degree of robustness to parameter variations.

(3) Selection of the Number of SimHash Bits CC

In the original experiments, C=5C=5 performs slightly better than C=7C=7 on Qasper and GovReport. This is because different tasks may have different sensitivities to the granularity of Token similarity partitioning, as particularly reflected in the performance with C=5C=5. Specifically, increasing CC by 1 doubles the number of groups, thereby changing the Token grouping granularity and the number of candidate tokens within each group. Consequently, different tasks may exhibit different performance trends. Therefore, following your suggestion, we conducted additional experiments.

To this end, we further conduct a unified evaluation of C=3,5,7,9C=3,5,7,9 on four evaluation tasks: Multi-News, LCC, NarrativeQA, and PassageCount. The corresponding average scores are 25.72, 27.49, 28.66, and 28.32, respectively. Among these configurations, C=7C=7 achieves the highest average score, outperforming C=5C=5 and C=9C=9 by 1.17 and 0.34 points, respectively.

These results indicate that although C=5C=5 has certain advantages on individual tasks, C=7C=7 achieves better overall performance under a more comprehensive evaluation setting. Therefore, we retain C=7C=7 as the default configuration.

(4) Independent Ablation of Mean Compensation

In the original submission, we only conducted an ablation study on the whole K/V Channel Pruning module to examine the overall contribution of channel pruning to model performance, without separately investigating the specific effects of K-side and V-side Mean Compensation. Following your suggestion, we added three independent experiments: removing only K-side Mean Compensation, removing only V-side Mean Compensation, and removing Mean Compensation on both sides.

On the four evaluation tasks such as Multi-News, LCC, NarrativeQA, and PassageCount, these three configurations achieve average scores of 27.16, 26.99, and 26.87, respectively. All are lower than PACT-KV's score of 28.66, with decreases of 1.50, 1.67, and 1.79 points, respectively.

These results indicate that both K-side and V-side Mean Compensation help preserve information after channel pruning, and together they constitute the channel compensation mechanism of PACT-KV.

(5) Independent Ablation of Star Absorption

In the original submission, Star Absorption, as the Anchor assignment strategy for Token Merging, was evaluated together with the overall merging pipeline, without separately isolating its contribution. Following your suggestion, we further conduct an independent ablation study of this strategy.

As described in Section 3.2.2 of our paper, Star Absorption selects Anchors according to integrated importance, directly maps each absorbed Token to its corresponding Anchor, and forms a star-shaped merging structure, thereby avoiding potential error propagation caused by chained merging. With all other configurations unchanged, removing Star Absorption reduces the average score across the four evaluation tasks from 28.66 to 28.19, a decrease of 0.47 points.

This result verifies the positive contribution of Star Absorption to overall compression quality and further supports our design choice of direct Anchor mapping rather than chained merging.

Weakness 3: The gains are limited and key comparisons are missing.

(1) Performance Gain

Although PACT-KV improves the average score over ThinK+SnapKV by 0.33 points, this difference should be interpreted in the context of the performance of Full KV. Full KV achieves an average score of 42.93, while PACT-KV and ThinK+SnapKV achieve 41.23 and 40.90, respectively. Accordingly, their performance gaps relative to Full KV are 1.70 and 2.03 points, respectively.

This means that PACT-KV not only achieves better task performance after compression but also reduces the remaining performance gap between ThinK+SnapKV and Full KV by approximately 16.3% (0.33/2.030.33/2.03). Considering that both methods have already achieved performance close to Full KV, the remaining room for improvement is inherently limited. Therefore, the 0.33-point improvement actually represents a further meaningful reduction in the remaining performance gap.

Consequently, evaluating the improvement solely based on the absolute score difference between the two compression methods does not fully capture the value of PACT-KV in further approaching Full KV performance under a high compression ratio.

(2) Combination Experiments of PACT-KV and AdaKV

To verify the orthogonality of PACT-KV and AdaKV, we further conduct combination experiments, evaluating the default AdaKV implementation (Ada-SnapKV), PACT-KV alone, and the performance of incorporating PACT-KV into AdaKV.

MethodNQAQaspMFHQA2WikiMusqGovRQMMNTRECTrivSAMPCntPReLCCRB-PAvg.
PACT-KV23.0138.9041.0145.9835.8321.5724.9222.3326.1674.0089.9740.577.2560.5058.2249.4541.23
AdaKV21.1536.0542.9445.3834.4521.3021.0821.5223.1872.5090.1338.944.5063.0060.2257.7240.88
PACT-KV + AdaKV22.2238.7542.2745.9235.1721.5225.1122.0326.0776.0089.9940.217.5062.0057.3949.5941.36

We conduct experiments on the same 16 LongBench evaluation tasks, and the results are presented in the table above. The default AdaKV implementation achieves an average score of 40.88, while PACT-KV alone achieves 41.23. Combining PACT-KV with AdaKV further improves the average score to 41.36.

Specifically, the combined method improves the average score by 0.48 points over the default AdaKV implementation and by a further 0.13 points over PACT-KV alone.

These results demonstrate that PACT-KV and AdaKV can work effectively together. Incorporating PACT-KV into AdaKV not only preserves the performance advantages of the original method but also brings additional performance gains. This further verifies the compatibility and complementarity between PACT-KV and AdaKV, supporting our claim regarding the orthogonality of the proposed method.

Since LeanK differs substantially from PACT-KV in its training paradigm, LeanK requires two-stage optimization to learn static channel masks, whereas PACT-KV is entirely training-free and does not require pre-learned channel masks. Therefore, we primarily select methods that require no additional training and can be directly applied to pretrained models, such as SPARK and ThinK, as experimental baselines. Incorporating LeanK into the same experimental framework would require additional reproduction of its two-stage mask training and inference implementation, which differs from the training-free evaluation setting adopted in this paper.

Furthermore, LeanK has not investigated its combination with Token-level Eviction methods such as SnapKV and H2O, or with Token Merging. Combining LeanK with token-level methods also involves implementation challenges. Among related studies, SPARK likewise primarily selects ThinK as its direct comparison method for channel pruning, while treating SnapKV and other methods as Token-level Eviction baselines and combination targets. This also reflects the experimental practice of selecting representative methods for comparison according to their specific compression mechanisms. Therefore, in this context, we primarily conduct combination experiments with AdaKV, which is compatible with our training-free setting.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.