- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
PACT-KV combines position-aware token merging with channel pruning and independent key/value compression. Pre-RoPE SimHash groups constrain key merging, recent-query attention responses estimate positional compatibility, and cluster-specific channel masks compress keys. Values use separate merging and fallback channel pruning. LongBench, retrieval, and runtime experiments evaluate quality under equivalent cache budgets.
Critical strengths
- The asymmetric K/V design is technically sensible: it distinguishes query matching from value aggregation and connects token grouping to a structured channel mask instead of simply stacking two compression methods.
- Equivalent-cache-budget comparisons across three models and combined token/channel baselines provide useful support for the quality benefit. The measured runtime also makes the prefill cost visible rather than reporting decode speed alone.
Critical weaknesses
- The core positional proxy is based on a short recent-query window, while the merge bias is estimated from aggregated importance ratios. Figure 2 provides motivating correlation, but it does not establish that these approximations remain faithful when future queries change substantially; that limitation directly affects the reliability of irreversible prompt compression.
- The efficiency table includes H2O, but omits SnapKV and the token/channel eviction combinations used in the quality comparison. PACT-KV also has higher TTFT than the listed methods. Thus, the reported throughput advantage does not yet establish a general quality–latency advantage over those additional alternatives.
Questions for authors
- How large is attention/output error for query shifts outside the probing window, and how sensitive are downstream results to the importance-ratio bias?
- What is the quality–TTFT–throughput comparison against SnapKV and combined channel-pruning eviction baselines under the same measured memory budget?
2 comments
Weakness2 & Question2
Rationale for Selecting the Original Experimental Baselines
First, the original efficiency evaluation focused on Token Merging methods and selected H2O as a representative baseline because its attention-importance estimation mechanism is similar to ours. We acknowledge that SnapKV is a Token-level Eviction method with low overhead. It selectively retains Tokens, incurring less Prefill overhead than Token Merging methods. Therefore, we did not select SnapKV as a primary efficiency baseline.
Second, for combining Channel Pruning and Token Eviction, the original paper evaluates the practical inference efficiency of the overall PACT-KV framework. Comparisons with representative Token-level compression methods had demonstrated PACT-KV's practical feasibility. Considering that channel-importance estimation and pruning typically introduce additional computational overhead, we did not exhaustively evaluate all Channel Pruning and Token Eviction combinations.
Furthermore, we inspected SPARK's publicly available implementation and found that its Channel Pruning modifies K/V elements through masking or restoration but still writes dense tensors of the original size back to the cache. Pruned channels are not physically removed at the storage level, and conventional dense Attention computation still processes the full channel dimension. Therefore, the nominal pruning ratio of this implementation does not directly translate into reductions in actual KV Cache capacity or Attention computation, making it difficult to use as a runtime-efficiency comparison under the same physical cache budget.
Additional Experimental Results (As Suggested)
Following your suggestion, we add SnapKV, ThinK+H2O, and ThinK+SnapKV to the runtime-efficiency evaluation. All methods are tested using LLaMA-3-8B-Instruct on a single RTX 3090 GPU with a KV Cache Budget of 256. We test 4K/8K Prompts with 128/1K Decode Tokens, reporting TTFT, TPOT, end-to-end throughput, and actual GPU memory consumption.
First, we acknowledge SnapKV's clear Prefill advantage: it achieves lower TTFT than PACT-KV in all four configurations and higher end-to-end throughput in shorter Decode scenarios. This demonstrates the efficiency advantage of lightweight Token Eviction methods for short-generation scenarios. On the other hand, PACT-KV achieves the lowest TPOT in all four configurations and higher end-to-end throughput than H2O, ThinK+H2O, and ThinK+SnapKV. At 1K Decode Tokens, PACT-KV also achieves higher end-to-end throughput than all tested methods, including SnapKV.
Furthermore, the additional experiments show that simply combining Token-level Eviction and Channel Pruning does not necessarily lead to better inference efficiency. Both ThinK+H2O and ThinK+SnapKV achieve lower end-to-end throughput than PACT-KV in our experiments. In contrast, PACT-KV adopts a cooperative design tailored to the distinct K/V representation spaces rather than simply applying two compression operations sequentially. It achieves better compression quality, lower Decode latency, and higher end-to-end throughput in longer-generation scenarios, highlighting the need for co-design across compression dimensions.
The runtime-efficiency comparison of different KV Cache compression methods is presented below.
| Prompt | Decode | Method | TTFT (s) | TPOT (ms) | E2E (tok/s) | Memory (GiB) |
|---|---|---|---|---|---|---|
| 4K | 128 | SnapKV | 1.351 | 39.24 | 20.12 | 16.54 |
| 4K | 128 | ThinK+H2O | 2.294 | 52.62 | 14.16 | 15.78 |
| 4K | 128 | ThinK+SnapKV | 2.036 | 45.20 | 16.31 | 15.93 |
| 4K | 128 | PACT-KV | 3.124 | 32.24 | 17.09 | 15.93 |
| 4K | 1024 | SnapKV | 1.342 | 39.88 | 24.03 | 16.54 |
| 4K | 1024 | ThinK+H2O | 2.326 | 54.64 | 17.33 | 15.78 |
| 4K | 1024 | ThinK+SnapKV | 2.074 | 45.19 | 20.82 | 16.63 |
| 4K | 1024 | PACT-KV | 3.174 | 34.99 | 25.77 | 16.04 |
| 8K | 128 | SnapKV | 3.282 | 42.80 | 14.64 | 19.72 |
| 8K | 128 | ThinK+H2O | 5.814 | 53.05 | 10.15 | 16.33 |
| 8K | 128 | ThinK+SnapKV | 5.067 | 48.18 | 11.37 | 16.42 |
| 8K | 128 | PACT-KV | 6.396 | 32.98 | 11.78 | 16.39 |
| 8K | 1024 | SnapKV | 3.225 | 40.97 | 22.45 | 19.76 |
| 8K | 1024 | ThinK+H2O | 5.827 | 55.16 | 16.22 | 16.34 |
| 8K | 1024 | ThinK+SnapKV | 5.090 | 46.40 | 19.17 | 16.68 |
| 8K | 1024 | PACT-KV | 6.392 | 35.54 | 23.53 | 16.50 |
Thank you very much for your valuable and positive comments.
Weakness1 & Question1
Impact of Query Shift and Error Stability
First, we would like to clarify the scope of the problem studied in this paper. The core objective of KV Cache compression is to preserve as much information as possible that is most valuable for subsequent generation under a limited storage budget, thereby achieving a balance between the compression ratio and model performance. However, during the Prefill stage, future Queries in the Decode stage are not yet available. Therefore, existing KV Cache compression methods can typically only evaluate Token importance based on the information available in the Prompt, retaining important information while compressing or evicting less important information.
Accordingly, we investigate a more specific practical question: During natural autoregressive Decode following a single Prompt, does the shift of subsequent Queries relative to the Prefill reference Query grow with generation length, causing approximation errors to accumulate and significantly affect model performance?
To address this question, we analyze the relationship between Query Shift and Logit Approximation Error, and derive an upper bound on the Attention Distribution Error between the original Decode process and the reconstruction after token merging. Detailed derivations are provided in the newly added Appendix E. The three results are:
\leq \frac{\|q_t-q_r\|_2\|d_j\|_2}{\sqrt D}$$ $$\Delta(q_t) \leq \Delta(q_r)+\Delta_{\mathrm{shift}}(q_t,q_r) \leq \Delta(q_r)+\frac{\lVert q_t-q_r\rVert_2}{\sqrt{D}} \max_{j,k}\lVert d_j-d_k\rVert_2$$ $$D_{\mathrm{KL}}(P\|\hat P) \leq \frac{1}{8} \left[ \Delta(q_r) + \Delta_{\mathrm{shift}}(q_t,q_r) \right]^2$$ On GovReport (512 Tokens) and LongCtx30 (1,500 Tokens), we track Query Shift, Query-induced Logit Drift, and Logit Error Range during long Decode. Figure 4 (Appendix E) shows that all three metrics rise within the first few tokens, then stabilize with local fluctuations and no sustained growth. Together, the analysis and experiments show that, under the evaluated natural autoregressive settings, Query Shift-induced Logit Approximation Error does not continuously accumulate with generation length. This supports determining a fixed Token Merging mapping during Prefill and retaining it throughout subsequent Decode in PACT-KV. ## Downstream Performance Sensitivity of Importance-ratio Bias Following your suggestion, we evaluate Bias selection on PassageCount, NarrativeQA, LCC, and Multi-News, reporting the average score across the four tasks. We examine three aspects: **(1) Compensation coefficient sensitivity.** Scaling the original $\beta$ by 0.5 and 1.5 yields average scores of 25.37 and 27.96, respectively, both below the original score of 28.66. The coefficient therefore affects downstream performance, with the original setting performing best among the tested configurations. **(2) Importance measurement.** To compare integrated attention importance with Key-only similarity, we test a cosine-similarity-based alternative: $$\boxed{\beta_j=\operatorname{CosSim}(k_i,k_j)-1}$$ Its compensated attention-weight ratio is: $$\frac{\hat a_j}{\hat a_i}=\exp\left(\operatorname{CosSim}(k_i,k_j)-1\right)$$ where $\hat a_j$ and $\hat a_i$ are the attention weights of the merged Token and Anchor Token, respectively. Key cosine similarity estimates their relative contribution: higher similarity gives a ratio closer to 1, while lower similarity gives a lower relative weight. This variant scores 25.97, below the original configuration. This indicates that Key similarity alone does not fully capture relative attention contributions, and that integrated importance ratios are more suitable for our Token Merging mechanism. **(3) Alternative compensation forms.** We further evaluate two alternative forms: $\beta=0$ and $\beta=\log(1+s_j/s_i)$, which achieve average scores of 24.00 and 22.90, respectively, both substantially lower than the original configuration. The results are summarized below: | Bias Configuration | Average Score Across Four Tasks | | :------------------------ | ------------------------------: | | Original: $\log(s_j/s_i)$ | **28.66** | | $1.5\log(s_j/s_i)$ | 27.96 | | Key-based Cosine Similarity| 25.97| | $0.5\log(s_j/s_i)$ | 25.37 | | $\beta=0$ | 24.00 | | $\beta=\log(1+s_j/s_i)$ | 22.90 | The original Importance-ratio Bias achieves the highest average score (28.66), exceeding $\beta=0$ by 4.66 points and $\beta=\log(1+s_j/s_i)$ by 5.76 points. These results further confirm the importance of preserving the original relative importance differences between tokens.