- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Proactive Staged Compression (PSC), a scheduling framework for efficient KV cache compression. Unlike conventional capacity-triggered compression (CTC), which performs large-scale compression when the KV cache reaches its memory limit, PSC distributes compression across multiple earlier stages with smaller removal ratios.
The motivation is that multiple small compression operations may reduce attention perturbation compared with a single aggressive compression. PSC is compressor-agnostic and can be integrated with existing eviction- and merging-based methods without modifying their underlying compression mechanisms.
Experiments across multiple models, benchmarks, and KV cache compression methods demonstrate performance improvements in most evaluated settings, particularly under tight memory budgets.
Critical strengths
-
Simple and intuitive motivation. The paper explores KV cache compression from the perspective of scheduling rather than designing another token selection or merging algorithm. The core idea is straightforward and easy to understand.
-
Good compatibility with existing methods. PSC does not require modifying the underlying compression algorithms and can be integrated with various eviction- and merging-based methods. This makes the approach flexible and potentially useful in practice.
-
Promising experimental results. The paper evaluates PSC across multiple models, compression methods, and datasets. Most configurations show improvements over conventional compression schedules, particularly under aggressive compression budgets. The attention perturbation experiments also provide supporting evidence for the proposed approach.
Critical weaknesses
-
Potential risks of irreversible early eviction. Multi-stage compression involves a sequence of greedy and irreversible decisions. Tokens considered unimportant at an early decoding step may become important later, but once evicted, they cannot be recovered. In contrast, delayed compression can make its selection decisions using more recent information while retaining the original tokens for longer. Although the paper provides experimental evidence that staged compression reduces attention perturbation, it does not fully establish when proactive compression is preferable to delayed selection.
-
Insufficient analysis of the source of improvement. PSC changes both the number of compression operations and their timing. It remains unclear whether the observed improvements mainly result from dividing one large compression into several smaller operations or from performing compression proactively. A more controlled comparison would help isolate these two factors and better justify the proposed scheduling strategy.
-
Potential inconsistency in the efficiency evaluation. According to the scheduling equations and the capacity-triggered fallback in Algorithm 1, the configurations used in Table 2 appear insufficient to trigger additional proactive compression events. In particular, with a prefill length of 4096, a cache capacity of 4200, a decoding horizon of 512, and the evaluated retention ratios and stage counts, Phase 1 appears infeasible, while the first Phase 2 staged compression would occur beyond the remaining decoding horizon. However, Table 2 reports noticeable performance differences between CTC and PSC. This apparent inconsistency requires clarification to establish whether the reported efficiency differences arise from the proposed scheduling mechanism.
Questions for authors
-
How does PSC address the risk of prematurely evicting tokens that may become important later in generation? Under what conditions would delayed compression be preferable to proactive compression?
-
Could the authors provide an ablation study that separates the effects of multi-stage compression and proactive scheduling? For example, comparing PSC with a strategy that performs multiple smaller compressions closer to the capacity limit would help clarify the source of the improvements.
-
Could the authors clarify the actual compression events triggered in the Table 2 efficiency experiments? Based on the stated configurations and Algorithm 1, it appears that no additional proactive compression events would occur within the decoding horizon. If this interpretation is incorrect, please provide the corresponding event schedules or clarify the implementation details.
4 comments
Dear Reviewer QeHW, thank you for your thoughtful suggestions. Following your feedback, we have updated the PDF with the scheduling implementation in Appendix B and the experimental parameters and cache statistics in Appendix C. Our discussion responses also present the additional scheduling comparisons. We would appreciate your assessment of these revisions when convenient. More comprehensive experiments will be incorporated into the main text and appendix as the evaluation is completed.
Thank you for the detailed response and the additional experiments.
I appreciate the clarification of the compression scheduling implementation, which addresses my concern regarding the compression events reported in Table 2.
However, I still have reservations about the core motivation of proactive compression. While both CTC and PSC involve irreversible eviction decisions, PSC removes tokens earlier, potentially losing information that may become important during subsequent generation.
The newly added delayed staged compression experiment is particularly interesting. Although the delayed strategy retains more tokens on average and therefore is not a strictly memory-matched comparison, its lower attention-output deviation suggests that earlier compression is not necessarily preferable. Moreover, the improvement of PSC over CTC in this experiment is relatively modest, and the evaluation includes only three examples.
Overall, while I appreciate the authors' additional efforts, I do not think the rebuttal fully resolves my main concerns regarding the advantages and generality of proactive compression.
I will therefore maintain my original rating of 2/4 (Weak Reject).
Thank you for your follow-up. We agree that early eviction can potentially remove information that becomes important later. However, earlier eviction does not inherently imply a higher risk of such information loss. Figures 7–9 in the appendix provide empirical evidence relevant to this concern: if an evicted token becomes important during subsequent generation, its absence can manifest as increased attention-output deviation from the uncompressed model. In these experiments, PSC generally exhibits smaller and smoother attention-output perturbations than CTC during subsequent decoding. Thus, in the evaluated settings, proactive staged compression does not translate into greater overall attention-output deviation.
This observation captures our core motivation: PSC balances the potential risk of earlier information removal against the disruption caused by a single large compression event. Our claim is that distributing a large compression operation across multiple decoding steps can better preserve the attention outputs of the uncompressed model, as demonstrated in our experiments.
Lower attention-output deviation alone does not guarantee better generation quality. For precisely this reason, our evidence goes beyond attention diagnostics: we conduct extensive task-level evaluations across multiple models, datasets, and underlying compression methods. PSC outperforms CTC in the majority of evaluated configurations. These task-level results substantiate its practical effectiveness, and the exceptions do not negate the broader empirical pattern. We therefore believe that PSC’s advantages should be assessed on the combined evidence from the attention analyses and the extensive task-level evaluations, rather than the three-example diagnostic alone.
We thank you again for the constructive discussion, which has helped us clarify the motivation and empirical evidence for PSC.
Early Eviction Risks and Controlled Scheduling Comparisons
Thank you for pointing out the risk of deleting tokens too early and the need to separate the effects of smaller deletions from earlier deletions.
W1 and Q1: Early eviction risks and delayed compression
We conducted extensive attention-perturbation experiments comparing attention outputs before and after compression. These experiments provide empirical evidence that PSC can perform earlier deletions while preserving attention outputs at least as well as CTC in the evaluated settings. Although it is difficult to prove that a deleted token will never become important later, this uncertainty applies to every eviction decision: any token removed by either CTC or PSC may become relevant during subsequent generation. Future token importance is therefore a shared challenge for both methods.
W2 and Q2: Comparing early and delayed small deletions
We added the requested comparison with a delayed version of staged compression. PSC starts its small deletions before the cache is full; the delayed version waits until the cache is full before each small deletion. Both use the same deletion amounts and number of events. At the end of each compared phase, they have removed the same total number of entries and keep the same number in the cache.
We tested three GovReport examples using DeepSeek-R1-Distill-Qwen-1.5B, one NVIDIA A100 80GB, H2O, cache capacity 512, R=0.20, and k=4. Each method processes the same 1,792 reference tokens generated by FullKV. We measure how much its attention output differs from FullKV; the percentage is the relative L2 difference, with lower values indicating attention outputs closer to FullKV.
PSC produces a smaller difference than CTC in this test. Delaying the small deletions reduces the difference further, while keeping more tokens during generation. This directly shows why earlier deletion is not always preferable: retaining information longer can improve later compression decisions.For this comparison, we process all 1,792 reference tokens even if EOS appears earlier. The table therefore quantifies the change in attention output while processing these reference tokens.
W3 and Q3: Compression schedule and event counts
The compression events follow from the clipping rule used in the implementation. The generic implementation first specifies k, computes r1,r2 using Equations (9) and (13), clips r1 to 0.95 when necessary, and recomputes the interval. It then performs k−1 small compressions followed by bulk compression before entering Phase 2.
With scheduler input Lp=4096, capacity Lm=4200, R=0.20, and k=4, small compressions are scheduled at steps 90, 180, and 270, followed by bulk compression at step 360. Thus, several proactive events can occur within 512 steps. The Phase-2 interval is 1,680, so no subsequent Phase-2 event occurs during this run.
These steps are calculated for Lp=4096; actual scheduling also depends on the appended instruction tokens. The calculated schedule explains the occurrence of proactive events; complete input lengths and capacity handling are addressed in the general clarification. The short Table 2 generation window does not establish the behavior of repeated Phase-2 compression.