acceptodds
Under review as a conference paper at ICLR 2027

Multiple Stages, More Gains: Proactive Staged Compression for Efficient LLM Inference

Abstract

The ever-growing Key-Value (KV) Cache generated during decoding becomes a bottleneck for large language models (LLMs) in long-context processing. Recent research has mainly focused on compressing the KV Cache through eviction and merging strategies, thereby enhancing long-context inference capability. However, most existing methods rely on aggressive compression at the memory limit, resulting in cumulative attention perturbation and model performance degradation. In this paper, we propose **PSC** (**P**roactive **S**taged **C**ompression), a compressor-agnostic scheduling framework that distributes KV Cache compression across multiple stages before the cache reaches its capacity limit. Our key insight is that decomposing compression into multiple stages with small removal ratios significantly reduces attention perturbation. From the cache capacity, prompt length and target retention ratio, we analytically derive the number of compression stages and the retained KV Cache size at each stage to obtain a closed-form two-phase compression schedule. Experimental results across various models and datasets show that incorporating PSC significantly enhances the performance of existing representative compression methods, achieving consistent accuracy improvement and accelerating inference by up to 1.17 compared to representative baselines.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Large Language Models, KV Cache, KV Cache Compression, Inference Efficiency

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_QeHWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Proactive Staged Compression (PSC), a scheduling framework for efficient KV cache compression. Unlike conventional capacity-triggered compression (CTC), which performs large-scale compression when the KV cache reaches its memory limit, PSC distributes compression across multiple earlier stages with smaller removal ratios.

The motivation is that multiple small compression operations may reduce attention perturbation compared with a single aggressive compression. PSC is compressor-agnostic and can be integrated with existing eviction- and merging-based methods without modifying their underlying compression mechanisms.

Experiments across multiple models, benchmarks, and KV cache compression methods demonstrate performance improvements in most evaluated settings, particularly under tight memory budgets.

Critical strengths
  1. Simple and intuitive motivation. The paper explores KV cache compression from the perspective of scheduling rather than designing another token selection or merging algorithm. The core idea is straightforward and easy to understand.

  2. Good compatibility with existing methods. PSC does not require modifying the underlying compression algorithms and can be integrated with various eviction- and merging-based methods. This makes the approach flexible and potentially useful in practice.

  3. Promising experimental results. The paper evaluates PSC across multiple models, compression methods, and datasets. Most configurations show improvements over conventional compression schedules, particularly under aggressive compression budgets. The attention perturbation experiments also provide supporting evidence for the proposed approach.

Critical weaknesses
  1. Potential risks of irreversible early eviction. Multi-stage compression involves a sequence of greedy and irreversible decisions. Tokens considered unimportant at an early decoding step may become important later, but once evicted, they cannot be recovered. In contrast, delayed compression can make its selection decisions using more recent information while retaining the original tokens for longer. Although the paper provides experimental evidence that staged compression reduces attention perturbation, it does not fully establish when proactive compression is preferable to delayed selection.

  2. Insufficient analysis of the source of improvement. PSC changes both the number of compression operations and their timing. It remains unclear whether the observed improvements mainly result from dividing one large compression into several smaller operations or from performing compression proactively. A more controlled comparison would help isolate these two factors and better justify the proposed scheduling strategy.

  3. Potential inconsistency in the efficiency evaluation. According to the scheduling equations and the capacity-triggered fallback in Algorithm 1, the configurations used in Table 2 appear insufficient to trigger additional proactive compression events. In particular, with a prefill length of 4096, a cache capacity of 4200, a decoding horizon of 512, and the evaluated retention ratios and stage counts, Phase 1 appears infeasible, while the first Phase 2 staged compression would occur beyond the remaining decoding horizon. However, Table 2 reports noticeable performance differences between CTC and PSC. This apparent inconsistency requires clarification to establish whether the reported efficiency differences arise from the proposed scheduling mechanism.

Questions for authors
  1. How does PSC address the risk of prematurely evicting tokens that may become important later in generation? Under what conditions would delayed compression be preferable to proactive compression?

  2. Could the authors provide an ablation study that separates the effects of multi-stage compression and proactive scheduling? For example, comparing PSC with a strategy that performs multiple smaller compressions closer to the capacity limit would help clarify the source of the improvements.

  3. Could the authors clarify the actual compression events triggered in the Table 2 efficiency experiments? Based on the stated configurations and Algorithm 1, it appears that no additional proactive compression events would occur within the decoding horizon. If this interpretation is incorrect, please provide the corresponding event schedules or clarify the implementation details.

4 comments
AuthorsOpenReview ↗

Early Eviction Risks and Controlled Scheduling Comparisons

Thank you for pointing out the risk of deleting tokens too early and the need to separate the effects of smaller deletions from earlier deletions.

W1 and Q1: Early eviction risks and delayed compression

We conducted extensive attention-perturbation experiments comparing attention outputs before and after compression. These experiments provide empirical evidence that PSC can perform earlier deletions while preserving attention outputs at least as well as CTC in the evaluated settings. Although it is difficult to prove that a deleted token will never become important later, this uncertainty applies to every eviction decision: any token removed by either CTC or PSC may become relevant during subsequent generation. Future token importance is therefore a shared challenge for both methods.

W2 and Q2: Comparing early and delayed small deletions

We added the requested comparison with a delayed version of staged compression. PSC starts its small deletions before the cache is full; the delayed version waits until the cache is full before each small deletion. Both use the same deletion amounts and number of events. At the end of each compared phase, they have removed the same total number of entries and keep the same number in the cache.

We tested three GovReport examples using DeepSeek-R1-Distill-Qwen-1.5B, one NVIDIA A100 80GB, H2O, cache capacity 512, R=0.20R=0.20, and k=4k=4. Each method processes the same 1,792 reference tokens generated by FullKV. We measure how much its attention output differs from FullKV; the percentage is the relative L2 difference, with lower values indicating attention outputs closer to FullKV.

ScheduleAttention-output difference from FullKV ↓Average tokens kept in cache
CTC53.43%298.95
PSC51.49%303.53
Delayed staged compression44.47%361.53

PSC produces a smaller difference than CTC in this test. Delaying the small deletions reduces the difference further, while keeping more tokens during generation. This directly shows why earlier deletion is not always preferable: retaining information longer can improve later compression decisions.For this comparison, we process all 1,792 reference tokens even if EOS appears earlier. The table therefore quantifies the change in attention output while processing these reference tokens.

W3 and Q3: Compression schedule and event counts

The compression events follow from the clipping rule used in the implementation. The generic implementation first specifies kk, computes r1,r2r_1,r_2 using Equations (9) and (13), clips r1r_1 to 0.950.95 when necessary, and recomputes the interval. It then performs k−1k-1 small compressions followed by bulk compression before entering Phase 2.

With scheduler input Lp=4096L_p=4096, capacity Lm=4200L_m=4200, R=0.20R=0.20, and k=4k=4, small compressions are scheduled at steps 90, 180, and 270, followed by bulk compression at step 360. Thus, several proactive events can occur within 512 steps. The Phase-2 interval is 1,680, so no subsequent Phase-2 event occurs during this run.

These steps are calculated for Lp=4096L_p=4096; actual scheduling also depends on the appended instruction tokens. The calculated schedule explains the occurrence of proactive events; complete input lengths and capacity handling are addressed in the general clarification. The short Table 2 generation window does not establish the behavior of repeated Phase-2 compression.

Dear Reviewer QeHW, thank you for your thoughtful suggestions. Following your feedback, we have updated the PDF with the scheduling implementation in Appendix B and the experimental parameters and cache statistics in Appendix C. Our discussion responses also present the additional scheduling comparisons. We would appreciate your assessment of these revisions when convenient. More comprehensive experiments will be incorporated into the main text and appendix as the evaluation is completed.

Thank you for the detailed response and the additional experiments.

I appreciate the clarification of the compression scheduling implementation, which addresses my concern regarding the compression events reported in Table 2.

However, I still have reservations about the core motivation of proactive compression. While both CTC and PSC involve irreversible eviction decisions, PSC removes tokens earlier, potentially losing information that may become important during subsequent generation.

The newly added delayed staged compression experiment is particularly interesting. Although the delayed strategy retains more tokens on average and therefore is not a strictly memory-matched comparison, its lower attention-output deviation suggests that earlier compression is not necessarily preferable. Moreover, the improvement of PSC over CTC in this experiment is relatively modest, and the evaluation includes only three examples.

Overall, while I appreciate the authors' additional efforts, I do not think the rebuttal fully resolves my main concerns regarding the advantages and generality of proactive compression.

I will therefore maintain my original rating of 2/4 (Weak Reject).

Thank you for your follow-up. We agree that early eviction can potentially remove information that becomes important later. However, earlier eviction does not inherently imply a higher risk of such information loss. Figures 7–9 in the appendix provide empirical evidence relevant to this concern: if an evicted token becomes important during subsequent generation, its absence can manifest as increased attention-output deviation from the uncompressed model. In these experiments, PSC generally exhibits smaller and smoother attention-output perturbations than CTC during subsequent decoding. Thus, in the evaluated settings, proactive staged compression does not translate into greater overall attention-output deviation.

This observation captures our core motivation: PSC balances the potential risk of earlier information removal against the disruption caused by a single large compression event. Our claim is that distributing a large compression operation across multiple decoding steps can better preserve the attention outputs of the uncompressed model, as demonstrated in our experiments.

Lower attention-output deviation alone does not guarantee better generation quality. For precisely this reason, our evidence goes beyond attention diagnostics: we conduct extensive task-level evaluations across multiple models, datasets, and underlying compression methods. PSC outperforms CTC in the majority of evaluated configurations. These task-level results substantiate its practical effectiveness, and the exceptions do not negate the broader empirical pattern. We therefore believe that PSC’s advantages should be assessed on the combined evidence from the attention analyses and the extensive task-level evaluations, rather than the three-example diagnostic alone.

We thank you again for the constructive discussion, which has helped us clarify the motivation and empirical evidence for PSC.

Reviewer_NTM8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Proactive Staged Compression (PSC), a scheduling strategy for KV cache compression. PSC introduces smaller proactive compression operations during generation and retains capacity-triggered compression in its two-phase schedule. It adjusts compression timing and volume while retaining the selection or merging mechanisms of existing compressors. Experiments across multiple models, tasks, and compression methods show promising quality improvements, supported by attention perturbation analysis.

Critical strengths
  1. A useful perspective on KV cache compression.

The paper focuses on when to compress, which complements existing work on which tokens to retain or merge. Its compatibility with multiple compressors makes the approach potentially useful in practice.

  1. Clear presentation and promising empirical results.

The figures and derivations explain the scheduling strategy well. The evaluation covers several models and tasks, with some substantial improvements—for example, KeepKV's GovReport score increases from 16.17 to 27.40 in Table 1. The fixed-continuation attention perturbation experiments provide additional support for the motivation.

Critical weaknesses
  1. Did staged compression occur in Table 2?

    With the GovReport settings (4096 input tokens, cache capacity 4200, and 512 generated tokens), Algorithm 1 appears to make PSC-6 at 10% retention fall back to CTC after 104 tokens. The next proactive compression would require another 1260 tokens, so it would not occur during the run. Please clarify the schedule used in Table 2 and report the number of proactive compression events.

  2. The efficiency metrics need clearer definitions.

    For KeepKV at 10% retention, PSC-6 increases throughput from 21.89 to 25.59 tokens/s, while per-sample latency also increases from 13.20 to 13.78 seconds. Please clarify the timing boundaries and token-counting conventions, particularly given the fixed decoding horizon described in Appendix E. This would make the practical meaning of the reported speedup clearer.

Questions for authors

Please refer to the weaknesses above.

3 comments
AuthorsOpenReview ↗

Staged Compression and Throughput

Thank you for pointing out the scheduling discrepancy and the unclear throughput definition.

W1: Staged compression events

The relevant detail is how the implementation handles the Phase-1 ratio near cache capacity. The implementation first specifies kk and computes r1,r2r_1,r_2 using Equations (9) and (13). When necessary, it clips r1r_1 to 0.950.95 and recomputes the interval. It then performs k−1k-1 small compressions followed by bulk compression, before entering Phase 2. Under these settings, the implemented schedule performs several small compressions during Phase 1.

With scheduler input Lp=4096L_p=4096, capacity Lm=4200L_m=4200, R=0.10R=0.10, and k=6k=6, the resulting Phase-1 integer interval is 83. Five small compressions are scheduled at steps 83, 166, 249, 332, and 415, followed by bulk compression at step 498. Therefore, the clipping-based schedule can perform several proactive compressions within 512 steps. Its Phase-2 interval is 1,260, so no subsequent Phase-2 event occurs during this horizon.

These steps are calculated for Lp=4096L_p=4096. Actual scheduling also depends on the instruction tokens appended to each input. The steps above use the specified scheduler input; complete input lengths and capacity handling are therefore relevant when interpreting individual runs.

W2: Throughput and latency

The runner generates exactly 512 token IDs per example, even if EOS appears earlier. However, the reported throughput counts whitespace-separated units in the text before EOS. This text count varies between methods, explaining why the reported throughput can increase even when generation takes longer.

For a direct speed comparison, both methods should generate the same number of tokens, and throughput should be that token count divided by total time. Recalculating the existing KeepKV results at a 10.00% budget gives:

ScheduleGenerated tokensTotal time (s)Token IDs/s
CTC, paired control51,2001,324.6738.65
PSC-651,2001,377.5837.17

Both methods generate 512 tokens for each of 100 examples. This table uses PSC-6's paired CTC control; the published CTC value averages multiple controls. The timer includes tokenization, GPU transfers, prefill, generation, text decoding, and loop overhead, excluding model loading.

PSC-6 takes longer to generate the same number of tokens in this comparison. The 3.99% latency increase represents the additional cost of PSC in this comparison; its main benefit is the quality improvement studied in the paper.

Thank you for the detailed clarifications. Your response has addressed my concerns about the compression schedule and the throughput measurements. I hope you will incorporate these details into the main paper, including the clipping-based scheduling behavior and the corrected throughput results, and update the algorithm description and efficiency claims accordingly.

Thank you for your follow-up and for confirming that our explanations address your concerns. The revised PDF includes the corrected algorithm and clipping-based scheduling details in Appendix B, along with the experimental configurations and throughput definitions in Appendix C. Given the scale of the full evaluation, the complete experimental results have not yet been added. We will complete these experiments and incorporate the results into the main text and appendix, together with the corrected throughput results and corresponding revisions to the efficiency claims. We sincerely appreciate your time and constructive feedback.

Reviewer_7PTzOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes PSC, a compressor-agnostic framework that changes when KV-cache compression is performed during autoregressive decoding. Instead of waiting until the cache reaches capacity and then removing a large fraction of tokens, PSC distributes compression across earlier decoding steps using a two-phase schedule with analytically derived intervals and removal ratios. The method is evaluated with 6 existing KV-cache compressors across Llama and Qwen models on summarization, open-ended generation, and mathematical reasoning tasks and showed improvements in task quality and decoding throughput.

Critical strengths
  1. The paper investigates a meaningful aspect of KV-cache compression: when compression should be applied, rather than only which tokens should be evicted. And it demonstrates that distributing compression decisions across different decoding steps may provide more informative attention signals and mitigate the performance degradation caused by one-shot eviction.

  2. The proposed PSC is simple and independent of the compression algorithm. Its closed-form scheduling rules make it straightforward to integrate with existing KV-cache compression methods without modifying the token-selection or merging mechanisms.

  3. The paper includes diagnostic analyses of attention-output perturbations and early-deletion regret which provides useful supporting evidence for the motivation.

Critical weaknesses
  1. The reported throughput improvements are difficult to reconcile with the latency measurements in Table 2, which it raises questions about whether the throughput gains represent faster inference or differences in output length. The paper should clarify the throughput definition and distinguish fixed-length decoding speed from end-to-end latency under natural generation.

  2. Comparisons against smaller capacity-triggered compression events, rolling eviction, and periodic compression schedules, such as those explored in Scissorhands[1] and R-KV[2], are missing. Without controlling compression frequency, removal granularity, and memory occupancy, it remains unclear whether PSC's improvements come specifically from compressing before saturation or simply from avoiding large one-shot deletions.

  3. The evaluation does not sufficiently establish the benefits of PSC for long-generation workloads, since the current outputs are at most 512 tokens, and output-length distributions and per-sample compression counts are not reported. The authors should evaluate longer outputs, e.g., reasoning traces on AIME and MATH-500 with reasoning models (as in R-KV [2]) or long-form writing such as LongBench-Write [3], and report gains as a function of output length or number of compression events.

[1] Zichang Liu, et, al, "Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time", NeurIPS 2023 [2] Zefan Cai, et, al, "R-KV: Redundancy-aware KV Cache Compression for Reasoning Models", NeurIPS 2025 [3] Yushi Bai, et, al, "LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs", ICLR 2025

Questions for authors
  1. Under the reported efficiency configurations, how many PSC-specific compression events actually occur? Can the authors provide the Phase 1 feasibility checks, Phase 2 intervals, and per-sample compression counts to clarify whether the scheduler falls back to CTC?

  2. Were the results in Table 1 and the main quality figures obtained using the same configurations described in Appendix E? If so, how can the reported quality differences arise as PSC seems to be equivalent to CTC under Algorithm 1 under these configurations? If different configurations were used, can the authors report their (L_p), (L_m), (L_d), and actual compression counts, and measure quality and efficiency under the same settings?

  3. How much of PSC's benefit comes from Phase 1 versus Phase 2, particularly when the prompt is close to cache capacity? Does the advantage persist on very long generations, and how do actual compression counts, per-step latency, and peak GPU memory change?

6 comments
AuthorsOpenReview ↗

Reasoning Evaluation, Baseline Comparisons, and Scheduling

Thank you for pointing out the efficiency inconsistency and requesting additional comparisons.

W1: Throughput and latency

The original throughput counts whitespace-separated units in EOS-truncated text, although decoding continues for a fixed token count. This explains why throughput can rise alongside latency. Using generated token IDs gives 38.65 tokens/s for the existing KeepKV paired CTC control and 37.17 tokens/s for PSC-6; both generate 51,200 tokens. PSC's main benefit is quality improvement with modest latency overhead.

W2 and W3: Baseline comparisons and reasoning evaluation

The paper already evaluates MATH-500 reasoning in Table 1 and Figure 13. We added comparisons on examples each from GovReport and QMSum, using DeepSeek-R1-Distill-Qwen-1.5B and one NVIDIA A100 80GB: prompt limit 512, cache capacity 1,024, R=0.20/0.50R=0.20/0.50, and EOS stopping or a 2,048-token cap. H2O CTC and PSC share the H2O selection rule. Mean ROUGE-L is:

DatasetRH2O CTCR-KVScissorhandsPSC-3PSC-4PSC-5
GovReport0.2014.0112.8213.5216.0616.9116.66
GovReport0.5016.7415.1716.0516.9815.4616.32

At R=0.20, PSC-4 exceeds R-KV and Scissorhands by 4.09 and 3.39 points. At R=0.50, PSC-3 achieves the highest score of 16.98. R-KV uses its official selector with an adjusted buffer; Scissorhands uses a Qwen/GQA port of its official persistence selector. These results apply to the stated truncated-input setting.

PSC's core idea is to slow KV-cache growth and postpone saturation through proactively scheduled removals. Given kk, it derives the compression intervals and removal ratios while preserving the underlying token-selection or merging rule. R-KV focuses on importance–redundancy scoring with fixed-buffer compression, and Scissorhands uses persistence of importance for capacity-triggered eviction. PSC contributes a compressor-independent schedule that controls when compression occurs and how much is removed.

Q1 and Q2: Scheduling and task configurations

We specify kk, compute r1,r2r_1,r_2, clip r1r_1 to 0.950.95 when needed, and recompute the interval. With scheduler input Lp=4096L_p=4096, Lm=4200L_m=4200, R=0.20R=0.20, and k=4k=4, small events are scheduled at steps 90, 180, and 270, followed by bulk compression at 360. The next Phase-2 interval is 1,680, beyond the 512-step horizon.

Task settings differ: the original MATH-500 driver defaults are (Lp,Ld,Lm)=(512,1000,800)(L_p,L_d,L_m)=(512,1000,800); complete prompts also include instructions. Thus, feasibility must be assessed for each task. Algorithm and capacity-handling details are addressed in our general response.

Q3: Phase contributions

Phase 1 uses the initial cache headroom; Phase 2 continues compression with Δl2=2(1−R)Lm/k\Delta l_2=2(1-R)L_m/k. We tested both phases on three GovReport examples using H2O, capacity 512, R=0.20R=0.20, k=4k=4, and the same 1,792 reference tokens:

ScheduleRelative attention-output L2 ↓Reference-token NLL ↓
CTC53.43%0.65
Phase 1 only52.22%0.60
Phase 2 only53.25%0.61
Both phases51.49%0.56

Both phases give the lowest error here; QMSum shows mixed effects. CTC and PSC perform 4 and 8 compressions, respectively, with similar mean step times (27.87/27.93 ms) and peak memory (3,416.21/3,416.29 MiB). The replay continues beyond EOS to compare attention behavior over the same token sequence.

Thank you for the detailed responses and for clarifying the implementation and the throughput measurement. I have also read the other reviewers’ comments and the authors’ responses to them. I still have a few concerns:

  1. The delayed staged variant outperforms PSC while keeping more tokens on average (361.53 vs. 303.53), which suggests that cache occupancy during decoding, rather than proactive timing, may drive part of the gains. Can the authors compare PSC, CTC, and the delayed variant at matched average occupancy, and report how much larger a budget CTC needs to match PSC? I think it is very important as the quality improvement is the main benefit of PSC.

  2. I think some essential experiment details are still missing . Can the authors report the (Lp, Ld, Lm, R, k) configuration, whether r1 is clipped, the number of compression events, the average cache occupancy, and the decoding setup for each dataset in Table 1?

  3. The new experiments only use 3 to 5 examples without variance, can the authors increase the number of examples and with variance?

Thank you for the follow-up. We agree that the average number of cached tokens is an important control, and we appreciate the opportunity to clarify the evaluation setup.

Q1. Comparing schedules at matched average occupancy.

We compared the three policies on 20 GovReport prompts with DeepSeek-R1-Distill-Qwen-1.5B on H2O. Each full prompt has 512 tokens, and decoding uses a fixed 1,536-token horizon. We calculate occupancy after compression at each single-token forward and score the generated summary before EOS. The three occupancy-matched rows differ by at most 0.19 cached tokens on average.

ScheduleCache limit (tokens)Retention ratioMean cached tokensROUGE-L mean ± SD
PSC-47680.20517.9618.26 ± 4.58
CTC, matched occupancy7680.3493517.7917.72 ± 4.78
Delayed staging, matched occupancy6720.20517.7817.61 ± 4.92
CTC, same peak and retention7680.20480.7016.05 ± 4.81
CTC, first tested budget to reach PSC's mean score1,5360.20870.0718.42 ± 6.08

When we match average cache occupancy across the full 1,536-token generation run, PSC's mean ROUGE-L is 0.54 points higher than CTC's and 0.65 points higher than delayed staging's.

The full run includes decoding steps after EOS, so we also match cache occupancy up to the EOS position of the run without eviction. We use the same window for all three methods on each prompt. The mean cache lengths are 618.19 tokens for PSC, 618.33 for CTC (capacity 768, retention 0.6188), and 618.32 for delayed staging (capacity 740, retention 0.20). With average occupancy matched this closely, PSC's mean ROUGE-L is 2.10 points higher than CTC's and 0.73 points higher than delayed staging's.

With the same peak capacity of 768 tokens and the same retention ratio of 0.20, PSC's mean ROUGE-L is 2.21 points higher than CTC's. At this retention ratio, 1,536 tokens is the smallest of the seven tested capacities at which CTC reaches PSC's mean score on these prompts, twice PSC's capacity.

Q2: Table 1 settings. These are the settings in the Table 1 evaluation command and driver. Both policies use R=0.20R=0.20. PSC uses k=4k=4.

DatasetContext cap (tokens)Fixed output horizon (tokens)Configured cache capacity (tokens)
GovReport4,0965124,200
QMSum4,0962564,150
MultiNews2,0483502,200
WritingPrompts100256250
ROCStories100128150
MATH-5005121,000800
GSM8K512512700

The context cap applies before appending the instruction and chat suffix. We decode greedily with batch size 1 for the full LdL_d tokens and score only the text before the first EOS.

We reconstructed the H2O results below using the archived scheduler and the Llama-3.1-8B-Instruct tokenizer. The reconstruction covers the full available summarization and math splits and the first 2,048 training examples each from WritingPrompts and ROCStories. We count each compression step once and average KV length after each decoding step, excluding prefill.

Full prompt lengths are reported as min / median / max. The last column counts prompts where r1r_1 was clipped.

DatasetPromptsPrompt tokensClipped prompts
GovReport2002,088 / 4,114 / 4,114195
QMSum2002,667 / 4,117 / 4,141199
MultiNews200198 / 2,067 / 2,067147
WritingPrompts2,048110 / 133 / 18945
ROCStories2,04849 / 56 / 700
MATH-50050052 / 94.5 / 8013
GSM8K1,31957 / 90 / 2200

Compression events and KV tokens are averaged per prompt. Parentheses give the mean number of proactive PSC events.

DatasetCTC eventsPSC events (proactive)CTC KV tokensPSC KV tokens
GovReport0.933.65 (2.75)1,728.032,879.86
QMSum0.993.98 (2.98)1,328.702,277.75
MultiNews0.581.63 (1.63)1,296.301,829.29
WritingPrompts1.003.01 (3.00)152.54185.87
ROCStories1.002.00 (2.00)87.9190.19
MATH-5001.012.39 (2.38)411.59393.30
GSM8K0.00231.00 (1.00)349.93269.29

In GSM8K, only 3 of 1,319 prompts reach the CTC cache limit of 700 tokens. Each triggers one event, yielding 3/13193/1319, or 0.0023 events per prompt. For H2O with PSC, Equation (9) values above 0.95 are clipped to r1=0.95r_1=0.95, and the compression interval is recalculated.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Clarification of PSC Scheduling and the Quality–Latency Trade-off

We thank all four reviewers for identifying the discrepancy between the algorithm description and the implementation, and for requesting a clearer explanation of PSC's efficiency. We would like to clarify two central points.

PSC's core idea is to slow the growth of the KV cache toward its capacity limit. We first specify kk, then compute r1r_1 and r2r_2 using Equations (9) and (13), given the prompt length, cache capacity, and retention ratio. Phase 1 and Phase 2 are consecutive parts of the same scheduling process: Phase 1 manages the initial growth toward capacity, and Phase 2 schedules compression during continued generation. Depending on the decoding length LdL_d, generation may end during Phase 1 or continue through both phases. The two-phase design therefore accommodates different generation lengths within the same framework. The revision will align the algorithm description with the implemented scheduling rules and address cache-capacity handling.

Repeated compression introduces additional work, so some latency overhead is expected. Our measurements show that this overhead can remain small: in the existing paired KeepKV comparison, PSC-6 takes 1,377.58 s versus 1,324.67 s for CTC to generate the same 51,200 tokens, an increase of only 3.99%. We consider this a modest cost for the quality improvements reported in the paper. PSC's primary benefit is improved generation quality under a limited KV-cache budget. The appropriate interpretation is a quality–latency trade-off, with throughput measured as generated token IDs per second.

AuthorsOpenReview ↗

Compression Scheduling and Efficiency Metrics

Thank you for pointing out the scheduling discrepancy and the ambiguity in the efficiency metric.

W1 and Q1

The key implementation detail is the clipping of the Phase-1 compression ratio when the prompt is close to capacity. In the implementation, we first specify kk and then compute r1r_1 and r2r_2 using Equations (9) and (13), given the other configuration parameters.

When the prompt is close to capacity, Equation (9) can yield r1>1r_1>1. Specifically, this occurs when Lm−Lp<(1−R)Lm/(k−1)L_m-L_p<(1-R)L_m/(k-1). For example, Lp=4096L_p=4096, Lm=4200L_m=4200, R=0.20R=0.20, and k=4k=4 give r1≈1.29r_1\approx1.29. The generic implementation clips r1r_1 to 0.950.95 and recomputes the interval. It then performs k−1k-1 small compressions and compresses to ⌊RLm⌋\lfloor RL_m\rfloor at the kk-th event, before entering Phase 2.

For these nominal lengths, the resulting schedule is:

ScheduleSmall-compression stepsRemoved per small eventBulk-compression stepPhase-2 interval
PSC-394, 188892822,240
PSC-490, 180, 270853601,680
PSC-586, 172, 258, 344814301,344
PSC-683, 166, 249, 332, 415784981,120

These are calculated scheduler-call steps, assuming scheduler input Lp=4096L_p=4096. They explain why several proactive events can occur within 512 steps, although no subsequent Phase-2 event occurs. Actual lengths also include the appended instruction.

To provide the timing test requested in Q1, we instrumented the H2O implementation on five fixed GovReport examples using DeepSeek-R1-Distill-Qwen-1.5B and one NVIDIA A100 80GB. Actual prompt length is 4,105, scheduler input is the final zero-based position 4,104, and the Lp > Lm fallback flag is false. PSC-4 removes 78, 78, 78, and 3,362 entries per layer at one-based output positions 83, 166, 249, and 332.

ScheduleExecuted tokens, totalTokens before EOS, totalTotal wall time (s)Token IDs/s
CTC2,5602,56077.8432.89
PSC-42,5602,56079.4932.21

W2 and Q2

Throughput and latency use the same elapsed time, but the original throughput numerator counts whitespace-separated units in EOS-truncated text. Generation nevertheless continues to the fixed token horizon. The text count can therefore increase even when execution takes longer.

For the same number of generated tokens, throughput is NLd/TNL_d/T, where NN is the sample count, LdL_d the executed output length, and TT the total time. The requested fixed-work counts are reported above; no EOS occurs within those horizons. Timing includes context prefill, generation, and output decoding, excluding model loading. These results quantify the latency cost of applying PSC.

Overall, PSC improves task scores across most of the evaluated compressor–task configurations, as reported in Table 1 and the additional evaluations in the appendix. The CaM/GSM8K decrease from 71.73% to 61.09% is an isolated exception to this broader trend. The results therefore support the overall effectiveness of PSC while showing that individual configurations can behave differently.

Dear Reviewer 823w, thank you again for your careful review. The revised PDF now includes the corrected scheduling algorithm in Appendix B and detailed experimental settings and measurement definitions in Appendix C. These revisions accompany our responses on compression events and throughput. When you have a chance, we would greatly appreciate your feedback on whether the clarifications address your questions. We would be happy to provide any further details.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.