- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
Optimization-based KV-cache compression methods often achieve better compression quality than eviction-based methods, but require expensive per-context optimization. To address this trade-off, the paper introduces STILL, a learned method that amortizes KV-cache compression, aiming to preserve compression quality while substantially reducing computational cost. STILL uses a separate cross-attention encoder for each Transformer layer to map the original keys and values into a smaller set of synthesized KV entries. Through experiments across various models, benchmarks, and compression ratios, the authors demonstrate that STILL achieves competitive compression quality on the tested benchmarks, sometimes outperforming existing optimization-based methods, while producing compressed KV caches significantly faster.
Critical strengths
-
KV-cache compression is an important problem, and the proposed approach seems promising.
-
The method is simple and intuitive.
-
The paper demonstrates strong compression results across the evaluated benchmarks.
Critical weaknesses
1.Insufficient comparison with related amortized context compression methods. The paper discusses several previous amortized KV-compression approaches, including KV-Distill, CCM, and Activation Beacon. However, it does not provide a direct experimental comparison with them. Additionally, related works such as Doc-to-LoRA and SHINE amortize context compression into LoRA adapters rather than KV caches, but are not discussed. A more thorough comparison with these approaches would help clarify the novelty and practical advantages of STILL.
-
Unclear inference-time efficiency benefits. Table 3 reports that generation with STILL is slower than generation with the uncompressed model (10.89 vs. 8.44 seconds per question). Why is this the case? Given the substantially smaller KV cache, one would expect generation to be considerably faster, as appears to be the case with LCLM. If STILL does not provide meaningful generation speedups, this raises questions about its practical utility. A more thorough evaluation of inference latency and throughput would help clarify its benefits.
-
Significant performance degradation under repeated long-context compression. The iterative compression experiments show substantial performance degradation as context length increases, with accuracy dropping to approximately 34–45% at 128K tokens. Importantly, this degradation is considerably larger than that of the full-context baseline. While some deterioration is expected under compression, these results provide limited evidence that STILL can reliably preserve information over extended sequences. Comparisons with other compression methods in this setting would also help establish whether STILL offers an advantage for long-horizon applications.
Questions for authors
Please see the weaknesses above.
3 comments
I thank the authors for their detailed rebuttal and additional experiments. The new results are encouraging and address some of my concerns. However, I feel that several important issues remain unresolved.
- Comparison with amortized compression methods. My main concern regarding the lack of comparisons with previous amortized context compression methods remains largely unaddressed. While the additional comparisons with eviction-based and reasoning-focused methods are useful, methods such as KV-Distill, CCM, and Activation Beacon are more directly related to the main contribution of STILL, namely amortizing context compression through a learned network. I believe direct comparisons with these approaches are important for establishing the advantages of the proposed method. The relationship to other amortized approaches, such as Doc-to-LoRA and SHINE, also deserves more discussion.
2.Inference-time efficiency. I appreciate the additional latency experiments presented in the response to reviewer fxJQ. The reported speedups of 1.05×, 1.15×, and 1.45× at 8K, 32K, and 128K contexts are encouraging. However, it remains unclear how these results reconcile with Table 3, where STILL is considerably slower than the uncompressed model (10.89 vs. 8.44 seconds per question). While I agree with the authors that the method may be particularly beneficial in settings where the same context is reused across multiple queries, this does not address its efficiency during long reasoning trajectories, which are a major use case in the paper. It would be useful to evaluate wall-clock speedups in both settings, including the overhead of repeated compression and how performance changes with generation length.
- Repeated long-context compression. The improvement from 44.81% to 61.81% accuracy at 128K is substantial and encouraging. However, this is still considerably below the full-context accuracy of 79.50%. More importantly, my original concern regarding the lack of comparisons with other compression methods in this setting remains unaddressed. Without such comparisons, it is difficult to determine whether STILL offers a meaningful advantage for long-horizon applications.
Thank you for the detailed feedback.
- To highlight the performance of Still with other amortized context compression methods, we present the following comparison of Still and Activation Beacon:
| Length | Qwen3-8B full | No context | Still | Beacon | Still − Beacon |
|---|---|---|---|---|---|
| 32K | 81.0 | 38.3 | 76.0 | 53.3 | +22.8 |
| 64K | 70.5 | 36.5 | 65.8 | 53.5 | +12.3 |
| 128K | 70.3 | 26.5 | 53.0 | 28.3 | +24.8 |
At matched horizon, Still outperforms Activation Beacon at every length. We trained Qwen3-8B Activation Beacon leveraging the paper’s code and ported it to the Qwen3 family in order to run this comparison. At 128K, our Qwen3 port degrades when memory surpasses 16K tokens (74% answer parse rate); with 8K memory Activation Beacon reaches 40.5 while Still reaches 53.8, a 13.3 point lead for Still at matched 16× compression. The Still used in this comparison is the same trained Still from the MEMENTO benchmarks (trained iterative compaction regime at 8k with math reasoning traces in the data mix).
Though Doc-to-LoRA and SHINE both amortize the KV cache of models, they do so by altering model parameters through the use of LoRAs. Still’s amortization takes place solely in the KV cache space, with base model weights unaffected. As SHINE or Doc-to-LoRA process additional contexts their updates are restricted to a space defined by the LoRA’s rank, whereas Still outputs new keys and values that can be attended to via standard attention, only limited by the size of the models context window. Still is built and evaluated to compact both ingested documents and the source model’s own reasoning, the later being an important setting neither Doc-to-LoRA or SHINE address.
- Thank you for pointing this out. Our Table 3 results do present Still (10.89s) as higher latency than the full context baseline (8.44s), but one can also notice the no-context control with higher latency than full context (10.13s). These results show latency is attributed to decode time, at a small context size of 8K tokens, a smaller cache saves little. However, as our prior table and the analysis below shows, Still’s latency advantages become clear at higher concurrency and under larger decode volumes:
| Requests | Raw tok/s | Still tok/s | Still / raw | Raw preemptions | Still preemptions |
|---|---|---|---|---|---|
| 1 | 137 | 143 | 1.04× | 0 | 0 |
| 8 | 649 | 819 | 1.26× | 0 | 0 |
| 16 | 826 | 1,245 | 1.51× | 3 | 0 |
| 32 | 916 | 1,685 | 1.84× | 19 | 0 |
| 64 | 918 | 1,441 | 1.57× | 114 | 100 |
From batch size 1 to 64 we compare stock vLLM and our implementation of Still in vLLM (one H100) with Qwen3-8B generating 30K tokens (25 compactions). At batch size 1 there is parity between the methods but as we increase concurrency Still achieves throughput advantages from 1.26× to 1.84× with fewer preemptions (KV cache full). At concurrency 32 the base model begins to run out of memory (19 preemptions, 3 at concurrency 16) while Still has 0 preemptions and reaches 1.84× higher throughput. We will include this more detailed latency table in the camera-ready.
- We defer to the Activation Beacon experiments above demonstrating Still’s competitive performance at 128K context length.
We hope we have better responded to your concerns regarding our submission. If we have, please consider an increase in score. Thank you again for your time reviewing.
Thank you for your informative review, and we appreciate you highlighting the simplicity of the method and strong results. We address your questions and concerns below:
1. Please see our responses to the above reviewers. We hope this provides further evidence for our method’s competitive advantage.
2. We appreciate pointing this out, total wall-clock is a slightly more nuanced question that both depends on the inference engine and the balance of the problem (namely, number of document tokens compared to number of sampled tokens). Still, as with any KV compression method across the sequence index, becomes more useful as (a) more prompts are used on the same context, and similarly (b) amortized settings where a KV cache hit rate can mean document prefills do not need to be recomputed.
3. Thank you for drawing attention to this figure. Before presenting new results below, we would like to note that the 128k-token evaluation extrapolates beyond the 32k-token training context, which may contribute to the observed degradation. Continuing training for 300 updates at the same 32k tokens, while (a) allowing the answer-distillation loss to backpropagate through later context processing into earlier compactions and (b) allowing the compactor’s attention-bias weights to train, improved accuracy on the same 128k-token evaluation cohort from 44.81% to 61.81%; full-context accuracy was 79.50%. The Qwen3-4B backbone remained frozen throughout. While still not full accuracy, we believe this is a promising signal that longer-context results will improve with scale of training, rather than being fundamentally limited.
We would be happy to answer any further questions you may have; if we have adequately addressed your concerns, we would be grateful if you could please consider raising your score. Thank you again for your review.