acceptodds
Under review as a conference paper at ICLR 2027

Still: Amortized KV Cache Compaction in a Single Forward Pass

Abstract

A bottleneck for longer-horizon deployments of LLMs is the memory stored in the KV cache. Iterative KV cache compaction can bound the KV cache growth, but existing approaches each have their tradeoffs. Selection and eviction methods are cheap but are bound to a less expressive token space, while synthesis methods utilize the continuous embedding space but often rely on costly per-context optimization. To balance both approaches, we introduce Still: a series of learned compressive maps at each layer of the KV cache, trained on a constructed MCQ dataset. We evaluate our KV compression method on Qwen and other open source models across long-document question answering, mathematical & scientific reasoning, and code generation. Still beats baseline methods at higher compression rates (8x to 200x), and on reasoning tasks can stay within 1.8 points of full-context performance on MATH-500 and GPQA-Diamond whilst using up to 3.8x less accumulated KV cache.

Primary area:
generative models
Keywords:
KV compression, long-context language models, memory-efficient transformers

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_m62qOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Optimization-based KV-cache compression methods often achieve better compression quality than eviction-based methods, but require expensive per-context optimization. To address this trade-off, the paper introduces STILL, a learned method that amortizes KV-cache compression, aiming to preserve compression quality while substantially reducing computational cost. STILL uses a separate cross-attention encoder for each Transformer layer to map the original keys and values into a smaller set of synthesized KV entries. Through experiments across various models, benchmarks, and compression ratios, the authors demonstrate that STILL achieves competitive compression quality on the tested benchmarks, sometimes outperforming existing optimization-based methods, while producing compressed KV caches significantly faster.

Critical strengths
  1. KV-cache compression is an important problem, and the proposed approach seems promising.

  2. The method is simple and intuitive.

  3. The paper demonstrates strong compression results across the evaluated benchmarks.

Critical weaknesses

1.Insufficient comparison with related amortized context compression methods. The paper discusses several previous amortized KV-compression approaches, including KV-Distill, CCM, and Activation Beacon. However, it does not provide a direct experimental comparison with them. Additionally, related works such as Doc-to-LoRA and SHINE amortize context compression into LoRA adapters rather than KV caches, but are not discussed. A more thorough comparison with these approaches would help clarify the novelty and practical advantages of STILL.

  1. Unclear inference-time efficiency benefits. Table 3 reports that generation with STILL is slower than generation with the uncompressed model (10.89 vs. 8.44 seconds per question). Why is this the case? Given the substantially smaller KV cache, one would expect generation to be considerably faster, as appears to be the case with LCLM. If STILL does not provide meaningful generation speedups, this raises questions about its practical utility. A more thorough evaluation of inference latency and throughput would help clarify its benefits.

  2. Significant performance degradation under repeated long-context compression. The iterative compression experiments show substantial performance degradation as context length increases, with accuracy dropping to approximately 34–45% at 128K tokens. Importantly, this degradation is considerably larger than that of the full-context baseline. While some deterioration is expected under compression, these results provide limited evidence that STILL can reliably preserve information over extended sequences. Comparisons with other compression methods in this setting would also help establish whether STILL offers an advantage for long-horizon applications.

Questions for authors

Please see the weaknesses above.

3 comments

Thank you for your informative review, and we appreciate you highlighting the simplicity of the method and strong results. We address your questions and concerns below:

1. Please see our responses to the above reviewers. We hope this provides further evidence for our method’s competitive advantage.

2. We appreciate pointing this out, total wall-clock is a slightly more nuanced question that both depends on the inference engine and the balance of the problem (namely, number of document tokens compared to number of sampled tokens). Still, as with any KV compression method across the sequence index, becomes more useful as (a) more prompts are used on the same context, and similarly (b) amortized settings where a KV cache hit rate can mean document prefills do not need to be recomputed.

3. Thank you for drawing attention to this figure. Before presenting new results below, we would like to note that the 128k-token evaluation extrapolates beyond the 32k-token training context, which may contribute to the observed degradation. Continuing training for 300 updates at the same 32k tokens, while (a) allowing the answer-distillation loss to backpropagate through later context processing into earlier compactions and (b) allowing the compactor’s attention-bias weights to train, improved accuracy on the same 128k-token evaluation cohort from 44.81% to 61.81%; full-context accuracy was 79.50%. The Qwen3-4B backbone remained frozen throughout. While still not full accuracy, we believe this is a promising signal that longer-context results will improve with scale of training, rather than being fundamentally limited.

We would be happy to answer any further questions you may have; if we have adequately addressed your concerns, we would be grateful if you could please consider raising your score. Thank you again for your review.

I thank the authors for their detailed rebuttal and additional experiments. The new results are encouraging and address some of my concerns. However, I feel that several important issues remain unresolved.

  1. Comparison with amortized compression methods. My main concern regarding the lack of comparisons with previous amortized context compression methods remains largely unaddressed. While the additional comparisons with eviction-based and reasoning-focused methods are useful, methods such as KV-Distill, CCM, and Activation Beacon are more directly related to the main contribution of STILL, namely amortizing context compression through a learned network. I believe direct comparisons with these approaches are important for establishing the advantages of the proposed method. The relationship to other amortized approaches, such as Doc-to-LoRA and SHINE, also deserves more discussion.

2.Inference-time efficiency. I appreciate the additional latency experiments presented in the response to reviewer fxJQ. The reported speedups of 1.05×, 1.15×, and 1.45× at 8K, 32K, and 128K contexts are encouraging. However, it remains unclear how these results reconcile with Table 3, where STILL is considerably slower than the uncompressed model (10.89 vs. 8.44 seconds per question). While I agree with the authors that the method may be particularly beneficial in settings where the same context is reused across multiple queries, this does not address its efficiency during long reasoning trajectories, which are a major use case in the paper. It would be useful to evaluate wall-clock speedups in both settings, including the overhead of repeated compression and how performance changes with generation length.

  1. Repeated long-context compression. The improvement from 44.81% to 61.81% accuracy at 128K is substantial and encouraging. However, this is still considerably below the full-context accuracy of 79.50%. More importantly, my original concern regarding the lack of comparisons with other compression methods in this setting remains unaddressed. Without such comparisons, it is difficult to determine whether STILL offers a meaningful advantage for long-horizon applications.

Thank you for the detailed feedback.

  1. To highlight the performance of Still with other amortized context compression methods, we present the following comparison of Still and Activation Beacon:
LengthQwen3-8B fullNo contextStillBeaconStill − Beacon
32K81.038.376.053.3+22.8
64K70.536.565.853.5+12.3
128K70.326.553.028.3+24.8

At matched horizon, Still outperforms Activation Beacon at every length. We trained Qwen3-8B Activation Beacon leveraging the paper’s code and ported it to the Qwen3 family in order to run this comparison. At 128K, our Qwen3 port degrades when memory surpasses 16K tokens (74% answer parse rate); with 8K memory Activation Beacon reaches 40.5 while Still reaches 53.8, a 13.3 point lead for Still at matched 16× compression. The Still used in this comparison is the same trained Still from the MEMENTO benchmarks (trained iterative compaction regime at 8k with math reasoning traces in the data mix).

Though Doc-to-LoRA and SHINE both amortize the KV cache of models, they do so by altering model parameters through the use of LoRAs. Still’s amortization takes place solely in the KV cache space, with base model weights unaffected. As SHINE or Doc-to-LoRA process additional contexts their updates are restricted to a space defined by the LoRA’s rank, whereas Still outputs new keys and values that can be attended to via standard attention, only limited by the size of the models context window. Still is built and evaluated to compact both ingested documents and the source model’s own reasoning, the later being an important setting neither Doc-to-LoRA or SHINE address.

  1. Thank you for pointing this out. Our Table 3 results do present Still (10.89s) as higher latency than the full context baseline (8.44s), but one can also notice the no-context control with higher latency than full context (10.13s). These results show latency is attributed to decode time, at a small context size of 8K tokens, a smaller cache saves little. However, as our prior table and the analysis below shows, Still’s latency advantages become clear at higher concurrency and under larger decode volumes:
RequestsRaw tok/sStill tok/sStill / rawRaw preemptionsStill preemptions
11371431.04×00
86498191.26×00
168261,2451.51×30
329161,6851.84×190
649181,4411.57×114100

From batch size 1 to 64 we compare stock vLLM and our implementation of Still in vLLM (one H100) with Qwen3-8B generating 30K tokens (25 compactions). At batch size 1 there is parity between the methods but as we increase concurrency Still achieves throughput advantages from 1.26× to 1.84× with fewer preemptions (KV cache full). At concurrency 32 the base model begins to run out of memory (19 preemptions, 3 at concurrency 16) while Still has 0 preemptions and reaches 1.84× higher throughput. We will include this more detailed latency table in the camera-ready.

  1. We defer to the Activation Beacon experiments above demonstrating Still’s competitive performance at 128K context length.

We hope we have better responded to your concerns regarding our submission. If we have, please consider an increase in score. Thank you again for your time reviewing.

Reviewer_637xOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Still, a learned KV cache compactor for frozen LLMs. At each layer, a single cross attention block with per head learned latent queries reads a block of T cache entries (keys with RoPE removed, concatenated with values) and outputs t new key and value entries that replace the block. The compactor is trained on document MCQ data with a KL loss to full context predictions, normalized by the no document KL. It is applied once to a prefilled prefix or repeatedly during generation with a raw lookahead buffer.

Critical strengths
  • The method removes per context optimization from synthesis based compaction. Each compaction is a single cross attention per layer. In the static comparison, it is orders of magnitude faster than Attention Matching and Cartridges and also more accurate under the same model, documents, and budget.

  • The base model is left unchanged, and compactors trained only on document MCQ run on the model's own reasoning traces. Compaction occurs in a large share of responses, and accuracy on MATH-500 and GPQA-D stays close to the uncompressed model at a moderate KV reduction.

Critical weaknesses
  • The paper lacks an analytical motivation, and its academic contribution is limited. The design rests on a conceptual argument and an analogy to amortized inference rather than on an observed problem or an analysis of the KV cache. The method itself applies a simplified version of an existing latent query resampler with a distillation objective. As a result, the paper reports what works but offers little understanding of why it works or what principle the design follows.

  • The accuracy advantage over MEMENTO is small and inconsistent. On average, the two methods show nearly the same accuracy change and KV usage, and Still does not show a consistent advantage across models and benchmarks. MEMENTO fine-tunes the base model and reports deltas against separately fine-tuned controls rather than the original base models, while Still trains a separate compactor per base model. Neither training cost is compared, and how much the fine-tuning itself shifts accuracy is not shown, so a clear comparison is not possible.

  • There is no analysis of what the trained compactor does. Its output is a per head softmax weighted average of cache rows followed by one shared linear map, initialized as position based local averaging. The paper does not show how far the learned attention moves from this initialization, which source entries each latent draws on, or how compact entries relate to the original keys and values. Without this, it cannot be determined whether the gain comes from learned synthesis, the paper's main claim, or from learned local pooling.

  • The reasoning evaluation lacks comparison with recent KV compression methods. MEMENTO is the only baseline on the reasoning benchmarks, and no training free decode time method is evaluated there. Recent methods designed for reasoning traces, such as R-KV[1] and TriAttention[2], are neither cited nor compared. A comparison under fair conditions is needed to show whether Still improves on current reasoning KV compression.

[1] "R-KV: Redundancy-aware KV Cache Compression for Reasoning Models", arXiv:2505.24133 [2] "TriAttention: Efficient Long Reasoning with Trigonometric KV Compression", arXiv:2604.04921

Questions for authors
  • How do the trained latent attention patterns differ from the local averaging initialization, and what accuracy does the compactor reach at initialization, before training?

  • Comparison with recent reasoning KV compression methods such as R-KV and TriAttention under fair conditions, on the same models and benchmarks as Table 1.

  • What is the training cost of one compactor for one base model, compared with MEMENTO's fine-tuning? Under your harness, how do MEMENTO's uncompressed controls compare with the original base models, which would show the effect of fine-tuning?

  • Query agnostic results on a standard long context benchmark (e.g., RULER, LongBench, SCBench) on Qwen3-4B at moderate and high compression ratios.

  • What observation or analysis of the KV cache is the design derived from? Beyond applying a simplified version of an existing latent query resampler with a distillation objective, what new understanding does the paper provide about how KV caches can be compressed?

4 comments

Thank you for your informative review, and we appreciate you highlighting the notable speedup and compact training overhead. We address your questions and concerns below:

Weaknesses

1. For our motivation, the main question we are interested in is the following: can learning simple, independent, per-layer transformations of KV caches enable higher compression than purely token space or forward pass-preserving methods? These have been trained directly (e.g. Cartridges) with success, but here we show we can train a cheap per-layer model that can still effectively compress the KV cache.

2. We agree the reported results are similar to MEMENTO, but would like to add that MEMENTO is designed specifically to compress model generated reasoning traces, while our method is generic. Still compresses any arbitrary KV cache, whether it be created by the model or be from the environment it is in. We do agree the presented results are marginal, and have gotten better results from further training. Specifically, as mentioned in the limitations, we have added additional data that resembles test time more, notably, math reasoning traces generated on-policy for each model (Qwen3-32B and Qwen3.8-27B train on traces from Qwen3-8B). This is the only change in training across architecture, loss, data, optimizations for the updated results:

ModelMethodAvg Δ ↑Avg r_AUC ↓
Qwen3-32BStill−0.30.47
MEMENTO−1.50.42
Phi-4-reasoningStill+0.60.38
MEMENTO−2.20.43
Qwen3-8BStill−1.00.47
MEMENTO−3.10.34
Olmo-3-7B-ThinkStill0.00.72
MEMENTO−0.70.89
Mean, 4 modelsStill−0.20.51
MEMENTO−1.90.52
Qwen3.8-27BStill+0.30.41

In our updated results, averaged across the four benchmarks, Still wins the head to head against MEMENTO in 6/8 comparisons and achieves effective parity to the base model with half the KV cache usage. We note that these comparisons leverage the raw base model as the control for Still, and the post-trained controls reported by MEMENTO. As Still does not train the base model weights, we believe this is the fairest comparison.

Our fine-grained evaluation scores:

ModelMethodAIME'26 Δ ↑AIME'26 r_AUC ↓MATH-500 Δ ↑MATH-500 r_AUC ↓GPQA-D Δ ↑GPQA-D r_AUC ↓LCB v6 Δ ↑LCB v6 r_AUC ↓
Qwen3-32BStill−2.40.51+0.60.34+0.80.50−0.30.51
MEMENTO−1.50.40−0.70.45−2.50.48−1.30.34
Phi-4-reasoningStill+1.80.45−0.90.24+0.50.41+0.80.44
MEMENTO−2.20.40−0.90.53−2.50.44−3.20.35
Qwen3-8BStill−2.10.48−1.00.41+0.30.49−1.00.50
MEMENTO−7.40.34+0.40.40−2.00.34−3.50.29
Olmo-3-7B-ThinkStill−1.20.72−0.10.73+1.50.75−0.30.70
MEMENTO−4.41.02+0.70.93+3.81.06−2.80.55
Qwen3.8-27BStill−1.50.47−0.20.41+1.00.38+2.00.40

Still outperforms MEMENTO on accuracy in 11/16 benchmarks, losing more than 1 point only 3/16 times (MEMENTO loses >1 point on 11/16 benchmarks).

As Still does not perform any weight training of the base model we believe the fairest comparison is against the untrained base model and as such do not present fine-tuned controls. We present MEMENTO’s results against its reported fine-tuned controls. While untested here, we further believe Still could be applied and co-trained in the fine-tuning of an LLM where we expect the above results to be maintained.

With regards to training compute: we claim comparable accuracy to memory trade-off with MEMENTO, and want to highlight that Still achieves this without modifying the base-model weights or generating summaries. Our training compute comparison shows Still to be 3.6-4.8x cheaper to train than MEMENTO, before considering Still’s cheaper data generation and lower training compute requirements:

MethodQwen3-8B training cost (10^18 FLOPs)Relative to STILL
STILL52.381.00×
MEMENTO (parameter matmuls only)189.723.62×
MEMENTO (+ one forward recomputation)252.964.83×

For MEMENTO’s two estimates, we use 6NT6NT and 8NT8NT, with NN being 8 billion parameters and T=31,000×5×2×12,750T = 31,000 \times 5 \times 2 \times 12,750 tokens. Both omit attention; the latter includes one additional forward pass for recomputation. The 12,750-token length is a proxy average over OpenMementos, not the exact dataset average in MEMENTO training.

3. Thank you for the scrutiny. We note that the experiments that were run were not the main focus of our paper. However, we have included here some statistics that demonstrate the importance of training Still. Our KL training greatly increases the downstream performance (+52.3 points after only 300 updates) compared to initialization. At initialization and under 8x compression Stills attention is highly concentrated on every 8th token. Through training, this attention is redistributed across keys and values, and both original target position mass and largest attention weight drop from 83% to 31-42%, demonstrating Still learning to attend to a diverse set of keys and values.

Evaluation conditionCorrect / 195MCQ accuracy
No context3316.9%
Compactor at initialization5427.7%
Compactor after 300 updates15680.0%
Full context18695.4%
Attention diagnosticInitializationTrained
Mass on original target position83.0%31.2%
Largest attention weight per latent83.0%42.1%
Normalized attention entropy0.0550.205
Effective source coverage18.4%47.0%

Our component ablations show that both training Z (our learned queries) and all of the key, value, and output projections contribute to improved performance. Training the output projection increases accuracy by 5 points, transforming representations adds gains beyond learned attention alone.

Z scopeKVO scopeTrain ZTrain KVOParams (M)Accuracy
Layer + headLayerNoV2.3642.3%
Layer + headLayerNoKV4.7249.5%
Layer + headLayerNoKVO7.0854.6%
Layer + headLayerYesKVO16.5259.2%

We agree that a mechanistic understanding of how Still and related KV cache compression work would be of use to the field and plan to add it to the current paper for a camera ready submission.

4. Thank you for pointing these out, we focused our comparison on MEMENTO as it is the strongest reasoning baseline and the table is numerically dense, but we agree that comparing against more methods will give future researchers more insight on benefits and tradeoffs. The following are results on TriAttention and R-KV on MATH-500:

Table R1: MATH-500 accuracy and KV usage on Qwen3-8B.

Method (Qwen3-8B)nRaw → methodΔ vs rawr_AUC
Still (ours)100096.7% → 95.7%−1.00.41
MEMENTO—89.7% → 90.1%+0.40.40
TriAttention100096.3% → 85.9%−10.40.32
R-KV100096.3% → 72.0%−24.30.46

While TriAttention and R-KV demonstrate similar KV cache compression to Still and MEMENTO they do so at the cost of considerable accuracy, far exceeding that of Still or MEMENTO. Still and MEMENTO often reach near parity or surpass the baseline, TriAttention and R-KV reduce model performance (-10.4 and -24.3 respectively) for their gain in KV cache footprint. We present results only on MATH500 and note that it is the least challenging benchmark tested. Still and MEMENTO both score -1 to +0.7 on MATH500 across all tested models.

We will run the remaining benchmarks for the camera ready.

Questions

For the first three questions (A, B, C), please refer to our responses and results above (for weaknesses 3, 4, and 2 respectively). For the remaining questions:

(D) We have now evaluated query-agnostic Still on the Qasper task from LongBench, using Qwen3-4B-Instruct-2507 and all 200 evaluation questions. Results below use the official answer-F1 metric on a held-out validation set and report means across eight matched training seeds, where we also evaluate the trainability on this benchmark:

CompressionInitial StillAfter supervised trainingAfter supervised training + RL
32×21.5625.8729.29
64×14.4923.1724.00

Full-context performance is 44.13 F1, and the original Still checkpoint achieves 37.03 F1 at 8× compression. Adaptation updates only the compactor, including its attention-bias parameters, and the Qwen backbone remains frozen. Our results here are currently limited to the Qasper task as these evaluations are compute and time-intensive, but we will run the remaining benchmarks for the camera-ready.

(E) Still is composed of simple per-layer compactors whose output KV’s do not necessarily follow a standard forward pass, both in terms of not corresponding to existing token id’s or a forward pass of any input vectors of the same shape, so the output space is more expressive. If this model class is expressive enough and trains well, then the compaction time is also greatly shortened, as all layers’ compressions can be parallelized.

We would be happy to answer any further questions you may have; if we have adequately addressed your concerns, we would be grateful if you could please consider raising your score. Thank you again for your review.

Thank you for the detailed response. The additional analysis shows that training moves the compactor beyond its initialization, and I appreciate the new results. However, my main concern remains the limited academic contribution: the design is still not derived from an observation or analysis of the KV cache, and the response restates the motivating question rather than offering new understanding of how KV caches can be compressed. I will therefore keep my score.

Thank you for the follow-up comments. We appreciate your decisions on score, but we would like to note that, to our knowledge, no other method learns a per-layer compression model over sequence length. Our work takes observations from Cartridges (Eyuboglu et al., 2026), showing per-layer directly-learned KV caches can perform well on diverse tasks (despite being optimized per-layer and not necessarily obeying the model’s own forward pass), and our hypothesis that we can learn a compression model over such Cartridges (and in fact this model per-layer can be quite simple and out-perform Cartridges and Attention Matching). In our rebuttal we have further shown this also outperforms a wider array of KV sequence compression alternatives.

We again thank you for your time reviewing and discussing, and respect whatever decision you decide to make.

Reviewer_fxJQOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
3
Soundness
2(+1)
Read review
Paper summary

This paper proposes STILL, a technique to compress the KV cache of a decoder-only transformers. It adopts a compactor to compress the KV cache at the token dimension with a fixed compression ratio, achieving a compression rates of 8x to 200x while preserving most of the model capability in downstream tasks, including the tasks concerned with reasoning. Experiments show it has advantage over baseline methods in terms of compression rates and absolute performance.

Critical strengths
  1. The method is clearly described with both figure illustration and formal definition. The architecture is easy to understand and natural in the context of compression.
  2. The authors adopt a per-layer and per-head parameterization, which helps to further strengthen the modeling capability.
  3. The method is shown to outperform some baseline methods.
Critical weaknesses
  1. Novelty: The idea of compressing KV cache at the dimension of sequence length is not new. Many recent works have explored this direction with different methods, and using cross attention is not novel enough. Some recent works include Dynamic Memory Compression (ICML 2024), KV-Distill, LycheeMemory, and the architecture used in DeepSeek V4. Most of these papers were cited in the paper but the authors need to better differentiate Still from this line of work.
  2. Comparison to some baselines is missing in table 1. Why are some methods, such H2O, SnapKV, and StreamingLLM, not shown in this table?
  3. Missing ablation studies: Many design choices, such as the per-layer and per-head parameterization, are not tested against ablated versions.
  4. Wall-clock inference time should be measured in the paper. There might be trade-off in the overhead of compression and the saved attention at inference time, and this should be experimented and reported in the paper.
Questions for authors

Please refer to my weakness section.

1 comment

Thank you for your informative review, and we appreciate you highlighting the readability of our paper and competitive advantage of our method. We address your questions and concerns below:

1. We agree that there are many other methods that study compressing the KV cache through the sequence length, including the cross attention method. Our contribution with Still, and the main question we are interested in, is the following: can learning simple, independent, per-layer transformations of KV caches enable higher compression than purely token space or forward pass-preserving methods? In particular considering models that individually manipulate each layer’s KV – this to our knowledge is novel for our method.

2. As Table 1 is a reasoning-focused benchmark and this table is numerically dense, we decided to focus on the most recent reasoning-focused compressor MEMENTO. However, we agree that it is still useful to run these baselines, and have run them on Math 500, with the exception of SnapKV as this method focuses on prompt compression (not reasoning) and the prompts are too small for SnapKV’s compression. There are a number of further benchmarks we wish to run, however we note these are very compute and time intensive and will therefore not be ready until our camera-ready submission. For Math 500, the following are the results:

BenchmarkModelFull-context accuracyMethodAccuracyΔ accuracy (pp)r_AUC
MATH-500Qwen3-8B96.4%H2O91.5%-4.90.492
MATH-500Qwen3-8B96.4%StreamingLLM81.0%-15.40.299
MATH-500Qwen3-32B96.4%H2O93.4%-3.00.474
MATH-500Qwen3-32B96.4%StreamingLLM84.8%-11.60.320
MATH-500OLMo-3-7B-Think97.4%H2O94.0%-3.40.964
MATH-500OLMo-3-7B-Think97.4%StreamingLLM81.3%-16.10.738

The lowest results from the paper (Still and MEMENTO) were -1.8% reduction in accuracy, so Still and MEMENTO remain the competitive models on this benchmark.

3. Thank you for pointing out; these design decisions were tested in our research process for this paper, but we agree it will be helpful to include these ablations. As we iterated to the final architecture of Still (Fast Diamond) we ablated architectural components and loss functions, even leveraging a toy full-rank matrix to study the optimization landscape of KV cache space.

We found simplification of the architecture to be a strong direction and after removing many pieces of the architecture (self-attention, feed forward network) we ablated training Still as both a global and local KV operator. Below is a table of ablations on the most recent version of Still including: per-layer and per-head parameterizations for compression latents Z and KVO linear maps, and freezing/training compression latents Z and KVO maps:

Z scopeKVO scopeTrain ZTrain KVOParams (M)Accuracy
Layer + headLayerNoV2.3642.3%
Layer + headLayerNoKV4.7249.5%
Layer + headLayerNoKVO7.0854.6%
Layer + headLayerYesKVO16.5259.2%
HeadLayer + headYesKVO56.8956.6%
Layer + headLayer + headYesKVO66.0661.2%

For all rows here, we use Qwen3-4B, 1024→128 compression, 101 updates, and 196 questions. The only better performing configuration was individual per-head KVO maps, but the performance increase comes at 2 points for an increase of more than 4x in trainable parameters.

4. Thank you for your comment. While we did present some latency results in the appendices of the paper we agree that the presented results do not paint the full latency picture for Still. To highlight the benefits of these KV compression methods and provide results that better resemble production deployments, where KV Cache hit rates are important, we present below timing results using compression+decode as the main latency measure.

In post document prefill we measure the time to compress the prefilled cache and then decode 512 tokens across growing input document sizes (8k to 128k). All benchmarked method’s latency improvements grow (ex-H2O) as raw document length increases. Still sees a reduction in time of 31% over raw prefill at a document size of 128k toks. Although Still does not provide the largest latency benefit it does so while maintaining the highest compressed performance of any method we benchmarked.

Method8k / 51232k / 512128k / 512
STILL1.05×1.15×1.45×
LCLM (encoding + decode)1.05×1.13×1.33×
H2O prefill adapter0.92×0.39×0.06×
SnapKV prefill adapter1.06×1.26×2.00×
Sink/recent1.07×1.27×2.10×

We would be happy to answer any further questions you may have; if we have adequately addressed your concerns, we would be grateful if you could please consider raising your score. Thank you again for your review.

Comments on the paper

1 comment
Area_Chair_dpFBOpenReview ↗

Discussion Phase

As the discussion period is rapidly coming to a close (Nov 18 AoE), I would like to remind the reviewers to please participate in discussion and acknowledge any comments posted by the authors. A secondary, important reminder: full disclosure of any AI usage is required and the reviewer must include their own human-written assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.