- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper tackles context compression through soft-token compression, rather than conventional KV-cache compression methods, which typically require processing the full context with the LLM before compression, incurring substantial computational costs. To this end, the authors revisit encoder–decoder architectures, using a lightweight encoder to compress the input context. The encoder processes the input in chunks, and its hidden representations are averaged over fixed-size blocks before being passed to the decoder. The authors develop a detailed training recipe to make this approach effective, including staged training and large-scale continual pretraining. Through extensive experiments using a 0.6B encoder and a 4B decoder, evaluated at compression ratios of 4×, 8×, and 16× on benchmarks including RULER, LongBench, LongHealth, and GSM8K, they demonstrate favorable tradeoffs between compression quality, inference speed, and memory usage. Finally, they explore two interesting applications: an agent that can selectively expand compressed context chunks through tool use, and online compaction of the model's own generated reasoning during inference.
Critical strengths
-
The paper addresses an important problem and revisits a natural and relatively simple approach to context compression, demonstrating that it can be competitive with more commonly used KV-cache compression methods.
-
The experimental evaluation is thorough, covering a broad range of benchmarks, compression ratios, and baselines. The training and model scale are also substantial for an academic study.
-
The additional experiments on agentic retrieval with selective context expansion and online latent compaction are interesting and demonstrate potential applications of the proposed approach beyond standard context compression.
Critical weaknesses
-
The paper discusses several previous amortized context compression approaches, including Gisting, ICAE, AutoCompressor, and Dynamic Memory Compression. However, it overlooks other relevant methods, including KV-Distill, CCM, Activation Beacon, as well as SHINE and Doc-to-LoRA, which amortize context compression by generating compressed KV caches, soft prompts, or context-specific LoRA adapters. Moreover, the paper does not directly compare against any of these methods experimentally. Such comparisons would be valuable for establishing the advantages of the proposed approach over existing amortized context compression methods.
-
The proposed method requires substantial training, making it considerably more expensive to develop and deploy than simpler alternatives such as inference-time KV-cache eviction or text-based summarization. The paper does not sufficiently discuss this additional cost or when it would be justified.
-
The authors train separate models for each compression ratio (4×, 8×, and 16×). This limits the flexibility of the approach, as supporting different compression budgets appears to require additional training rather than simply adjusting the compression ratio at inference time.
-
While the method achieves impressive inference speedups, its compression quality, although competitive, is often not the best among the evaluated methods. In many applications, preserving downstream performance may be more important than minimizing compression time, provided that latency remains acceptable. Thus, although the proposed method establishes new points on the accuracy–efficiency Pareto frontier, it is unclear how valuable these operating points are in practical settings where compression quality is prioritized.
Questions for authors
-
The paper neurips 2026 paper “Training Transformers for KV Cache Compressibility” studies how adapting the decoder during training can improve its amenability to subsequent KV-cache compression. Since the proposed method also updates the decoder, it would be interesting to investigate whether the resulting decoder is more amenable to other compression methods, such as KVzip or Attention Matching. Such an experiment could also help disentangle the contributions of encoder-based compression and decoder adaptation. I encourage the authors to discuss and cite this related work.
-
Could the authors provide more details on how the synthetic memory-expansion trajectories are generated? In particular, how are the relevant chunks selected during trajectory generation, and how is the model supervised to identify which compressed chunks to expand
I am open to raising my score if the authors adequately address the weaknesses and questions raised above.
10 comments
Related amortized compression methods
Thank you for these pointers. The revised related work now cites and discusses KV-Distill, CCM, Activation Beacon, SHINE, and Doc-to-LoRA.
A fair comparison requires retraining each method on the same base model and training data, which is beyond our compute budget. We leave this as future work and have updated the corresponding sections in our manuscript to reflect that. We emphasize that our main contribution is a deep and scaled analysis of soft token compression, in which we compare to a wide range of reasonable and SoTA KV cache compression baselines such as SnapKV, Expected Attention, KVzip, FastKVzip, and Attention Matching. For soft-token compressors, our controlled sweep in Section 5 trains the main design choices from prior work with the same data, scale, and decoder, and Appendix C.1 shows how prior methods map to the choices we sweep.
Training cost
We agree that LCLM has a substantial one-time offline training cost. However, during inference, the cost is much smaller than all other methods as seen in our Pareto frontier curves. While the baselines do not require an offline training stage, they have a recurring cost for every single request: KVzip and FastKVzip prefill the full context every time, and Cartridges reports roughly 30 minutes on an 8×H100 node for LLaMA-8B per corpus. Once a model has served enough contexts, one training run costs much less than paying those per-context costs repeatedly.
The relevant comparison is therefore similar to speculative decoding or model distillation. Training a draft model or distilled serving model can be expensive, but the cost is amortized over a production workload in exchange for recurring inference-time savings, which are much more expensive. Our work focuses on this amortized trade-off between quality, latency, and memory.
Moreover, ideally, LCLM would be incorporated as an extra training stage in a standard LLM pretraining pipeline. Against the compute spent pretraining and post-training, the compute for training the compressor is marginal.
Thank you for your feedback and for recognizing the importance of the problem, the simplicity of the approach, the thoroughness and scale of the evaluation, and the agentic expansion and online compaction applications. We address each concern below.