acceptodds
Under review as a conference paper at ICLR 2027

End-to-End Context Compression at Scale

Abstract

Long-context language model inference is bottlenecked by memory, as the KV cache grows with context length. Recent techniques to compress the KV cache fall short: they either degrade model quality substantially or require considerable time and compute to compress a single long prompt. Furthermore, many methods require the input to fit within the target model's context window, and are generally incompatible with modern production inference engines. Encoder-decoder compressors, which map a long token sequence to a shorter sequence of latent embeddings consumed by a decoder, are an appealing alternative in principle. However, existing approaches are not competitive with KV cache compression on the accuracy-efficiency frontier. In this work, we revisit encoder-decoder compression and close this gap. We first perform an architecture search, pre-training many variants from scratch to determine how best to design and train encoder-decoder compressors. Guided by our findings, we continually pre-train a family of 0.6B-encoder, 4B-decoder models on nearly 400B tokens each, at compression ratios of 1:4, 1:8, and 1:16. We introduce Latent Context Language Models (LCLMs), a family of compressors that improve the Pareto frontier across general-task performance, compression speed, and peak memory usage. LCLMs also unlock new efficient downstream applications: they serve as backbones for long-horizon agents that skim a compressed long context and adaptively expand relevant segments on demand, and they can compact the model's own generated reasoning into latent memory during inference, which is nearly lossless on the full MATH test set while being highly efficient, reducing peak KV usage by up to 7.9x.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Compression, End-to-End, Encoder-Decoder

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_tiCEOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper tackles context compression through soft-token compression, rather than conventional KV-cache compression methods, which typically require processing the full context with the LLM before compression, incurring substantial computational costs. To this end, the authors revisit encoder–decoder architectures, using a lightweight encoder to compress the input context. The encoder processes the input in chunks, and its hidden representations are averaged over fixed-size blocks before being passed to the decoder. The authors develop a detailed training recipe to make this approach effective, including staged training and large-scale continual pretraining. Through extensive experiments using a 0.6B encoder and a 4B decoder, evaluated at compression ratios of 4×, 8×, and 16× on benchmarks including RULER, LongBench, LongHealth, and GSM8K, they demonstrate favorable tradeoffs between compression quality, inference speed, and memory usage. Finally, they explore two interesting applications: an agent that can selectively expand compressed context chunks through tool use, and online compaction of the model's own generated reasoning during inference.

Critical strengths
  1. The paper addresses an important problem and revisits a natural and relatively simple approach to context compression, demonstrating that it can be competitive with more commonly used KV-cache compression methods.

  2. The experimental evaluation is thorough, covering a broad range of benchmarks, compression ratios, and baselines. The training and model scale are also substantial for an academic study.

  3. The additional experiments on agentic retrieval with selective context expansion and online latent compaction are interesting and demonstrate potential applications of the proposed approach beyond standard context compression.

Critical weaknesses
  1. The paper discusses several previous amortized context compression approaches, including Gisting, ICAE, AutoCompressor, and Dynamic Memory Compression. However, it overlooks other relevant methods, including KV-Distill, CCM, Activation Beacon, as well as SHINE and Doc-to-LoRA, which amortize context compression by generating compressed KV caches, soft prompts, or context-specific LoRA adapters. Moreover, the paper does not directly compare against any of these methods experimentally. Such comparisons would be valuable for establishing the advantages of the proposed approach over existing amortized context compression methods.

  2. The proposed method requires substantial training, making it considerably more expensive to develop and deploy than simpler alternatives such as inference-time KV-cache eviction or text-based summarization. The paper does not sufficiently discuss this additional cost or when it would be justified.

  3. The authors train separate models for each compression ratio (4×, 8×, and 16×). This limits the flexibility of the approach, as supporting different compression budgets appears to require additional training rather than simply adjusting the compression ratio at inference time.

  4. While the method achieves impressive inference speedups, its compression quality, although competitive, is often not the best among the evaluated methods. In many applications, preserving downstream performance may be more important than minimizing compression time, provided that latency remains acceptable. Thus, although the proposed method establishes new points on the accuracy–efficiency Pareto frontier, it is unclear how valuable these operating points are in practical settings where compression quality is prioritized.

Questions for authors
  1. The paper neurips 2026 paper “Training Transformers for KV Cache Compressibility” studies how adapting the decoder during training can improve its amenability to subsequent KV-cache compression. Since the proposed method also updates the decoder, it would be interesting to investigate whether the resulting decoder is more amenable to other compression methods, such as KVzip or Attention Matching. Such an experiment could also help disentangle the contributions of encoder-based compression and decoder adaptation. I encourage the authors to discuss and cite this related work.

  2. Could the authors provide more details on how the synthetic memory-expansion trajectories are generated? In particular, how are the relevant chunks selected during trajectory generation, and how is the model supervised to identify which compressed chunks to expand

I am open to raising my score if the authors adequately address the weaknesses and questions raised above.

10 comments

Thank you for your feedback and for recognizing the importance of the problem, the simplicity of the approach, the thoroughness and scale of the evaluation, and the agentic expansion and online compaction applications. We address each concern below.

AuthorsOpenReview ↗

Related amortized compression methods

Thank you for these pointers. The revised related work now cites and discusses KV-Distill, CCM, Activation Beacon, SHINE, and Doc-to-LoRA.

A fair comparison requires retraining each method on the same base model and training data, which is beyond our compute budget. We leave this as future work and have updated the corresponding sections in our manuscript to reflect that. We emphasize that our main contribution is a deep and scaled analysis of soft token compression, in which we compare to a wide range of reasonable and SoTA KV cache compression baselines such as SnapKV, Expected Attention, KVzip, FastKVzip, and Attention Matching. For soft-token compressors, our controlled sweep in Section 5 trains the main design choices from prior work with the same data, scale, and decoder, and Appendix C.1 shows how prior methods map to the choices we sweep.

AuthorsOpenReview ↗

Training cost

We agree that LCLM has a substantial one-time offline training cost. However, during inference, the cost is much smaller than all other methods as seen in our Pareto frontier curves. While the baselines do not require an offline training stage, they have a recurring cost for every single request: KVzip and FastKVzip prefill the full context every time, and Cartridges reports roughly 30 minutes on an 8×H100 node for LLaMA-8B per corpus. Once a model has served enough contexts, one training run costs much less than paying those per-context costs repeatedly.

The relevant comparison is therefore similar to speculative decoding or model distillation. Training a draft model or distilled serving model can be expensive, but the cost is amortized over a production workload in exchange for recurring inference-time savings, which are much more expensive. Our work focuses on this amortized trade-off between quality, latency, and memory.

Moreover, ideally, LCLM would be incorporated as an extra training stage in a standard LLM pretraining pipeline. Against the compute spent pretraining and post-training, the compute for training the compressor is marginal.

Reviewer_d79HOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces Latent Context Language Models (LCLMs), a family of encoder-decoder models designed for end-to-end soft-token context compression. The authors conduct a systematic architecture search (sweeping pooling operators, window sizes, attention masks, adapters, and initialization) by pre-training variants from scratch on 38B tokens. Guided by these findings, they continually pre-train a family of models (0.6B encoder, 4B decoder based on Qwen3) on ~400B tokens at compression ratios of 1:4, 1:8, and 1:16 using a multi-stage training recipe. They evaluate LCLMs against various KV cache compression baselines on RULER, LongBench, LongHealth, and GSM8K, demonstrating that LCLMs achieve a new Pareto frontier in the accuracy vs. time-to-first-token (TTFT) and peak memory trade-offs. Furthermore, the paper explores two downstream applications: an agentic system where the model can selectively expand compressed chunks via tool calls to improve Needle-in-a-Haystack retrieval, and "online latent compaction," which compresses the model's own generated Chain-of-Thought during inference to save KV cache memory with minimal performance degradation on MATH.

Critical strengths
  1. Rigorous, controlled architectural search isolating design choices. The paper conducts a comprehensive ablation study (Section 5) by pre-training many encoder-decoder variants from scratch under matched data, scale, and decoder conditions. This effectively disentangles confounding variables present in prior work (e.g., different base models, data mixtures, or training scales).

  2. Strong empirical Pareto improvements combined with novel, practical applications. LCLMs demonstrably establish a new Pareto frontier over existing KV cache compression methods in terms of TTFT and peak GPU memory, particularly at high compression ratios (16x) and long contexts (up to 1M tokens). Crucially, this efficiency does not come at the cost of task-agnostic quality, as shown by strong retention on GSM8K and RULER.

Critical weaknesses
  1. Unfair systems-level comparison due to unoptimized baseline implementations. The core claim of establishing a new Pareto frontier relies heavily on TTFT and memory comparisons (Figures 1 and 3). However, Section 6 explicitly notes that LCLM measurements are taken using Hugging Face Transformers with batched encoder passes, but acknowledges that "prefill and generation can be further accelerated with standard inference frameworks such as vLLM... Therefore, the throughput measurements reported here are conservative." Conversely, the KV cache baselines (e.g., SnapKV, KVzip) are run via the kvpress library, also seemingly without custom optimized kernels. This weakness is consequential because the primary selling point of the paper is systems-level efficiency, yet the benchmarking methodology lacks the rigorous, optimized serving infrastructure required to make definitive wall-clock claims.
Questions for authors
  1. Baseline Optimization: Were any of the KV cache baselines (SnapKV, KVzip, etc.) evaluated using optimized inference engines like vLLM or SGLang with custom kernels? If not, how do you justify the "8.8x faster" claim when the baselines are running in unoptimized HuggingFace Transformers, given that eviction strategies can be highly optimized in CUDA?
  2. Training Cost: Could you clarify the total compute budget (e.g., in GPU-hours) required for the 4-stage training recipe of the scaled models?
4 comments

Thank you for your feedback and for recognizing the rigorous architecture search, the Pareto frontier curves, and the new practical applications. We address each of your points below, and we report new experiments run during the discussion period inspired by your feedback.

AuthorsOpenReview ↗

Baseline comparison and optimization

Thank you for raising this important point. First of all, we want to clarify that all methods in Figures 1 and 3, including our own decoder, run in the same framework: Hugging Face Transformers on one H200. The 8.8× is therefore an apples-to-apples comparison, not an optimized system against unoptimized baselines.

On the other hand, none of the KV cache baselines has a vLLM or SGLang implementation or released custom kernels, thus we have to run them through the KVPress library. Writing new serving kernels for each baseline is outside the scope of this paper. Moreover, even optimized KV cache compression methods must first prefill the full context with the base LLM and then evict or compress the tokens. So the TTFT of these methods is lower-bounded by the base model's full-context prefill time.

To show the speedup, we ran the TTFT comparison on vLLM. On the same H200 device, we measured the TTFT of the base model with the full context, which is the lower bound for any KV cache compression method, and the TTFT of LCLM, which includes the encoder pass and the decoder prefill over the latents.

ContextBase modelLCLM 4×LCLM 8×LCLM 16×
4k55.4 ms26.9 ms (2.1×)20.7 ms (2.7×)18.5 ms (3.0×)
8k123.1 ms48.3 ms (2.5×)36.3 ms (3.4×)30.1 ms (4.1×)
16k308.9 ms94.5 ms (3.3×)66.5 ms (4.6×)54.9 ms (5.6×)
64k2,782 ms469 ms (5.9×)279 ms (10.0×)214 ms (13.0×)

LCLM is 2.1–13.0× faster than the lower bound, i.e., the default prefilling time. We use the same H200 as in our other setup, with BF16, batch size 1, CUDA graphs on, and prefix caching off, and we do 30 warmup runs and report the fastest of 20 runs. This information is now in Table 38 (Appendix F.2), and Section 6 states that all speedups over baselines are measured within the same framework.

AuthorsOpenReview ↗

Training cost

We train our models on 16 nodes of H200 GPUs. The total number of tokens is 396.5B over the four stages (38.83B / 77.65B / 182.51B / 97.54B). We report the GPU setup in Appendix F.1 and the per-stage token counts in Table 5 and Appendix F.1.

We agree that LCLM has a substantial one-time offline training cost. However, during inference, the cost is much smaller than all other methods as seen in our Pareto frontier curves. While the baselines do not require an offline training stage, they have a recurring cost for every single request: KVzip and FastKVzip prefill the full context every time, and Cartridges reports roughly 30 minutes on an 8×H100 node for LLaMA-8B per corpus. Once a model has served enough contexts, one training run costs much less than paying those per-context costs repeatedly.

The relevant comparison is therefore similar to speculative decoding or model distillation. Training a draft model or distilled serving model can be expensive, but the cost is amortized over a production workload in exchange for recurring inference-time savings, which are much more expensive. Our work focuses on this amortized trade-off between quality, latency, and memory.

Moreover, ideally, LCLM would be incorporated as an extra training stage in a standard LLM pretraining pipeline. Against the compute spent pretraining and post-training, the compute for training the compressor is marginal.

Thanks again for your feedback. We made a strong effort to address all your points, including edits to our draft and additional evaluations, and we would greatly appreciate it if you would consider increasing your score accordingly. Do you have any other questions?

Reviewer_a92YOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies learned context compression through a small encoder and an adapted language-model decoder. A controlled architecture sweep motivates causal encoding over 1,024-token windows, mean pooling, and an MLP adapter. A staged training pipeline then produces 0.6B-encoder/4B-decoder models at 4x, 8x, and 16x compression. The resulting models offer a useful accuracy-prefill-cost tradeoff against several KV-cache compression methods. Two additional applications explore retrieving raw text from compressed context and compacting generated reasoning online.

Critical strengths
  1. The systems motivation is compelling: compressing before decoder prefill avoids materializing the full decoder cache. Figures 1 and 3 and Section 6 provide evidence of lower TTFT and peak memory under a common single-H200, Transformers-based protocol, including encoder memory. At 4x compression, Table 9 reports 46.04 on LongBench EN16 versus 45.21 for the original decoder, with RULER 4k accuracy remaining close to the original model. This is a practically useful operating point.
  2. The paper goes beyond a single architecture proposal. The small-scale controlled sweep, full-scale comparisons, detailed data mixtures, and training schedule provide useful empirical guidance. The compression-disabled evaluations in Table 6 are particularly helpful for distinguishing some decoder drift from compression effects. The work's contribution lies in the training recipe and demonstrated operating points, rather than a fundamentally new encoder-decoder construction.
  3. The experiments cover synthetic retrieval, diverse LongBench tasks, clinical-document QA, and dense mathematical prompts. The online-compaction result on all 5,000 MATH test problems is also promising: accuracy changes from 92.46% to 90.52%, while average per-problem peak decoder context falls by 3.6x (Figure 5).
Critical weaknesses
  1. The relationship between the main results and architectural ablations is unclear. Table 9 (p. 33) gives the 8x mean-pooling model RULER 4k/8k/16k scores of 89.19/88.06/85.76, whereas Table 23 (p. 39) gives 85.42/84.48/82.47. The concat row is effectively unchanged, so the apparent ranking of mean versus concat reverses. Related discrepancies occur at 4x and 16x. These may represent different checkpoints or evaluation protocols, but their relationship is not identified. This matters for interpreting both the chosen architecture and the reported frontier.
  2. The demonstrated quality range is narrower than the broad capability-preservation narrative. Figure 3 measures resource scaling to 1M tokens, but RULER quality is evaluated only at 4k-16k (Table 8), with other main evaluations around 64k. Even at 16k, 16x compression gives 67.04 versus 93.74 for the original model, and LongBench CN5 falls from 46.63 to 31.38 (Table 9). These results still support a valuable tradeoff, especially at 4x, but they do not establish robust quality at the longest advertised resource settings.
  3. The agentic expansion experiment does not yet establish efficient selective retrieval. Section 7 uses 1,024-token segments, while Figure 4 permits 16/16/32 expansion calls at 4k/8k/16k, enough to expose the entire context. Table 36 reports accuracy without realized expansion counts, expanded-token budgets, or latency. Thus the gain could arise from broad decompression rather than identifying a small relevant subset. Similarly, Section 8 explicitly measures active decoder positions rather than allocated memory; its 7.9x maximum reduction should not be interpreted as an end-to-end GPU-memory reduction.
Questions for authors
  1. Which checkpoints and evaluation settings produce Tables 9 and 23? Please provide a compact mapping of training recipe, checkpoint selection, prompts, and decoding settings, and reconcile the mean-versus-concat ranking.
  2. Can the authors add a limited extension of the existing RULER evaluation to longer contexts, such as 64k and 128k, or explicitly restrict the quality-preservation claim to the evaluated range and compression settings?
  3. How many distinct segments are actually expanded per example, and what are the resulting total tokens and latency? A comparison against sequential or random expansion under the same small token budget, preferably on the five tasks not used for budget selection, would clarify whether the latent context enables selective retrieval.
7 comments

Thank you for your thoughtful review and for recognizing the motivation of improving TTFT by compressing tokens before decoder prefill, the value of the controlled sweep, training recipe, and comprehensive evaluations, and the breadth of our experiments, including agentic expansion and online compaction. We address each of your concerns below, and we report new experiments run during the discussion period to address your feedback.

AuthorsOpenReview ↗

Relationship between Table 9 and Table 23

We thank the reviewer for flagging the discrepancy. The two tables use two different SFT (stage 3) checkpoints. Table 23 was trained with an earlier version of our SFT data. The Mean pooling rows of Table 9 use an improved SFT dataset (Appendix B.4). Both mean pooling models start from the same stage-2 continual pre-training checkpoints. Given our compute budget, we only reran stage 3 for the mean pooling models. The concat model was not retrained, so its rows in Table 9 and Table 23 are the same checkpoints.

In Table 23, both pooling methods use the earlier SFT data, so it is a fair comparison between concatenation and mean pooling. In Table 9, mean pooling uses the improved SFT data while concatenation still uses the earlier data, so Table 9 should not be read as a comparison between pooling methods. The evaluations are conducted in the same way for every row in both tables.

We show mean pooling with the earlier and the improved SFT data in the table below (RULER 4k / 8k / 16k):

Mean poolingEarlier SFT data (Table 23)Improved SFT data (Table 9)
4×91.76 / 91.03 / 89.9693.44 / 92.49 / 90.93
8×85.42 / 84.48 / 82.4789.19 / 88.06 / 85.76
16×75.06 / 70.23 / 65.9177.01 / 72.10 / 67.04

The improved SFT data makes mean pooling 1.0–3.8 points better at every compression ratio and context length. Based on your feedback, we have changed the captions of Tables 9 and 23 to now state which SFT data each row uses.

AuthorsOpenReview ↗

Quality at longer contexts

Following your suggestion, we extended the RULER evaluation to 32k, 64k, and 128k during the rebuttal period. In parentheses, we show the retention rate of LCLM's accuracies compared to the full-context base model.

Model4k8k16k32k64k128k
Qwen3-4B-Instruct-2507 (full context)94.4193.5893.7492.6788.9579.21
LCLM 4×93.44 (99%)92.49 (99%)90.93 (97%)88.85 (96%)83.16 (93%)75.37 (95%)
LCLM 8×89.19 (94%)88.06 (94%)85.76 (91%)80.55 (87%)72.92 (82%)64.82 (82%)
LCLM 16×77.01 (82%)72.10 (77%)67.04 (72%)61.74 (67%)55.71 (63%)51.04 (64%)

Similar to the previous timing evaluation metric, we also measured TTFT in vLLM on one H200 (in ms) for those context lengths. In parentheses, we show the speedup over the base model with the full context.

Model4k8k16k32k64k128k
Qwen3-4B-Instruct-2507 (full context)55.4123.1308.9875.32,7829,734
LCLM 4×26.9 (2.1×)48.3 (2.5×)94.5 (3.3×)202.4 (4.3×)469 (5.9×)1,207 (8.1×)
LCLM 8×20.7 (2.7×)36.3 (3.4×)66.5 (4.6×)133.1 (6.6×)279 (10.0×)634 (15.4×)
LCLM 16×18.5 (3.0×)30.1 (4.1×)54.9 (5.6×)106.5 (8.2×)214 (13.0×)444 (21.9×)

We see that LCLMs sweep a frontier, with 4× keeping 93–99% of the full-context accuracy at every length up to 128k while TTFT is 2.1–8.1× lower, and 16× retaining 63–82% of the accuracy for a 3–22× lower TTFT. To address your feedback, we have added these new results in Appendix F.3 (Table 39), and have refined our exact claims to be in line with the evaluations we run: the paper now says that accuracy is evaluated up to 128k, and that Figure 3 measures only time and memory up to 1M tokens.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.