acceptodds
Under review as a conference paper at ICLR 2027

Do We Need Learned Compressors? Adapting LLMs to Parameter-Free KV-Cache Compression for Long-Context Reasoning

Abstract

Large language models (LLMs) face growing memory costs during extended reasoning because the key-value (KV) cache expands with every generated token. Recent methods address this by training auxiliary modules to produce compressed representations of past reasoning states. We ask whether the compression function itself needs to be learned. We propose a parameter-free alternative that periodically replaces fixed-stride windows of reasoning tokens with a single KV pair obtained by simply averaging their pre-rotary key and value representations. Rather than optimizing a learned compressor, we freeze the base model and train lightweight LoRA adapters to reason directly from these fixed summary vectors. Across several long-context reasoning benchmarks and compression ratios, our approach performs competitively with learned KV-cache compression baseline while adding zero auxiliary parameters for compression. Our results suggest that effective reasoning over compressed history does not necessarily require the compression operator itself to be learned.

Primary area:
foundation or frontier models, including LLMs
Keywords:
long context reasoning, KV cache compression, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_6mM4OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a KV Cache compression method with only lightweight training. The implementation shows that the proposed method achieves state-of-the-art results compared with other methods, with and without training, and also shows the applicability of the method under different compression ratios.

Critical strengths
  1. The proposed method only requires lightweight training, which is helpful for the huge training overhead of current large language models, and also increases the practicality of the method.
  2. This paper shows the effectiveness of the method, but more experiments are needed to further support it.
Critical weaknesses
  1. The paper does not report inference speed or memory-saving improvements, which are essential for evaluating the practical benefits of KV Cache compression.
  2. The comparison method is insufficient. In Section 2, the paper mainly discusses works from 2023 to 2024, without covering more advanced methods or conducting comparisons from the past year. This makes it difficult to highlight the innovation of the methods.
  3. Evaluating only on Qwen2.5-1.5B is insufficient. Further discussion is needed on performance across different architectures, larger parameter sizes, and the more advanced Qwen3 series to highlight the applicability of the method.
  4. The overall quality of the paper is not high. There is much room for improvement in both the writing and layout. Section 3.1 seems to belong in the experimental section.
  5. Further clarification is needed on the contribution of the proposed method, as well as the differences from previous methods.
  6. Authors need to recognize the difference between /cite and /citep in LaTeX writing.
Questions for authors
  1. Could the author be able to present the actual curve records of the training process in the next version? This would be beneficial for enhancing the reliability of the results.
  2. Can the proposed method be evaluated on more datasets or tasks?
  3. Based on the results presented and the overall quality of the writing, I remain skeptical about the reliability of the method. I hope the author can supplement the results with more experimental data; otherwise, I will still consider further lowering the score.
2 comments
AuthorsOpenReview ↗

In summary, we have added a new model and task, R-KV as a baseline, inference and training cost measurements, training curves, an explicit contributions list, and corrected the formatting.

We thank the reviewer for the constructive feedback. Below, we address each point in turn, including new experiments on additional models, tasks, and baselines, inference and training cost measurements, training curves, and revisions to the writing.

Models. We repeat the full pipeline on Phi-4-mini-instruct (3.8B), a model from a different family and at a larger scale (Section 4.3, Table 3). We train a new teacher and distill both BR and our method from the same teacher checkpoint and trajectories. Our method matches or exceeds BR at every compression ratio on both LinSys and StarGraph, with the clearest gains on LinSys at higher compression (0.297 vs. 0.172 at 8×).

Tasks. We add MATH (Hendrycks et al., 2021), an open-ended mathematical reasoning benchmark that is more diverse than our synthetic tasks (Section 4.3, Table 3; setup in Appendix A.1). Our method outperforms BR at four of five compression ratios, with the largest margin at 2× (0.426 vs. 0.340). Together with LinSys and StarGraph, our evaluation now covers structured algebraic deduction, graph search, and open-ended mathematical problem solving.

Comparisons. We have added R-KV (Cai et al., 2025), a recent KV-cache compression method designed specifically for reasoning models, as a training-free baseline (Table 2). R-KV is the strongest training-free baseline, matching our method on LinSys at 2× (0.926), but it collapses beyond 4× (0.086 at 8×) and remains well below the adapted policies on StarGraph (0.598 vs. 0.820 at 2×). We have also expanded Section 2 to cover recent reasoning-specific KV compression methods, including R-KV and ThinKV (Ramachandran et al., 2026). ThinKV has no publicly available implementation at the time of writing, so we could not include it as a baseline.

Training curves. We thank the reviewer for this suggestion. We have added the distillation loss curves for both BR and our method across all compression ratios on LinSys and StarGraph (Appendix A.3). Both methods converge at every ratio, with final loss increasing with the compression ratio, and our method reaches lower final loss, most markedly on LinSys. At higher ratios (c ≥ 8), our method shows transient loss spikes from which training recovers.

Inference speed and memory. We thank the reviewer for this comment. Our original submission reported KV-cache memory savings in Section 4.2 (KV-cache memory scaling): for example, at c = 4 on LinSys, our method reduces the generated cache by 3.99× and the total cache by 3.56×. We have now added measured decoding latency and peak memory for the uncompressed teacher, BR, and our method (Appendix A.4, Table 6). At each window boundary, both compressed methods incur one additional forward step, but the shorter cache makes every other decoding step cheaper. At high compression, this saving dominates: at c = 32, our method decodes at 70.4 tokens/s and BR at 72.3 tokens/s, compared with 66.4 tokens/s for the uncompressed teacher. At lower compression ratios, the boundary overhead outweighs this saving. At the model size and sequence lengths we evaluate, peak memory is dominated by model weights, since the KV cache occupies only about 29 MB per 1,000 tokens. The memory benefit of compression therefore grows with generation length and batch size, where the cache dominates memory use.

Contributions. We thank the reviewer for this comment. Prior learned approaches to reasoning-time KV compression, such as Breadcrumbs Reasoning, train a dedicated module to produce each compressed summary. We ask whether this learned aggregation is necessary. Our method replaces each window of generated KV entries with their mean and adapts the model with lightweight LoRA, adding no new tokens or auxiliary compression parameters. Under a controlled comparison, it matches or outperforms BR (Tables 1–2), and this generally holds across tasks and models (Table 3). Ablations show that both the aggregation rule and adaptation are necessary (Section 4.4).

Writing and formatting. We thank the reviewer for bringing this to our attention. We have corrected the citation formatting and reorganized the paper’s structure.

Thanks to the author for the last-minute reply.

Although the author's reply can address some of my concerns, I think the overall quality of the paper is not up to the acceptance standard, and KV cache compression needs a more efficient comparison to highlight the effect. The results shown in Table 6 are not convincing to me, and the corresponding description is not readable.

We hope that in future submissions and Rebuttal processes, authors can mark the key parts of the revision in blue or other colors.

At the same time, the author can pay more attention to the writing quality in the future. The planning and layout of Figure 2 can be further modified to make the information density higher and the comparison more intuitive. Other typography and readability aspects have room for improvement.

Reviewer_rECyOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes a simple method to train efficient LLM reasoners that periodically compress their KV caches. Each window of generated tokens is replaced by the mean of its keys and values, and LoRA adapters on a frozen base model are distilled from an uncompressed teacher to reason from these averaged entries. The authors argue that the method is parameter-free yet competitive with learned KV compressors, suggesting that the compression operator need not be learned to reason effectively over compressed KV representations. The method is validated with Qwen2.5-1.5B-Instruct on two synthetic reasoning tasks (LinSys and StarGraph), against a reproduced Breadcrumbs Reasoning (BR) baseline and training-free eviction methods.

Critical strengths

None.

Critical weaknesses
  1. The experiments and methodology do not support the main claim of the paper. The paper states that "the compression operator itself never learns". The averaging rule is indeed fixed, but the inputs are not. The entire model operates with learned LoRA weights, and these weights might also learn how to generate the compressed representation. The actual comparison the paper carries out is between uniform averaging and a summary computed by a special token attending to previous representations. The authors also state that "Our method exhibits substantially less degradation on the same task despite using a fixed, parameter-free compressor.", and the abstract describes LoRA adapters trained "to reason directly from these fixed summary vectors". Both claims are incorrect, since the model that produces the KV representations was trained with LoRA weights.

  2. Experiments are extremely limited. The paper evaluates the method on only two datasets, both of which are synthetic tasks. The paper also claims to investigate long-context reasoning, whereas the generation and cache budgets are at most 2,000 tokens (1,000 for LinSys). Additionally, only a single model is used for the experiments. Regarding the training-free baselines, I believe there are some relevant reasoning-KV-compression methods like ThinKV (Ramachandran et al 2026) and R-KV (Cai et al 2025) that should be more competitive than SnapKV. Finally, the reported BR numbers are substantially lower than the published numbers (e.g., 0.066 vs 0.367 on LinSys at 16x), which questions the validity of the experiments. The paper attributes this gap to the teacher model, but its LinSys teacher is stronger than BR's teacher. While on LinSys the reported gains hold even against BR's published numbers, they fall behind on StarGraph.

These weaknesses are significant because they put the central claims of the work in question. While the second weakness could potentially be addressed by more rigorous experiments, the first concerns the essence of the method, and I currently do not see how it can be resolved without reframing the paper's central claim.

Questions for authors

I have listed my main concerns in the weaknesses section. If the authors think I have misunderstood the paper or the experiments, please clarify.

1 comment
AuthorsOpenReview ↗

In summary, we have narrowed our central claim to learned aggregation, added MATH, Phi-4-mini, and R-KV, and reported published BR results alongside our reproduction.

We thank the reviewer for the detailed review. Below, we address each concern in turn, including a narrowed central claim, new experiments with MATH, Phi-4-mini, and R-KV, and a fuller comparison with the published BR results.

Scope of the main claim. We thank the reviewer for raising this point. We agree that while the averaging rule itself has no parameters, its inputs are not fixed; the adapted model both produces the states that are averaged and reasons from the result. Our experiments therefore compare aggregation rules for the reserved slot (a fixed mean versus a learned beacon token’s attention), applied in both cases to learned representations. We have revised the manuscript throughout to reflect this. We removed the description of the summary vectors as “fixed,” and replaced the claim that “the compression operator itself never learns” with a statement that the LoRA adapters learn both how to produce informative representations for averaging and how to interpret the averaged result. We also narrowed our conclusions from “the compression operator need not be learned” to “a learned aggregation mechanism is not necessarily required.”

Datasets. We now include MATH (Hendrycks et al., 2021), an open-ended, non-synthetic mathematical reasoning benchmark (Table 3; setup in Appendix A.1). Our method outperforms BR at four of five compression ratios, with the largest margin at 2× (0.426 vs. 0.340). We also note that MATH is substantially more challenging under compression for both methods, and we discuss this in the Limitations section.

Models. We repeat the full pipeline on Phi-4-mini-instruct (3.8B), a model from a different family and at a larger scale (Table 3). We train a new teacher, distill both BR and our method from the same teacher checkpoint and trajectories, and find that our method matches or exceeds BR at every compression ratio on both LinSys and StarGraph.

Training-free baselines. We have added R-KV (Cai et al., 2025) to our training-free baselines (Table 2). As the reviewer anticipated, R-KV is considerably more competitive than SnapKV: on LinSys it matches our method at 2× (0.926). However, it collapses beyond 4× (0.086 at 8×), and on StarGraph it remains well below the adapted policies (0.598 vs. 0.820 at 2×co). We were unable to include ThinKV (Ramachandran et al., 2026), as no implementation is publicly available at the time of writing.

Gap to published BR results. We thank the reviewer for raising this point. To clarify, we attributed the gap to the training regime rather than the teacher: the published SR BR results use joint RL-distillation, whereas we reproduce the two-step setting so that both methods share the same teacher and distillation data (l.388). The cited 16× example is the largest discrepancy; at the remaining ratios, the gap is broadly in line with the joint vs. two-step differences reported by Monea et al. (2025, Fig. 5). As the original implementation is not public, we reimplemented BR following its reported configuration as closely as possible. For transparency, we have added the published SR BR results at all compression ratios and on both tasks to Tables 1 and 2, clearly marked as reported results. Training curves for our BR reproduction (Appendix A.3) show smooth convergence at all compression ratios, with final loss increasing monotonically with the compression ratio.

Reviewer_YVrVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents a reasoning-time KV-cache compression method that uses fixed mean pooling instead of a learned aggregation operator. It replaces each window of generated-token KV states with a single layer-wise KV pair and trains LoRA adapters through distillation from an uncompressed teacher to enable reasoning with the compressed cache. Experiments with Qwen2.5-1.5B-Instruct on LinSys and StarGraph show substantially higher accuracy than the reproduced Breadcrumbs Reasoning (BR) baseline on LinSys and broadly comparable performance on StarGraph, under both fixed generation-cache and fixed generation-length budgets.

Critical strengths
  1. Simple compression with potential training-cost savings. The method replaces learned beacon representations with fixed mean pooling and uses LoRA for adaptation, without introducing a new vocabulary token or an auxiliary trainable aggregation module. Adaptation can also be restricted to the MLP gate projections: on LinSys at compression ratio 4, gate-only LoRA achieves 85.6% accuracy versus 90.6% for adaptation across all attention and MLP projections (Table 3). This provides a training-cost/accuracy trade-off by reducing the number of trainable adapter parameters and their associated gradient and optimizer-state storage. The potential savings are appealing, although training-time and peak-memory reductions are not quantified.
  2. An informative answer to whether the compression operator must be learned. Within the evaluated settings, the paper provides evidence that effective reasoning under KV-cache compression does not necessarily require a learned aggregation operator. This is a useful conceptual contribution: it challenges the necessity of increasing compressor complexity and highlights adapting the model to a fixed compression rule as an alternative design direction.
Critical weaknesses
  1. Training and inference costs are insufficiently characterized. Parameter-free aggregation still requires LoRA distillation, yet its training time and peak training memory are not compared with BR. Although KV-cache memory reductions are reported, end-to-end inference latency and throughput are not quantified. It is also unclear whether adapters are shared across tasks and compression ratios or trained separately for each setting. These omissions limit assessment of the practical cost-accuracy trade-off.
  2. The contribution of self-distillation is not isolated. The authors use self-distillation to adapt the compressed model, but do not isolate its contribution to performance relative to the uncompressed teacher. Comparing against an uncompressed student with matched initialization, LoRA settings, data, and self-distillation budget would help separate the effects of self-distillation and KV compression.
  3. Generalization beyond the evaluated tasks is unclear. Training and evaluation are confined to two synthetic tasks, with no reported transfer to unseen tasks or larger problem scales. It therefore remains unclear whether the learned adaptation supports broader reasoning workloads without retraining.
Questions for authors
  1. Adapter sharing and efficiency: Are separate LoRA adapters trained for each task and compression ratio, or is one adapter shared? Please report adaptation time and peak training memory relative to BR, and inference latency and throughput relative to BR and the uncompressed model under matched generation budgets, including the boundary-processing overhead.
  2. Contribution of self-distillation: How does an uncompressed student trained with the same initialization, LoRA settings, data, and self-distillation budget perform relative to the teacher and compressed student?
  3. Generalization: Is there evidence that an adapter trained on these tasks transfers to unseen reasoning tasks or larger problem scales without retraining?
1 comment
AuthorsOpenReview ↗

In summary, we have added training and inference cost comparisons, an uncompressed self-distillation control, and clarified that adapters are trained per task and ratio.

We thank the reviewer for the thoughtful review and for recognizing the contribution of our approach. Below, we address each question in turn, including new training and inference cost comparisons, an uncompressed self-distillation control, and clarification of adapter sharing.

Adapter sharing. Adapters are trained separately for each task and compression ratio, following the single-ratio BR setting against which we compare. We have revised Section 4.1 to state this explicitly.

Training and inference cost. We have added both comparisons (Appendix A.4). For training, our method reduces peak memory relative to BR by 1.23× on Qwen2.5-1.5B and 1.95× on Phi-4-mini, and reduces time per step by 15% and 20%, respectively, as it trains only LoRA adapters while BR uses full fine-tuning. For inference under matched generation budgets, a normal decoding step costs 15.05 ms for the uncompressed teacher. With compression, normal steps become cheaper as the cache shrinks (13.40 ms for BR and 13.71 ms for ours at c = 32). Both compressed methods incur one additional forward step at each window boundary, which occurs once every c tokens: the boundary-processing overhead is 13.7 ms for BR and 15.7 ms for ours at c = 32, and 15.8 ms and 16.9 ms at c = 4.

Contribution of self-distillation. We thank the reviewer for suggesting this control. We trained an uncompressed student on LinSys with the same initialization, LoRA configuration, distillation data, and training budget as our compressed policies. It reaches 94.5% accuracy, compared with 93.4% for the teacher, indicating that self-distillation alone provides only a small gain. Relative to this uncompressed student, our method loses 1.5 points at 2× and 3.9 points at 4× (Table 1), so the accuracy of our compressed policies mainly reflects the cost of compression.

Generalization. We thank the reviewer for this question. In our setting, the base model cannot solve these tasks reliably on its own, so each teacher is first trained on its task with RL, and each compressed policy is distilled from that teacher. The adapters are therefore task-specific by design, and we do not evaluate transfer to unseen tasks or larger problem scales without retraining. Instead, we show that the method itself generalizes: applying the same recipe to a new model family (Phi-4-mini-instruct) and a new task (MATH), our method matches or outperforms BR in most settings (Section 4.3, Table 3).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.