acceptodds
Under review as a conference paper at ICLR 2027

SIDEKV: SIDE-INFORMATION-AWARE CONTEXT COMPRESSION FOR KV CACHE

Abstract

Long-context inference in large language models (LLMs) incurs substantial memory and computation through the key–value (KV) cache. Learned context compression replaces a long historical block with a compact representation, but existing approaches typically construct this representation from the discarded context alone, despite correlated context remaining available to the predictor. We introduce SideKV, a side-information-aware context-compression framework that conditions the rate of a learned summary on the retained context. A lightweight encoder compresses the discarded block into summary vectors, while a conditional rate model exploits the retained context as side information; the pretrained LLM remains frozen. Across 13 LLMs, with 320 of 1312 cache slots retained, conditioning on side information reduces the variational rate to – of that of an otherwise identical unconditional model and lowers excess predictive distortion on 11 of the 13 LLMs. The distortion advantage also persists across additional corpora, rate–distortion settings, and correlated time series. Among learned compressed representations, SideKV substantially improves over mean pooling while avoiding adaptation of the pretrained predictor, and achieves a mean excess distortion of bits/token. It requires only full-context prefill arithmetic and time to first token at the main setting. On LongBench, SideKV also achieves the highest mean score among the learned compressed representations evaluated. These results show that exploiting retained context as side information is a promising direction for lightweight KV-cache compression.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV cache compression, context compression, long-context inference, rate–distortion, side information, frozen large language models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_n6ppOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SideKV, a KV cache compression method that builds on the intuition that the compressed context does not need to contain the information that remaining (initial and final) parts already have. The paper proposes a new training objective to train the encoder with this capability. A thorough evaluation is provided, spanning over a large number of models and compression approaches.

Critical strengths

This paper provides a well-motivated and novel approach, that allows compressed soft inputs to focus on modeling the parts that remaining parts do not already have. The experiments are extremely broad and thorough, spanning a wide range of models and compression approaches. The resulting approach enables significant speedup compared to full kv cache setup.

Critical weaknesses

The core weakness is practical value. Most evaluations rely on continuation negative log-likelihood (NLL) measured on pretraining corpora, which may not necessarily translate to practical downstream performance. The paper provides evaluation on LongBench in Table 4, but the performance drop compared to the full baseline is significant (dropping from 30.9 to 18.8 on average), making practical applicability of the approach questionable.

I acknowledge that the performance is better than the baselines compared in Table 4, and much faster than other high-performant baselines in Table 14, but the overall performance drop would be unacceptable in practical terms. The large drop in some of other baselines’ performance may also suggest that the specific experiment setup causes a large universal drop in performance (e.g. too aggressive compression ratio), making the comparison itself less valuable. It would be useful to know if the performance gain would also translate to more practical evaluation regimes (e.g., with modest compression ratio).

Questions for authors

Please see critical weakness.

Reviewer_wmo7OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SIDEKV studies learned KV-cache compression from a side-information perspective. The context is split into a retained prefix S, a discarded block X, and a retained recent window W; a learned encoder maps X into summary vectors Z, fed to a frozen LLM between S and W. The key idea is to condition the rate penalty on the retained context Y = (S, W), giving a variational rate surrogate that decomposes into conditional mutual information plus a mismatch term. Evaluated on 13 frozen LLMs plus robustness studies across corpora, budgets, and reuse.

Critical strengths

The idea is clean and the theory is unusually honest about what the "rate" is and isn't. The side-information ablation is well isolated — same encoder, data, and optimization, differing only in the rate model — and the effect is consistent across LLMs and corpora.

Critical weaknesses

The rate savings are not storage savings: there's no quantizer or coder, so the cache is still k fixed vectors regardless of the measured rate, and the rate model is used only at training time. No error bars anywhere, which matters for several small-margin conclusions. The ablation is confounded by parameter count (the conditional model has ~2× the parameters), and on LongBench the downstream gap to selection and quantization remains large (18.8 vs. 23.5 for ReST-KV, 30.8 for 4-bit KIVI vs. 30.9 full context).

Questions for authors

Can you report error bars for the key comparisons? Can you add a parameter-matched unconditional baseline? What is the actual byte footprint of storing Z?

Reviewer_GGTHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies context compression for the KV cache. It splits a long context into a retained prefix S, a discarded historical block X, and a retained recent window W. A lightweight encoder compresses X into k Gaussian summary vectors Z. A conditional rate model r_psi takes Y = S, W as side information and provides a prior for Z. The variational rate surrogate is the KL divergence between the encoder distribution q_phi and r_psi. The training objective combines this rate term with the predictive loss, and only the encoder and the rate model are trained while the pretrained LLM stays frozen. At inference, Z is projected to the hidden size, rescaled, and inserted as input-layer tokens in the sequence S, Z, W. It is not written into the per-layer KV cache.

The experiments cover 13 models from 0.5B to 16B parameters, including dense, mixture-of-experts, latent-attention, and interleaved local-global architectures. The main corpus is SlimPajama, with additional results on WikiText, PG19, five public-domain novels, Python source, and two correlated time series. The main setting retains 320 cache slots out of 1312. Conditioning on the retained context reduces the variational rate to 15% to 67% of the unconditional model and lowers excess predictive distortion on 11 of the 13 models. Mean excess distortion is 0.050 bits/token. Prefill arithmetic is 0.25 times that of the full context, and TTFT is 0.30 times. On LongBench, the mean score is 18.8, above MeanPool at 15.8 and 500xCompressor at 14.7, but below the full context at 30.9.

Critical strengths

The controlled comparison is well designed. SideKV and the conditional-prior baseline share the encoder, the latent dimension, the frozen LLM, the data, and the optimization procedure, and differ only in the rate model. This isolates the effect of the retained context instead of mixing several changes together. The 13 models span dense, mixture-of-experts, latent-attention, and interleaved local-global architectures, so the result is not an artifact of one model. This is the most convincing part of the paper.

The implementation is lightweight, and the empirical study is broad and honest. The method adds only 18.3M parameters, 0.12% of Qwen2.5-14B, keeps the LLM frozen, and uses one quarter of the full-context prefill arithmetic. The authors also add a matched cache-budget control. Table 19 raises eviction to 320 and 352 slots, which rules out the explanation that SideKV wins only because it uses 32 more slots. These controls make the empirical section useful even if the framing is questioned.

Critical weaknesses

First, the rate is not an operational compression rate, so the central promise is not delivered. There is no quantizer, entropy coder, or bitstream. k is fixed at 32, and a lower Rvar does not reduce cache slots, memory, or latency. r_psi is discarded at inference, Y never enters the deployed model, and the side information is only a training-time conditional prior. In other words, this is a regularization technique, not a compression scheme. The title and abstract frame it as side-information-aware compression, which leads readers to expect an actual bit rate or memory gain. Table 13 reports peak-memory reductions of only 1.10 times at the main setting and 1.87 times at a 4096-token block, at batch 1 and for a single forward pass. For a KV-cache compression paper, this is the key number, and it is modest. At batch 1, weights dominate memory, and the measurement does not cover the long-context and high-concurrency regime where the KV cache actually dominates.

Second, the main evaluation metric does not support the conclusion. Excess distortion is measured on a continuation that follows an intact 256-token recent window. The model already sees the immediate context, so the discarded middle block has limited influence on next-token prediction. The 0.050 bits/token figure is therefore optimistic. Many cells in Tables 1 and 9 differ by very little, and the paper reports no random seeds, variance, or confidence intervals. Negative excess distortion appears in several places. The authors explain it as compression occasionally beating the full context, but it looks more like positional shift or measurement noise than a stable phenomenon. When the discarded block actually matters, SideKV scores 18.8 on LongBench, against 30.9 for the full context and 23.5 for ReST-KV. Table 18 also shows that at M = 8192 without retraining, SideKV degrades to 0.906 while selection methods remain at 0.074. The method is sensitive to the block length used in training, and its practical scope is narrower than the abstract suggests.

Questions for authors
  1. Can you provide a rate-distortion curve with k or lambda on the horizontal axis and distortion on the vertical axis, showing SideKV, the unconditional variant, and eviction? If SideKV cannot reach the same distortion with fewer slots, what does a lower Rvar mean in deployment?
  2. At inference both X and Y are available. Why must the encoder see only X? Please add a baseline whose encoder sees both X and Y. If the jointly conditioned encoder is better, the Wyner-Ziv style encoder constraint is a self-imposed restriction rather than a necessary design.
  3. How are positional encodings assigned to the summary slots and to the window W? Does W keep the same positions as in the full context? Can positional shift explain part of the negative excess distortion and the sensitivity of the Gemma-3 models?
  4. Are the conclusions of Tables 1 and 9 stable across training seeds? Can you report means, standard deviations, and paired tests? Is a difference such as +0.009 bits/token above noise?
  5. Table 13 measures batch 1 and a single forward pass. Can you report peak memory at batch 8 or 16, at contexts above 32K, and during autoregressive decoding? What share is due to encoder activations?
  6. On LongBench, the full-context score for Qwen3-8B is only 8.1. Is this setup correct? Is it appropriate to average metrics from 16 heterogeneous tasks directly? Can you report results by task category?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.