acceptodds
Under review as a conference paper at ICLR 2027

BellowsKV: One Latent Cache for Many Layers, Expanded on Demand

Abstract

Large language models (LLMs) are now widely utilized as the backbone of agents that reason over many turns, call tools, and pursue long-horizon tasks. To serve such a context without recomputing the past, a model stores the attention keys and values of all past tokens in a key-value (KV) cache. Each layer maintains its own cache and re-reads it at every generation step, so KV-cache memory can become the main bottleneck for GPU serving. Most compression methods operate within a single layer, by quantizing the cache entries, reducing the channel dimension, or evicting less important tokens from the cache. All three lose substantial accuracy at high compression ratios. A more recent line of work shares one compressed cache across adjacent layers to go beyond this limit. However, existing cross-layer methods also degrade sharply at high compression ratios. We find that the KV cache representations of adjacent layers are quite different, whereas the input hidden states are highly similar. We also find that, under the same memory budget, reducing the dimension of this shared latent loses far more accuracy than quantizing it. Based on these findings, we propose BellowsKV, a training-free method that maps the hidden states of n adjacent layers to one shared latent per token. At inference time, each layer reads its own keys and values from this latent through a layer-specific projection. We keep the latent at full dimension and compress it in the two other ways instead: BellowsKV-Quant stores the latent at low precision, which is nearly lossless because the latent has no dominant outlier channels; BellowsKV-Evict makes one eviction decision per block, which the layers can share because they rank token importance very similarly. Across Llama-3.1-8B, Mistral-7B and Qwen2.5-7B, BellowsKV-Quant retains about 99% of full-cache accuracy at 3.6 compression and 97–98% at 13–14, where the strongest existing cross-layer method retains at most 84% at 16.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
LLM, KVCache, Quantization, Pruning, Efficiency, Eviction, KV Cache Sharing

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ngjZOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors first observe that the keys and values of adjacent layers are nearly orthogonal, even though the hidden states from which they are derived are highly similar. Building upon this, the authors propose BellowsKV, a training-free method that maps the hidden states of n adjacent layers to one shared latent per token. They keep the latent and compress it in two other ways instead: BellowsKV-Quant stores it with uniform low-bit quantization, and BellowsKV-Evict drops tokens with a single eviction decision for the whole block. The authors compare their BellowsKV with various cache compression methods on Llama-3.1-8B, Mistral-7B, and Qwen2.5-7B, using benchmarks including LongBench and Protocol C. Notably, BellowsKV-Quant retains about 99% of full cache accuracy at 3.6 times compression and 97-98% at 13-14 times, where the strongest existing cross-layer method retains at most 84% at 16 times.

Critical strengths
  1. The authors conduct an in-depth examination of the hidden states and KV cache features in traditional full-attention-only LLMs, yielding some insightful findings.
  2. The authors compare a wide range of baseline methods across three LLM families, demonstrating the effectiveness and superiority of BellowsKV-Quant.
Critical weaknesses
  1. BellowsKV remains a low-rank compression of multi-layer hidden states. Therefore, the use of the term "full rank" in Figure 1c might be an overclaim. Furthermore, the authors do not clearly demonstrate in the paper that compressing hidden states is more effective than compressing the KV cache. If such a discussion already exists in the paper, I would ask the authors to clarify and highlight it. If not, I strongly ask the authors to add it.
  2. Although the authors conduct extensive experiments, the models evaluated in this paper are mostly released over a year ago. The lack of evaluation on the latest models decreases the paper’s timeliness and overall contribution.
Questions for authors

Validation of newly released LLMs presents a challenge, largely because most frontier open-source LLMs are hybrid models. In these models, only a subset of layers use full attention, which retains the complete KV cache, thereby creating a bottleneck for verifying such training-free cache optimization methods. Furthermore, since current open-source hybrid models typically interleave efficient layers with full-attention layers, full-attention layers are not adjacent. It remains a question whether compression strategies designed for full-attention models remain effective in such configurations.

Reviewer_YFM2OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes BellowsKV, which shares a latent KV cache across nearby layers without retraining the model. Each layer uses its own projection to read from the shared cache. The method combines layer sharing with quantization and optional token eviction. It achieves 13–14× cache compression while keeping about 97–98% of the baseline task scores.

Critical strengths
  1. The paper shows that nearby layers have similar hidden states, even when their keys and values differ. This gives a clear reason for sharing a latent cache across layers.
  2. BellowsKV achieves 13–14× cache compression while retaining about 97–98% of the baseline task scores, without retraining the model.
  3. The experiments compare layer sharing, quantization, and token eviction. They help explain why keeping the latent width and reducing its precision can preserve accuracy better than further reducing its width.
Critical weaknesses
  1. Each layer needs to reconstruct past keys during decoding, which adds computation. The reported decoding speed is close to the baseline, so the practical benefit mainly comes from memory savings. A breakdown of the reconstruction cost would help clarify this trade-off.
  2. The experiments only cover GQA models. Since MLA already stores a compressed latent KV cache, it is unclear how much further benefit BellowsKV can offer. Testing an MLA model would help show whether the method applies more broadly.
Questions for authors
  1. How much decoding time is spent reconstructing past keys, and how does this cost change with context length and batch size?
  2. Can BellowsKV be applied to MLA models? Since MLA already compresses the KV cache, would sharing across layers still provide useful memory savings without hurting accuracy?
Reviewer_wjsEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

the paper addresses shared kv block representation between adjacent transformer layers without retraining, and shows that while the keys/values of adjacent layers are nearly orthogonal, the inputs to attention are highly correlated (due to residual connections). The authors suggest BellowsKV, a calibrated projections mapping for stacked hidden states of nn layes to one full-rank latent per token. Then, each layer reads its keys/values through a fixed up projection. Thus, the projection attained by layer level sharing, where low-bit uniform quantization is applied to the latent rep., a correction ("whisper") is fitted at prefill, and finally, a redundant (shared) blocks are evicted.

The authors evaluated their method on three 7-8B models on LongBench and two 30B models (BroseComp-Plus), showing the method retains 97-99% of the full cache accuracy at 13-14x compression with all metadata counted, whereas prior cross layer methods degrades significantly.

Critical strengths
  • The main finding - that prior corss-layer methods fail through rank truncation, not through sharing, is supported by a controlled experiment. (xKV-hold: the rival's basis at near-full rank with int4 storage can recover 97.6-99.2 on Llama/Mistral models). Such claims with isolation experiments are rare in this space.

  • Accounting - The measured compressoin ratio include all metadata (group scales, whisper coefficients, residual window), comparisons provided showing that every baseline stores more bytes, and evaluations are full-sample with per-task tables and sample counts.

  • Broad baseline coverage - every running configuration has five per-layer quantization techniques and five cross layer methods, plus an eviction control with identical scoring.

  • honesty - eight implemented and rejected negative results, unexplained anomalies flagged. An honest decode-cost analysis ("sharing buts capacity, never speed").

  • Practicality - adoption cost is low. One-shot gradient-free calibration (~20 minutes per model), factors fold into the weights, decode without custom kernels.

Critical weaknesses
  1. The serving payoff is asserted, not measured. The motivation is serving economics (cache volume, batch size), but evidence stops at decode-speed parity and allocator-level capacity numbers (14.8x kv pool under vLLM). There is no throughput-vs-batch curve, no TTFT or prefill overhead measurement at long context. Since decode parity means the entire benefit is capacity, the paper's central promise, that capacity governs into cheaper serving, needs at least one end-to-end measurement.

  2. The compression does not cover the workload motivation. The introduction relies on long reasoning traces and multi-turn agents, but decode-time tokens stay per-layer in fp16. Only the prompt is compressed. For heavy generation workloads the effective compression ratio declines as generation continues. This scope boundary should be stated explicitly, and ideally quantified (i.e., effective ratio vs. generation length).

  3. The eviction operator creates tension with the paper's own argument. The introduction argues eviction "cannot be undone when regret". Yet, BellowsKV-Evict evicts tokens irreversibly. Scores come from recent queries only, so tokens which are unimportant now may matter for later queries and turns, which is precisely the multi-turn setting the paper targets (e.g., Protocol C in single-turn). The block-level verdict also loses to per-layer eviction at equal keep ratio and wins only at equal memory. The knobs (pooling window 13, sink 4, recent 64) are fixed with no sensitivity analysis .

  4. Evaluation scale sits well below the motivation. The pitch is 131K-1M token contexts. Yet, the experiment cap at 32k, and models are mostly 7-8B. Ratios would improve at longer context (fixed overheads shrink), but accuracy at >100k, where a singled shared eviction error can be fatal for needle in haystack style retrieval is untested.

  5. Novelty over the SVD-LLM/Palu lineage is incremental (the paper credits this honestly - "the primitive is not ours"). The deltas - cross-layer second moment block, then frozen rather than per-sequence basis, the bits (not rank) evidence, are real but should be weighted against how close MiniCache already came conceptuallym and against what this adds beyond it.

  6. Two unexplained anomalies hinders the story's universitality. Every INT4 storage of xKV's basis collapses on Qwen ("we have no explanation") and at ~30x on Mistral the naive stack (xLV + INT4 + pooled SnapKV) beats the composed operator.

Questions for authors
  1. Can you report one end-to-end serving measurement (throughput vs batch size, or TTFT at 32k+) under vLLM, converting the capacity gain into the serving terms the introduction uses? What is the measured prefill overhead of the stacking, projections and whisper fitting as a function of the prefill length?

  2. Why do generated tokens remain per-layer fp16? Is this fundamental (the latent is fitted on prompt statistics) or an engineering choice? What is the effective compression ratio as a function of generation length, e.g., at 32k prompt + 8k generation?

  3. How does the shared blocks eviction verdict behave when later queries revisit earlier context, in e.g., multi-turn dialogue or agent trajectories where turn kk reads evidence from turn 1? Does the regret argument from your introduction applies to BellowsKV-Evic? If not, why?

  4. How sensitive is the result to the eviction knobs (pooling window 13, sink 4, recent 64) and to the calibration corpus (32 windows, half Wiki-Text2 half code)? Was any single choice load bearing?

  5. Do you have a conjecture for the Qwen INT4 collapse of xKV-hold, and for the Mistral model ~30x row where the native stack wins? Does either bound where BellowsKV should be applied?

  6. The folded projections adds about n∗p2n*p^2 parameters per block per side beyond the original n∗d∗pn*d*p (small, yet, nonzero). Could you state this overhead explicitly and note the regime (model width vs block size) where sharing is beneficial ?

== minor comments ==

  • Eq. (2): consider writing it as an explicit minimization (argmin over W-down, W-up), stating that the expectation is over the calibration distribution, and naming G a second-moment (uncentered covariance) matrix. As written, readers may not register Eq. (2) as an optimization problem until Appendix B.1.
  • One sentence of intuition for the whitened decomposition would help: any factorization of W alone decouples per layer because W is block-diagonal; the cross-layer redundancy enters only through G's off-diagonal blocks, and if hidden states were white (G = I), plain SVD of W would be optimal. (This answers the "why not QR/SVD of W directly?" question.)
  • Clarify that blocks partition layers, not tokens, early in Section 3 - the phrase "block of n layers" can be misread as a sequence partition on first pass.
  • Define p explicitly as total KV width (kv-heads x head-dim) when introduced.
  • Abstract: "each layer read its own keys" -> "reads".
  • Section 4: "Protocal C" -> "Protocol C".
  • "For each block of n adjacent layers": the n should be math-italic.
  • "a final block shorter than n is kept as is": consider "(possibly) shorter".
  • AI-use statement: "revise grammar mistake, We did not use" - grammar and capitalization in the grammar statement.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.