- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
In this paper, the authors first observe that the keys and values of adjacent layers are nearly orthogonal, even though the hidden states from which they are derived are highly similar. Building upon this, the authors propose BellowsKV, a training-free method that maps the hidden states of n adjacent layers to one shared latent per token. They keep the latent and compress it in two other ways instead: BellowsKV-Quant stores it with uniform low-bit quantization, and BellowsKV-Evict drops tokens with a single eviction decision for the whole block. The authors compare their BellowsKV with various cache compression methods on Llama-3.1-8B, Mistral-7B, and Qwen2.5-7B, using benchmarks including LongBench and Protocol C. Notably, BellowsKV-Quant retains about 99% of full cache accuracy at 3.6 times compression and 97-98% at 13-14 times, where the strongest existing cross-layer method retains at most 84% at 16 times.
Critical strengths
- The authors conduct an in-depth examination of the hidden states and KV cache features in traditional full-attention-only LLMs, yielding some insightful findings.
- The authors compare a wide range of baseline methods across three LLM families, demonstrating the effectiveness and superiority of BellowsKV-Quant.
Critical weaknesses
- BellowsKV remains a low-rank compression of multi-layer hidden states. Therefore, the use of the term "full rank" in Figure 1c might be an overclaim. Furthermore, the authors do not clearly demonstrate in the paper that compressing hidden states is more effective than compressing the KV cache. If such a discussion already exists in the paper, I would ask the authors to clarify and highlight it. If not, I strongly ask the authors to add it.
- Although the authors conduct extensive experiments, the models evaluated in this paper are mostly released over a year ago. The lack of evaluation on the latest models decreases the paper’s timeliness and overall contribution.
Questions for authors
Validation of newly released LLMs presents a challenge, largely because most frontier open-source LLMs are hybrid models. In these models, only a subset of layers use full attention, which retains the complete KV cache, thereby creating a bottleneck for verifying such training-free cache optimization methods. Furthermore, since current open-source hybrid models typically interleave efficient layers with full-attention layers, full-attention layers are not adjacent. It remains a question whether compression strategies designed for full-attention models remain effective in such configurations.