acceptodds
Desk rejected at ICLR 2027

Small Frequency Corrections Can Change What Survives KV Cache Compression

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Compressing a key-value cache before its next question is known requires choosing what to retain without knowing which evidence will matter. Value energy measures entry strength but does not distinguish isolated keys from those with many similar neighbors. We introduce TwinKV, a training-free method that discounts value energy by nonlocal post-RoPE key frequency. Prefix attention allocates head capacities, while retained entries preserve their original keys and values under an exact storage budget. Across four language models, TwinKV exceeds five evaluated compressed baselines in mean score on LongBench, LooGLE, and RULER at 50% KV removal. Component controls isolate the frequency contribution. On Llama-3.2-1B RULER at 75% removal, normalized frequency weights average 0.95, yet change 7% of nonprotected retained positions and improve value-only retention by about 5.5 points under both uniform and adapted capacities. Permuting the weights within heads weakens this gain. These results show that modest frequency corrections can change retention and answering outcomes, with effects that depend on the model and task.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV cache compression, long-context inference, memory-efficient inference, large language models

Related papers

Discussion (0)

Sign in to comment.