- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
LLC is a training-free encoding of KV cache across all heads using single low-rank code. Backed by an attention implementation that avoids full KV reconstruction, the authors introduce a two-tier cache which keeps full KV states in the first-tier, then a second-tier of compressed states. LLC shows minimal accuracy degradation across Longbench and RULER under a fixed KV budget, while achieving speedup over dense attention.
Critical strengths
1] Main strength of LLC is the breadth of validation against different eviction and second-tier admission policies. Reusing the same codec beyond its calibration length and evaluating quantized (INT8/INT4) codes further supports the practicality of LLC.
2] LLC is backed by custom attention implementation, which directly consumes the compressed codes without materializing the full KV states. This is evaluated on A100 GPU and shows speedup over dense SDPA.
Critical weaknesses
1] Main weakness I see on LLC is the limited evaluation scope to Longbench and RULER, both similar in long-input-context, relatively short output. Therefore, the evaluation does not establish whether LLC's codec and two-tier eviction remain effective during extended generation, where generated tokens increasingly dominate the cache. With LLM tasks diversifying with the growing popularity of reasoning, LLC should be validated against a broader range of benchmarks.
2] Efficiency evaluation is limited to per-layer attention speedup (Figure 3), per-layer decode latency (Table 9), then analytical FLOPs comparison (Table 8). End-to-end throughput gain, prefill latency, or cache management overhead is not profiled.
Questions for authors
1] On top of the shown effectiveness of LLC with local-global interleaved architecture of Gemma3, and sequence mixing/partial RoPE architecture of Qwen3.5, can LLC be applied to recent architectures, such as Gemma 3n/4 with cross-layer KV sharing, interleaved RoPE/NoPE layers in Llama 4, or shared K=V design in DeepSeek V4? Would these designs require changes to codec calibration, rank allocation, or eviction management?
2] Can the authors evaluate LLC on decode-heavy reasoning benchmarks such as AIME or MATH? Validating fixed codec and two-tier eviction policy's effectiveness, and decode efficiency against these generated-token heavy benchmarks will increase the impact of LLC. For these harder benchmarks, I recommend using reasoning models such as DeepSeek-R1-Distill-Qwen/Llama.
3] Figure 3 and Table 9 measures per-layer attention speedup at matched token counts, using prepared caches. Can the authors report end-to-end latency, including breakdown of encoding/admission/eviction/cache update overheads? A sweep over different input/output lengths would help clarify the speedup sensitivity to cache occupancy and management.
4] Can the authors provide comparison with prior low-rank methods mentioned in the manuscript, Palu, EigenAttention, and OjaKV? The authors should directly compare LLC's codec with these methods while either keeping the same or disabling eviction and reconstruction-error-based admission, basically keeping the same tokens.
5 comments
Response to W2 and Q3
Evaluation protocol. Both methods use one-shot full-history prefill. The LLC timing prototype uses sink/window resident selection, constructs its compressed tier as each layer finishes prefill, and applies demotion and compression during decoding. Both methods use GPU-resident inputs and prebuilt CUDA graphs, excluding model loading, tokenization, host transfer and graph preparation.
We added full-model measurements on Qwen3-4B-Thinking-2507 using one H100 80GB. Dense uses FlashAttention-2; LLC uses direct-code attention with 2K BF16 resident tokens and BF16 r256 or r512 codes. For input length L, the compressed capacity is L − 2,048 (exact-tier budget) + 128 (decode steps).
Context and batch sweep. Each row 128 decode tokens, one warmup and five measured repeats in a fresh process. Paired timing and memory cells show dense / LLC.
| Input | Batch | LLC rank | Prefill + cache (s) | Decode (s) | Total request (s) | Peak allocated (GiB) | Decode (s)peedup | Request speedup |
|---|---|---|---|---|---|---|---|---|
| 16K | 2 | 256 | 1.084–1.090 / 1.165–1.189 | 0.818 / 0.774 | 1.893–1.894 / 1.882–1.883 | 15.21 / 12.07 | 1.06× | 1.01× |
| 16K | 4 | 512 | 2.158–2.161 / 2.269–2.274 | 1.013 / 1.014 | 3.159–3.162 / 3.222–3.223 | 22.86 / 18.15 | 1.00× | 0.98× |
| 16K | 4 | 256 | 2.163–2.176 / 2.261–2.269 | 1.013 / 0.894 | 3.161–3.174 / 3.094 | 22.79 / 16.51 | 1.13× | 1.02–1.03× |
| 16K | 8 | 256 | 4.330–4.389 / 4.489–4.506 | 1.400 / 1.095–1.096 | 5.714–5.715 / 5.506 | 37.96 / 25.40 | 1.28× | 1.04× |
| 32K | 2 | 256 | 3.017–3.034 / 3.140–3.143 | 1.039–1.040 / 0.829–0.830 | 4.071–4.078 / 3.925–3.930 | 22.76 / 15.69 | 1.25× | 1.04× |
| 32K | 4 | 512 | 6.005–6.012 / 6.183–6.191 | 1.402 / 1.157 | 7.359–7.363 / 7.251–7.270 | 37.96 / 26.51 | 1.21× | 1.01× |
| 32K | 4 | 256 | 6.037–6.040 / 6.207–6.229 | 1.401 / 0.988–0.989 | 7.433–7.437 / 7.130–7.140 | 37.89 / 23.75 | 1.42× | 1.04× |
| 32K | 8 | 512 | 12.101–12.103 / 12.301–12.312 | 2.206–2.207 / 1.626–1.628 | 14.842–14.855 / 13.827–13.851 | 68.22 / 45.33 | 1.36× | 1.07× |
| 32K | 8 | 256 | 12.249–12.327 / 12.392–12.398 | 2.209 / 1.247–1.248 | 14.804–14.819 / 13.539–13.552 | 68.15 / 39.88 | 1.77× | 1.09× |
| 32K | 16 | 256 | OOM / 26.334–26.572 | OOM / 1.766 | OOM / 31.258–31.485 | OOM / 72.14 | — | — |
| 64K | 2 | 256 | 9.590–9.631 / 9.797–9.809 | 1.403–1.404 / 0.908 | 11.046–11.050 / 10.664–10.668 | 37.85 / 22.93 | 1.55× | 1.04× |
| 64K | 4 | 512 | 19.297–19.318 / 19.497–19.517 | 2.172–2.173 / 1.412–1.413 | 21.803–21.818 / 20.783–20.845 | 68.14 / 43.25 | 1.54× | 1.05× |
| 64K | 4 | 256 | 19.314–19.385 / 19.498–19.515 | 2.171 / 1.162–1.163 | 21.354–21.356 / 20.605–20.610 | 68.07 / 38.24 | 1.87× | 1.04× |
| 64K | 8 | 256 | OOM / 40.841–40.888 | OOM / 1.540 | OOM / 45.542–45.913 | OOM / 68.85 | — | — |
LLC reduces peak memory across all completed comparisons, and its decode speedup generally increases with context length and batch size, reaching 1.87× at r256 and 1.54× at r512. Decode speedups translate into smaller total-request gains because prefill accounts for most of the runtime in these 128-step decoding workloads.
The memory reduction also enables larger workloads: LLC r256 completes 32K/batch 16 and 64K/batch 8, where dense runs out of memory. LLC r512 also runs out of memory in these two configurations.
Cache-management and attention breakdown. We microbenchmark a single layer with a populated 2K BF16 resident tier and 6K BF16 r512 codes, receiving one 128-token prefill chunk, at batch 4 on an H100. The layer has 32 query heads and 8 KV heads, with 128 dimensions per head. We use the norm shortcut for error scoring. Attention evaluates 128 queries over the 2K resident positions, 6K compressed positions and 128 incoming positions, with causal masking within the incoming chunk. All rows report medians over 50 paired trials, per 128-token chunk per layer.
| Stage | Time (ms) |
|---|---|
| Fused K/V gathering, encoding and input norm | 0.0215 |
| Resident-ring update | 0.0088 |
| Error scoring, admission/eviction and cache writes | 0.0967 |
| Resident + codec attention (128-query chunk) | 2.9936 |
The first row projects demoted K/V into latent codes and computes their squared input norms for error scoring. The resident-ring update replaces the demoted slots with incoming K/V. The third row scores reconstruction error, selects which codes to retain, and writes them to the compressed tier. The attention row includes query projection, both tier scans, value decoding and a shared softmax normalization across both tiers.
Response to W1 and Q2 — decoding-heavy applicatyion
We added a generation-heavy reasoning evaluation using Qwen3-4B-Thinking-2507 on a fixed 200-problem MATH500 subset (indices 0–199). Generation is greedy with a 32,768-token output limit. LLC uses the same fixed calibration-fitted post-RoPE codec throughout generation, with r512 BF16 codes, BF16 resident K/V, ValueDiff exact-tier selection and reconstruction-based compressed-tier retention.
The VECTOR controls use a post-RoPE K-to-V predictor under the same ValueDiff cache management, retaining BF16 full-width keys in the second tier.
| Method | Resident + second-tier budget | Nominal MiB/layer | Accuracy (%) |
|---|---|---|---|
| Dense | Full cache | Growing | 94.5 |
| LLC r512 | 2K+4K | 12 | 90.5 |
| LLC r512 | 2K+8K | 16 | 90.5 |
| LLC r512 | 4K+4K | 20 | 91.0 |
| LLC r512 | 4K+8K | 24 | 92.0 |
| VECTOR post-RoPE | 2K+4K | 16 | 90.0 |
| VECTOR post-RoPE | 2K+8K | 24 | 91.5 |
LLC retains strong reasoning accuracy as generated tokens dominate the history: the 4K+8K setting is 2.5 percentage points below dense with a bounded 24-MiB/layer payload.
Thank you for the careful and thoughtful review. We appreciate the time you took to engage with our work and the helpful suggestions for strengthening the paper.
Response to Q1 — Architectural applicability
LLC's two-tier framework operates on the attention cache and can accommodate different cache-sharing and positional-encoding designs. Our evaluated Gemma3 and Qwen3.5 integrations apply LLC to full-attention layers while preserving native sliding-window and recurrent-state handling, respectively. For the architectures in the question, we propose the following adaptations.
Cross-layer KV sharing. When multiple layers consume the same K/V cache, one codec can be calibrated at the producing layer, and its codes and reconstructions reused across the consuming layers. Sharing therefore does not require separate calibration or duplicated code storage for each consumer. Cache updates and eviction must be coordinated at the shared cache, preserving the consumers' attention masks and positional semantics. If admission uses consumer-dependent errors, its score should account for the consuming layers.
Interleaved RoPE/NoPE. Each codec is calibrated in the key space actually used by its layer: rotated keys for RoPE layers and unrotated keys for NoPE layers, including the model's normalization. The same Gram-based fitting procedure applies. Our pre- and post-RoPE LLC evaluations already demonstrate that the codec can operate in either key space. NoPE removes the need to handle key rotation; whether it improves compression is an empirical question. The two-tier eviction mechanism remains applicable.
Shared K=V. A native shared vector should be encoded once, with its reconstruction reused for both key and value roles. This retains the low-rank tier and reconstruction-based admission while avoiding duplicated K/V storage. Our Head-Joint ablation supports flexibility in codec structure, although it does not directly evaluate native K=V attention. For DeepSeek V4, an integration would also need to preserve its native compressed-memory and sparse-selection mechanisms.