acceptodds
Under review as a conference paper at ICLR 2027

LLC: A Linear Low-Rank Codec for Joint Key-Value Compression Before Token Eviction

Abstract

Token eviction controls KV cache growth but permanently discards states that may become relevant later. Retaining these states in compressed form requires a compact representation that remains useful to attention. We introduce LLC, a closed-form, training-free codec that jointly encodes post-RoPE keys and values across all heads into a shared low-rank code. Fixed linear maps enable direct attention over the codes without reconstructing full-width KV states. We deploy the codec in a two-tier cache: a base eviction policy manages exact residence, while a separately budgeted compressed tier retains demoted tokens with the lowest reconstruction errors. The joint representation improves reconstruction fidelity and yields stronger accuracy–memory trade-offs across models and eviction policies. Across RULER at 8k, 16k, and 32k, LLC approaches dense accuracy with less KV cache memory; at 32k, it stays within 1.5 points of dense across four models using 31–38% of the dense cache. On LongBench, LLC outperforms a prior codec in all eight matched-memory comparisons. Under CUDA Graph replay, an A100 implementation achieves – speedups over matched dense SDPA in single-token, per-layer attention microbenchmarks from 8k to 64k. INT8 quantization of the stored codes further halves their storage with negligible accuracy loss. Together, LLC mitigates irreversible eviction without sacrificing memory efficiency.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
KV cache compression, efficient LLM inference, low-rank approximation, long-context language models, token eviction

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_eirWOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

LLC is a training-free encoding of KV cache across all heads using single low-rank code. Backed by an attention implementation that avoids full KV reconstruction, the authors introduce a two-tier cache which keeps full KV states in the first-tier, then a second-tier of compressed states. LLC shows minimal accuracy degradation across Longbench and RULER under a fixed KV budget, while achieving speedup over dense attention.

Critical strengths

1] Main strength of LLC is the breadth of validation against different eviction and second-tier admission policies. Reusing the same codec beyond its calibration length and evaluating quantized (INT8/INT4) codes further supports the practicality of LLC.

2] LLC is backed by custom attention implementation, which directly consumes the compressed codes without materializing the full KV states. This is evaluated on A100 GPU and shows speedup over dense SDPA.

Critical weaknesses

1] Main weakness I see on LLC is the limited evaluation scope to Longbench and RULER, both similar in long-input-context, relatively short output. Therefore, the evaluation does not establish whether LLC's codec and two-tier eviction remain effective during extended generation, where generated tokens increasingly dominate the cache. With LLM tasks diversifying with the growing popularity of reasoning, LLC should be validated against a broader range of benchmarks.

2] Efficiency evaluation is limited to per-layer attention speedup (Figure 3), per-layer decode latency (Table 9), then analytical FLOPs comparison (Table 8). End-to-end throughput gain, prefill latency, or cache management overhead is not profiled.

Questions for authors

1] On top of the shown effectiveness of LLC with local-global interleaved architecture of Gemma3, and sequence mixing/partial RoPE architecture of Qwen3.5, can LLC be applied to recent architectures, such as Gemma 3n/4 with cross-layer KV sharing, interleaved RoPE/NoPE layers in Llama 4, or shared K=V design in DeepSeek V4? Would these designs require changes to codec calibration, rank allocation, or eviction management?

2] Can the authors evaluate LLC on decode-heavy reasoning benchmarks such as AIME or MATH? Validating fixed codec and two-tier eviction policy's effectiveness, and decode efficiency against these generated-token heavy benchmarks will increase the impact of LLC. For these harder benchmarks, I recommend using reasoning models such as DeepSeek-R1-Distill-Qwen/Llama.

3] Figure 3 and Table 9 measures per-layer attention speedup at matched token counts, using prepared caches. Can the authors report end-to-end latency, including breakdown of encoding/admission/eviction/cache update overheads? A sweep over different input/output lengths would help clarify the speedup sensitivity to cache occupancy and management.

4] Can the authors provide comparison with prior low-rank methods mentioned in the manuscript, Palu, EigenAttention, and OjaKV? The authors should directly compare LLC's codec with these methods while either keeping the same or disabling eviction and reconstruction-error-based admission, basically keeping the same tokens.

5 comments

Thank you for the careful and thoughtful review. We appreciate the time you took to engage with our work and the helpful suggestions for strengthening the paper.

Response to Q1 — Architectural applicability

LLC's two-tier framework operates on the attention cache and can accommodate different cache-sharing and positional-encoding designs. Our evaluated Gemma3 and Qwen3.5 integrations apply LLC to full-attention layers while preserving native sliding-window and recurrent-state handling, respectively. For the architectures in the question, we propose the following adaptations.

Cross-layer KV sharing. When multiple layers consume the same K/V cache, one codec can be calibrated at the producing layer, and its codes and reconstructions reused across the consuming layers. Sharing therefore does not require separate calibration or duplicated code storage for each consumer. Cache updates and eviction must be coordinated at the shared cache, preserving the consumers' attention masks and positional semantics. If admission uses consumer-dependent errors, its score should account for the consuming layers.

Interleaved RoPE/NoPE. Each codec is calibrated in the key space actually used by its layer: rotated keys for RoPE layers and unrotated keys for NoPE layers, including the model's normalization. The same Gram-based fitting procedure applies. Our pre- and post-RoPE LLC evaluations already demonstrate that the codec can operate in either key space. NoPE removes the need to handle key rotation; whether it improves compression is an empirical question. The two-tier eviction mechanism remains applicable.

Shared K=V. A native shared vector should be encoded once, with its reconstruction reused for both key and value roles. This retains the low-rank tier and reconstruction-based admission while avoiding duplicated K/V storage. Our Head-Joint ablation supports flexibility in codec structure, although it does not directly evaluate native K=V attention. For DeepSeek V4, an integration would also need to preserve its native compressed-memory and sparse-selection mechanisms.

Response to W2 and Q3

Evaluation protocol. Both methods use one-shot full-history prefill. The LLC timing prototype uses sink/window resident selection, constructs its compressed tier as each layer finishes prefill, and applies demotion and compression during decoding. Both methods use GPU-resident inputs and prebuilt CUDA graphs, excluding model loading, tokenization, host transfer and graph preparation.

We added full-model measurements on Qwen3-4B-Thinking-2507 using one H100 80GB. Dense uses FlashAttention-2; LLC uses direct-code attention with 2K BF16 resident tokens and BF16 r256 or r512 codes. For input length L, the compressed capacity is L − 2,048 (exact-tier budget) + 128 (decode steps).

Context and batch sweep. Each row 128 decode tokens, one warmup and five measured repeats in a fresh process. Paired timing and memory cells show dense / LLC.

InputBatchLLC rankPrefill + cache (s)Decode (s)Total request (s)Peak allocated (GiB)Decode (s)peedupRequest speedup
16K22561.084–1.090 / 1.165–1.1890.818 / 0.7741.893–1.894 / 1.882–1.88315.21 / 12.071.06×1.01×
16K45122.158–2.161 / 2.269–2.2741.013 / 1.0143.159–3.162 / 3.222–3.22322.86 / 18.151.00×0.98×
16K42562.163–2.176 / 2.261–2.2691.013 / 0.8943.161–3.174 / 3.09422.79 / 16.511.13×1.02–1.03×
16K82564.330–4.389 / 4.489–4.5061.400 / 1.095–1.0965.714–5.715 / 5.50637.96 / 25.401.28×1.04×
32K22563.017–3.034 / 3.140–3.1431.039–1.040 / 0.829–0.8304.071–4.078 / 3.925–3.93022.76 / 15.691.25×1.04×
32K45126.005–6.012 / 6.183–6.1911.402 / 1.1577.359–7.363 / 7.251–7.27037.96 / 26.511.21×1.01×
32K42566.037–6.040 / 6.207–6.2291.401 / 0.988–0.9897.433–7.437 / 7.130–7.14037.89 / 23.751.42×1.04×
32K851212.101–12.103 / 12.301–12.3122.206–2.207 / 1.626–1.62814.842–14.855 / 13.827–13.85168.22 / 45.331.36×1.07×
32K825612.249–12.327 / 12.392–12.3982.209 / 1.247–1.24814.804–14.819 / 13.539–13.55268.15 / 39.881.77×1.09×
32K16256OOM / 26.334–26.572OOM / 1.766OOM / 31.258–31.485OOM / 72.14——
64K22569.590–9.631 / 9.797–9.8091.403–1.404 / 0.90811.046–11.050 / 10.664–10.66837.85 / 22.931.55×1.04×
64K451219.297–19.318 / 19.497–19.5172.172–2.173 / 1.412–1.41321.803–21.818 / 20.783–20.84568.14 / 43.251.54×1.05×
64K425619.314–19.385 / 19.498–19.5152.171 / 1.162–1.16321.354–21.356 / 20.605–20.61068.07 / 38.241.87×1.04×
64K8256OOM / 40.841–40.888OOM / 1.540OOM / 45.542–45.913OOM / 68.85——

LLC reduces peak memory across all completed comparisons, and its decode speedup generally increases with context length and batch size, reaching 1.87× at r256 and 1.54× at r512. Decode speedups translate into smaller total-request gains because prefill accounts for most of the runtime in these 128-step decoding workloads.

The memory reduction also enables larger workloads: LLC r256 completes 32K/batch 16 and 64K/batch 8, where dense runs out of memory. LLC r512 also runs out of memory in these two configurations.

Cache-management and attention breakdown. We microbenchmark a single layer with a populated 2K BF16 resident tier and 6K BF16 r512 codes, receiving one 128-token prefill chunk, at batch 4 on an H100. The layer has 32 query heads and 8 KV heads, with 128 dimensions per head. We use the norm shortcut for error scoring. Attention evaluates 128 queries over the 2K resident positions, 6K compressed positions and 128 incoming positions, with causal masking within the incoming chunk. All rows report medians over 50 paired trials, per 128-token chunk per layer.

StageTime (ms)
Fused K/V gathering, encoding and input norm0.0215
Resident-ring update0.0088
Error scoring, admission/eviction and cache writes0.0967
Resident + codec attention (128-query chunk)2.9936

The first row projects demoted K/V into latent codes and computes their squared input norms for error scoring. The resident-ring update replaces the demoted slots with incoming K/V. The third row scores reconstruction error, selects which codes to retain, and writes them to the compressed tier. The attention row includes query projection, both tier scans, value decoding and a shared softmax normalization across both tiers.

Response to W1 and Q2 — decoding-heavy applicatyion

We added a generation-heavy reasoning evaluation using Qwen3-4B-Thinking-2507 on a fixed 200-problem MATH500 subset (indices 0–199). Generation is greedy with a 32,768-token output limit. LLC uses the same fixed calibration-fitted post-RoPE codec throughout generation, with r512 BF16 codes, BF16 resident K/V, ValueDiff exact-tier selection and reconstruction-based compressed-tier retention.

The VECTOR controls use a post-RoPE K-to-V predictor under the same ValueDiff cache management, retaining BF16 full-width keys in the second tier.

MethodResident + second-tier budgetNominal MiB/layerAccuracy (%)
DenseFull cacheGrowing94.5
LLC r5122K+4K1290.5
LLC r5122K+8K1690.5
LLC r5124K+4K2091.0
LLC r5124K+8K2492.0
VECTOR post-RoPE2K+4K1690.0
VECTOR post-RoPE2K+8K2491.5

LLC retains strong reasoning accuracy as generated tokens dominate the history: the 4K+8K setting is 2.5 percentage points below dense with a bounded 24-MiB/layer payload.

Reviewer_Q4DvOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes LLC, a training-free codec for retaining KV states that a token-eviction policy would otherwise discard. For each layer, LLC concatenates the post-RoPE keys and values of all KV heads into one 2HD-dimensional vector, fits a rank-r orthonormal basis by eigendecomposing the uncentered second-moment matrix of calibration states, and stores each demoted token only as its r-dimensional code. Because the maps act after RoPE, the key decoder can be folded into the query and the value decoder applied after code accumulation, which gives an exact joint softmax over an exact tier and a code tier without materializing reconstructed K/V (Section 3.2, Appendix B). LLC sits beneath an unchanged base eviction policy as a separately budgeted second tier that, when full, retains the codes with the lowest normalized reconstruction residual. Experiments cover six models (full-attention, sliding-window hybrid, and linear-attention hybrid), four eviction policies (StreamingLLM, SnapKV, KeyDiff, ValueDiff), RULER at 8k/16k/32k with maps calibrated on sequences of at most 8k tokens, and LongBench, with a factorized per-head PCA codec (Sep) and a VECTOR-style codec implemented in the same two-tier cache as the main baselines. Supporting analyses include a 2x2 factorial over K/V and head pooling, pre- vs. post-RoPE comparisons, an admission-rule ablation, INT8/INT4 code quantization, and A100 per-layer decode-attention microbenchmarks reporting 1.31-4.10x speedups over dense SDPA.

Critical strengths
  1. Practical, training-free, policy-agnostic design with an exact execution path. The codec needs one calibration pass and one eigendecomposition per layer (Appendix A) and leaves the base policy's exact-selection rule untouched (Algorithm 1). Because it codes post-RoPE states, decoding is token-independent, so attention over codes is algebraically exact rather than approximate. I checked the absorption, the online-softmax merge, and the FLOP/byte accounting (Eqs. B.1-B.4, Section B.2, Table 8), and they are correct. This matters because the second tier can be deployed on existing checkpoints without retraining or materializing reconstructed K/V.

  2. Broad and transparent evidence that a low-rank second tier improves on exact-only eviction. Across six models (including two hybrid architectures), four eviction policies, RULER 8k/16k and LongBench, LLC often exceeds exact-only caches that use more memory. For example, Qwen3-4B with ValueDiff at 2k+6k and r=512 reaches 95.86 at 14 MiB/layer, versus 94.70 for a 4k exact cache at 16 MiB and 95.85 dense at 32 MiB (Table 7). Gemma-3-4B at 16k reaches 87.78 at 20 MiB, versus 85.51 for an 8k exact cache at 32 MiB (Tables 3, 24). The full grids and the explicit reporting of non-monotonic and below-floor cells (Section 4.1, C.1.3, D.1-D.3) make this the paper's most credible contribution.

  3. Length transfer and informative design analyses. LLC's fixed post-RoPE maps transfer from 8k calibration to 32k without refitting and, at r=512, stay within 1.5 points of dense on all four models at 40-48 instead of 128 MiB/layer (Table 3), whereas the post-RoPE VECTOR adaptation degrades sharply in the same setting. The pre-/post-RoPE study (Tables 5, 14-16) shows that post-RoPE coding costs about 0.04-0.05 in R^2 but changes RULER-8K accuracy by less than one point on five of six models, while enabling absorption. The factorial (Table 11) shows that cross-head pooling is the larger reconstruction gain for all six models.

Critical weaknesses
  1. The benefit of joint K/V coding, the paper's headline design choice, is not isolated, and its end-task payoff over the strongest factorized variant is small (Section 3.1; Tables 1, 11-13; Appendix C.1).

First, the finding that LLC is highest in every column of Tables 1 and 11 holds by construction on the calibration distribution. Each factorized variant spans a feasible rank-r subspace for LLC's unconstrained eigenproblem (Eq. 2), and the pair R^2 weights K and V by raw scale, which is exactly LLC's own objective. Only the size of the margins and the KV-Joint vs. Head-Joint comparison carry evidential weight for the shared-source argument.

Second, the factorized variants use uniform rank splits (r_K = r_V, uniform per head; Appendix C.1). LLC's margin therefore conflates cross-block correlation with adaptive rank allocation. A block-diagonal codec that keeps the top-r eigenvalues across its blocks would separate the two using the same statistics.

Third, on end tasks LLC leads Head-Joint by 0.05-1.07 points at the single reported operating point (Table 13). The paper itself notes that Head-Joint wins 7 of 9 Llama-3.1-8B cells at r=512 and that Sep is nominally best in about half of the Gemma2-9B grid (C.1.3). Head-Joint also needs half of LLC's leading scan FLOPs (Table 12).

Finally, the main-text accuracy-memory comparison (Figure 2) includes only Sep, the weakest variant.

This matters because the joint code is what distinguishes LLC from existing post-hoc low-rank KV codecs. As evaluated, the evidence supports pooling across heads more clearly than pooling K with V.

  1. External comparisons do not establish an advantage over the strongest alternatives at matched memory (Sections 2, 4.2-4.3; Tables 3-5, 14, 26).

VECTOR is evaluated in post-RoPE form, whereas Section 2 describes the original as predicting values from pre-RoPE keys within its own allocation scheme. Its sharp 32k degradations in the main text (Table 3: 78.23 on Qwen3-4B; 45.87 and 0.00 on Llama-3.1-8B) largely disappear in pre-RoPE form (Table 26: 93.95; 84.60 and 81.01). LLC at r=512 still beats pre-RoPE VECTOR with less memory in those four 32k cells, so there the issue is the framing and size of the gap. On LongBench, however, pre-RoPE VECTOR scores above post-RoPE LLC at 2k+6k on Qwen3-4B (44.84 vs. 44.52) and Llama-3.1-8B (38.13 vs. 37.19) (Table 5b), albeit at 20 vs. 14 MiB/layer. The abstract's claim of winning all eight matched-memory comparisons is made only against the post-RoPE variant. The negative, sign-flipping VECTOR R^2 values on Llama (Table 14: -1.67 pre-RoPE on RULER-8K, -1.08 post-RoPE on C4) also raise the question of whether the OLS predictor is regularized as in the original.

No low-bit second tier is evaluated, although Section 2 presents MiKV, which keeps states selected for eviction at lower precision, as the precision-axis counterpart. With 2HD = 2048, storing demoted K/V at 4 or 2 bits matches the per-token payload of r=512 or r=256 BF16 codes (ignoring scales). Table 7 quantizes the codes, but it does not show whether codes or quantized K/V are the better use of the same second-tier memory.

The abstract and conclusion claim improved trade-offs over alternative codecs, and one internal variant plus one adapted implementation is thin support for that claim.

  1. The headline numbers lack base-policy controls and uncertainty estimates (Sections 1, 4.2-4.3; Tables 3-4, 6, 17-18, 23-26).

At 32k, no ValueDiff exact-only result (B_c = 0) and no iso-memory exact-only cache (e.g., 10k exact tokens = 40 MiB) is reported. The claim of staying within 0.6 points of dense at 31.25% of the payload therefore cannot be attributed to LLC rather than to ValueDiff. At 16k, where floors are given, an 8k exact-only ValueDiff cache is already within 1.3 points of dense on Qwen3-4B (94.24 vs. 95.47) and Qwen3.5-4B (95.15 vs. 95.88), though not on Gemma-3-4B (85.51 vs. 89.61) (Tables 23-25). Table 4 omits B_c = 0 floors, absolute dense scores, and the input truncation length.

RULER uses 50 samples per task without repetitions or confidence intervals, yet several conclusions rest on sub-point margins (Table 6: +0.89 and -0.14 vs. FIFO under ValueDiff; Table 4: 0.19 for one matched-memory pair). The B_e = 8192, B_c = 0 path, which should retain essentially the full context, also differs from the separately measured dense score by 0.17 points on Qwen3-4B (95.68 vs. 95.85, Table 17).

Finally, adding a compressed tier sometimes falls below the exact-only floor: StreamingLLM 2k+2k, r=512 gives 40.65 vs. 43.17 (Table 17); SnapKV 2k+2k, r=512 gives 79.49 vs. 81.85 (Table 18); StreamingLLM 4k+4k, r=256 gives 38.41 vs. 43.18 (Table 23). Pre-RoPE LLC also drops from 92.99 to 79.23 as B_c grows from 8k to 16k at 32k (Table 26). These cases are reported candidly but neither diagnosed nor guarded against, which matters for a method whose premise is mitigating eviction loss.

Questions for authors
  1. Please add Sep and Head-Joint variants whose total rank r is allocated by keeping the top-r eigenvalues across their blocks (same Gram statistics), and report key-only and value-only R^2 together with attention-score or attention-output error for all four variants. If adaptive allocation closes most of the gap, the half-FLOP factorized codec becomes the better design; a clear residual margin would largely resolve Weakness 1.

  2. Please report pre-RoPE VECTOR (ideally with its original allocation scheme) in Tables 3 and 4 at matched memory, e.g., VECTOR at 2k+3k vs. LLC r=512 at 2k+6k, and state whether the OLS map is regularized. This determines whether the matched-memory LongBench claim and the long-context advantage hold against the original method.

  3. Please evaluate the same two-tier cache with demoted K/V stored at 4 bits (about the r=512 BF16 payload) and 2 bits (about r=256), e.g., with a MiKV- or KIVI-style quantizer, at least on Qwen3-4B and Llama-3.1-8B under ValueDiff and SnapKV. A competitive low-bit tier would substantially weaken the case for low-rank coding, while a clear LLC win would strengthen the central claim.

6 comments

Thank you for the detailed and thoughtful review. Your comments helped us sharpen the comparisons and clarify the evidence supporting our design choices.

We first clarify the evaluation setting: our main cache-limited experiments process prompts in 128-token chunks, applying eviction and, when a compressed tier is enabled, demotion and compression whenever the corresponding budgets are exceeded, during both prefill and decoding. This follows the block-processing setting studied in KeyDiff, CAOTE and ValueDiff [1–3]; QuoKA also studies sparse attention during chunked prefill [4].

This is a more demanding accuracy setting than dense prefill followed by eviction. Cache decisions are made before the remaining prompt is available. Once the budget is exceeded, later prompt tokens attend to retained or reconstructed history, so approximation can affect subsequent hidden states and the K/V computed from them. Dense prefill followed by eviction preserves full-history attention while processing the prompt and does not introduce this propagation during prefill. We will revise the manuscript to explicitly describe the prefill and decoding protocols, including chunk size and when eviction, demotion and compression occur.

Response to W1 and Q1

We agree that calibration-objective optimality alone does not establish the best end-task codec. We added the requested allocation controls and separate K/V and attention diagnostics.

Adaptive Sep and Head-Joint retain the top-r eigenvalues across their permitted blocks within each layer, using the same Gram statistics. Below are held-out joint R² means over six models, with 100 RULER and 100 C4 sequences per model.

CodecRULER r256RULER r512C4 r256C4 r512
Sep, uniform0.61690.74220.64110.7491
Sep, adaptive0.68960.79760.69830.7982
Head-Joint, uniform0.80100.88040.80740.8843
Head-Joint, adaptive0.82590.89600.82850.8979
LLC0.84820.91300.85290.9165

Adaptive Head-Joint closes 42–53% of the uniform Head-Joint gap; LLC retains R² margins of 0.0223/0.0170 on RULER and 0.0244/0.0186 on C4 at r256/r512. Thus a held-out reconstruction advantage remains after controlling allocation, although cross-head pooling accounts for the larger gain.

We also evaluate separate K/V R² and both attention errors on 100 matched RULER-8K sequences per model. Each cell gives r256 / r512. These are fixed dense-trajectory diagnostics with matched states, queries and causal support, without eviction. Score/output errors are relative L2, with outputs measured before the output projection.

ModelCodecK R² ↑V R² ↑Score error ↓Output error ↓
Qwen3-4BUniform Sep0.7205 / 0.83010.3758 / 0.53770.4331 / 0.31581.0026 / 0.9286
Qwen3-4BAdaptive Sep0.7695 / 0.85260.1535 / 0.27100.3769 / 0.27961.0438 / 1.0214
Qwen3-4BUniform Head-Joint0.8870 / 0.93080.6529 / 0.79410.2277 / 0.17600.7230 / 0.5324
Qwen3-4BAdaptive Head-Joint0.8941 / 0.93650.4055 / 0.54620.2213 / 0.16620.8802 / 0.7400
Qwen3-4BLLC0.9065 / 0.94670.5892 / 0.69030.2072 / 0.15260.9129 / 0.6970
Llama-3.1-8BUniform Sep0.6719 / 0.77850.3537 / 0.50980.1819 / 0.13970.9415 / 0.8405
Llama-3.1-8BAdaptive Sep0.7807 / 0.90380.0284 / 0.04400.1374 / 0.08850.9902 / 0.9807
Llama-3.1-8BUniform Head-Joint0.8977 / 0.93750.6098 / 0.75600.0909 / 0.06840.7339 / 0.5987
Llama-3.1-8BAdaptive Head-Joint0.9356 / 0.97150.1460 / 0.30000.0694 / 0.04820.9375 / 0.8440
Llama-3.1-8BLLC0.9366 / 0.97350.4251 / 0.51400.0683 / 0.04620.8200 / 0.7786

Across all eight Qwen/Llama × RULER/C4 × rank cells, LLC has the highest K R² and lowest score error, while uniform Head-Joint has the highest V R² and lowest output error.

We compare uniform Head-Joint and LLC on RULER-8K under ValueDiff, with a fixed 2K exact tier, r256 and BF16. Both use the same two-tier mechanism, base eviction policy and reconstruction-error-based compressed-tier retention rule. Codec-dependent residuals can produce different retained token sets.

Exact + compressedModelUniform Head-JointLLCLLC minus HJ, pp
2K+2KQwen3-4B93.7894.30+0.51
2K+2KLlama-3.1-8B80.7883.82+3.04
2K+2KGemma3-4B86.8587.45+0.61
2K+4KQwen3-4B94.2094.55+0.35
2K+4KLlama-3.1-8B84.9984.35−0.64
2K+4KGemma3-4B89.7989.83+0.03
2K+6KQwen3-4B94.5294.86+0.34
2K+6KLlama-3.1-8B89.8690.93+1.07
2K+6KGemma3-4B90.7490.79+0.05

LLC achieves higher accuracy on all three models at the smallest tested compressed-tier budget. At larger budgets, the differences are generally small and model-dependent, with Head-Joint outperforming LLC on Llama at 2K+4K. The 2K+2K/2K+4K comparisons have matching saved inputs and configurations apart from codec artifacts and output paths.

We acknowledge Head-Joint as a strong codec option within our two-tier cache framework, sharing the base eviction policy and reconstruction-error-based compressed-tier retention. Its competitive performance demonstrates that the framework accommodates factorized coding. Full joint LLC retains a reconstruction and attention-score advantage after adaptive allocation, while downstream accuracy depends on the model and budget.

We also acknowledge Head-Joint's computational advantage: it requires half LLC's leading compressed-tier scan FLOPs under Table 12's accounting. This makes it a promising direction for further kernel optimization and latency evaluation.

Q2 response — VECTOR-type coding, RoPE space and fitting

Comparison within the shared cache framework

As stated in Section 4 under “Codec baselines,” our experiments compare reconstruction mechanisms within the proposed two-tier cache. VECTOR supplies the full-width-key, OLS value-reconstruction baseline under this shared cache management. We additionally evaluate its pre-RoPE variant to distinguish the effect of RoPE space from the codec comparison.

Post-RoPE LLC permits direct attention over codes: its fixed linear decoder can be absorbed into attention, reusing the projected query across tokens and decoding values after accumulation. Pre-RoPE coding retains position-dependent rotation inside the token scan. Our choice therefore has a concrete computational rationale, and the new pre-/post-RoPE controls below assess its accuracy cost separately.

Pre-RoPE VECTOR is the stronger comparator in the cited long-context cases. LLC r512 still outperforms it with less memory in the four cited 32K cells, although the margins are smaller than against post-RoPE VECTOR.

Pre-/Post-RoPE VECTOR: Equal-Capacity and Equal-Memory Comparisons

The table combines the cited equal-token comparison, the requested matched-memory evaluation and additional LLC operating points under ValueDiff. Budgets denote exact + compressed tokens; memory is nominal MiB/layer.

MethodModelBudgetLongBench scoreNominal MiB/layer
Pre-RoPE VECTORQwen3-4B2K+6K44.8420
Pre-RoPE VECTORQwen3-4B2K+3K43.4214
Post-RoPE VECTORQwen3-4B2K+3K42.5814
Pre-RoPE LLC r512Qwen3-4B2K+6K45.4514
Post-RoPE LLC r512Qwen3-4B2K+6K44.5214
Post-RoPE LLC r512Qwen3-4B4K+2K46.5318
Pre-RoPE VECTORLlama-3.1-8B2K+6K38.1320
Pre-RoPE VECTORLlama-3.1-8B2K+3K41.3714
Post-RoPE VECTORLlama-3.1-8B2K+3K37.6414
Pre-RoPE LLC r512Llama-3.1-8B2K+6K37.2814
Post-RoPE LLC r512Llama-3.1-8B2K+6K37.1914
Post-RoPE LLC r512Llama-3.1-8B4K+2K43.0018

The 2K+6K comparison uses equal token capacities but different memory; the 14-MiB rows compare equal nominal payloads. The 18-MiB LLC rows show additional accuracy–memory points. The eight reported matched-memory wins concern the post-RoPE VECTOR adaptation.

The new LongBench controls isolate the accuracy cost of post-RoPE placement for each codec. On Qwen3, the pre-to-post score differences are 0.93 points for LLC and 0.84 for VECTOR; on Llama, they are 0.09 for LLC and 3.73 for VECTOR. Thus, LLC retains nearly the same accuracy on Llama while enabling direct attention over post-RoPE codes. This supports the computational choice without implying that RoPE placement is immaterial for VECTOR.

On Qwen3, both pre- and post-RoPE LLC outperform VECTOR at the matched 14-MiB payload. On Llama, pre-RoPE VECTOR scores higher at 14 MiB, while LLC reaches higher accuracy with an 18-MiB budget: 43.00 versus 41.37. Thus, LLC's favorable operating point on Llama uses 4 MiB/layer more storage, whereas its Qwen3 advantage holds at matched memory. We will revise the abstract and conclusion to reflect this model-dependent accuracy–memory trade-off.

VECTOR Predictor fitting

We were unable to find a publicly available author implementation of VECTOR and therefore implemented the prediction mechanism described in its manuscript. Section 3.1 specifies ordinary least squares through an unpenalized squared-error objective; we found no stated ridge penalty or regularization setting in the paper. Following this specification, our per-layer K→V map uses unregularized OLS, solving accumulated normal equations in FP64 with a least-squares fallback.

To control calibration differences between codecs, we use the same calibration dataset for LLC and VECTOR, as described in the calibration appendix.

Reviewer_Q56dOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This work fits one PCA basis per layer offline, on post rope Ks and Vs from all heads. It's used as a second tier behind an eviction policy where tokens that would be evicted are instead encoded into a low-rank tier instead depending on reconstruction error. Since the basis is after rope decoding can be done into the attention so attention runs directly on the codes without reconstructing KVs. They evaluate on ruler and LongBench on a few 4 to 9B models and also offer an a100 kernel microbench and int8 codes.

Critical strengths

Attention runs directly on post rope codes and it’s implemented with a real kernel, so LLC is deployable

Critical weaknesses

important baselines missing. No INT4 KV or MiKV at matched bytes. INT4 on the full cache is ~25% of dense which is actually below LLC. The only external baseline, ie VECTOR, is run postRoPE even though its designed for normal pre RoPE. Your Table 5b shows pre-RoPE VECTOR does better in several test, seems to go against the commentary you do between the two methods..

Novelty there is a lot of work on MLA style low rank KVs, PCA encodings, and keeping evicted tokens cheaply are kinda all known (MiKV, LESS, MixedDimKV, TransMLA/MHA2MLA, several of them uncited). It's really unclear therefore why this paper should be accepted in the conference, given that both strong baselines are missing, and all these other methods not present.

Why? A few related works (very related)

  • Keeping evicted toks in a lower fidelity, aka cheaper form is present in LESS [1], MiKV, MixedDimKV [2] which stores tokens as low dim projections
  • Joint K/V latents with absorbed attention are what happens when you convert to MLA anyway [3, 4].
  • Low rank KV shared across heads / layers is in Palu and xKV [5].

Are [1–5] cited?

[1] H. Dong et al. Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference. ICML 2024. arXiv:2402.09398. [2] Beyond Token Eviction: Mixed-Dimension Budget Allocation for Efficient KV Cache Compression (MixedDimKV). arXiv:2603.20616, 2026. [3] F. Meng et al. TransMLA: Multi-Head Latent Attention Is All You Need. arXiv:2502.07864, 2025. [4] T. Ji et al. Towards Economical Inference: Enabling DeepSeek’s Multi-Head Latent Attention in Any Transformer-based LLMs (MHA2MLA). ACL 2025. arXiv:2502.14837. [5] C.-C. Chang et al. xKV: Cross-Layer SVD for KV-Cache Compression. arXiv:2503.18893, 2025.

Questions for authors
  • How does LLC compare with plain 4-bit KV quan at same memory? Also how does it compare with keeping the evicted tokens at low precision instead of low rank (as in mikv)? If LLC clearly wins both, I’d cosnider raising my score.

  • Why is Vector run on post rope keys when its designed for pre-RoPE ones and table 5 shows it does better in many cases any way? How does LLC compare with pre rope Vector at the same memory?

3 comments

Response to Q1

Thank you for the careful review and for highlighting the baseline and related-work comparisons.

We added full-cache quantization and low-precision retention experiments. The results support the storage efficiency of quantized LLC codes, while also showing that directly quantized K/V is a competitive alternative.

Evaluation protocol. Following the ValueDiff evaluation setup, our main cache-limited experiments process prompts in 128-token chunks and apply eviction and compression during both prefill and decoding. Later prompt tokens attend to retained or reconstructed history, so approximation errors can propagate into subsequent hidden states and K/V.

The full-cache INT8/INT4 controls likewise quantize new K/V on every cache append during prefill and decoding. Prefill-time error propagation may contribute to the sharp degradation of the tested plain asymmetric INT4 quantizer. The dense baseline uses unchunked prefill: without cache approximation, chunked and one-shot causal prefill are mathematically equivalent, although numerical differences can remain. We will clarify these settings in the manuscript.

Quantization study and MiKV comparison

We evaluate Qwen3-4B on RULER-8K (13 tasks × 50 examples). Tiered runs retain 2,048 resident tokens, including protected tokens, and quantize and demote throughout 128-token chunked prefill and decoding.

MiKV-style K/V uses grouped asymmetric quantization with Q/K balancing. LLC r512 codes use symmetric per-token INT8/INT4 or asymmetric INT4 per coordinate over 64-token groups. Both methods use the same per-token, per-head asymmetric quantizer for INT8 residents.

AlgorithmEviction policyResident budgetResident precisionDemoted budgetDemoted precisionNominal MiB/layerRULER score
DenseNoneFull cacheBF16——32.0095.85
Raw K/V quantizationNoneFull cacheINT8——16.5095.93
Raw K/V quantizationNoneFull cacheINT4——8.500.00
MiKVH2O2,048BF166,144INT8, grouped20.7595.80
MiKVValueDiff2,048BF166,144INT8, grouped20.7595.80
MiKVH2O2,048BF166,144INT4, grouped14.7595.65
MiKVValueDiff2,048BF166,144INT4, grouped14.7595.88
LLC r512H2O2,048BF166,144INT8, per-token11.0194.73
LLC r512ValueDiff2,048BF166,144INT8, per-token11.0195.44
LLC r512H2O2,048BF166,144INT4, per-token9.5191.69
LLC r512ValueDiff2,048BF166,144INT4, per-token9.5193.63
MiKVH2O2,048INT86,144INT8, grouped16.8895.80
MiKVValueDiff2,048INT86,144INT8, grouped16.8895.78
MiKVH2O2,048INT86,144INT4, grouped10.8895.82
MiKVValueDiff2,048INT86,144INT4, grouped10.8895.85
LLC r512H2O2,048INT86,144INT8, per-token7.1494.94
LLC r512ValueDiff2,048INT86,144INT8, per-token7.1495.26
LLC r512H2O2,048INT86,144INT4, per-token5.6490.97
LLC r512ValueDiff2,048INT86,144INT4, per-token5.6493.75
LLC r512H2O2,048INT86,144INT4, token-group5.8195.01
LLC r512ValueDiff2,048INT86,144INT4, token-group5.8195.73
LLC r512H2O2,048INT86,144INT4, token-group5.8494.89
LLC r512ValueDiff2,048INT86,144INT4, token-group5.8495.53

Memory is nominal packed storage at 8,192 retained tokens, including quantizer metadata and K-balancing overhead.

Compared with MiKV. MiKV was originally introduced with H2O; we also test ValueDiff. Under both policies, LLC trades some accuracy for substantially lower storage. Grouping each code coordinate across tokens improves INT4 quantization over using one scale per token, preserving much of the accuracy at roughly half MiKV's nominal storage. These results support combining low-rank coding with quantization to improve cache compactness.

Compared with plain per-token INT8/INT4 K/V. Plain INT8 preserves accuracy but requires more storage. Plain INT4 degrades sharply during chunked prefill, whereas quantized LLC codes retain high accuracy at smaller payloads. This suggests that LLC's latent representation tolerates low-bit quantization better than the tested plain raw-K/V representation. MiKV's strong results also show that quantizer design matters.

Together, these results support LLC as a compact, quantizable cache representation with a tunable memory–accuracy trade-off.

Why post-RoPE coding

Post-RoPE coding is an intentional computational choice: LLC absorbs its fixed linear decoder into attention, reusing the projected query across tokens and decoding values after accumulation. This enables direct attention over codes without materializing reconstructed K/V or performing position-dependent decoding.

The following LongBench comparisons use ValueDiff and match nominal memory within each row. Budgets denote resident + demoted tokens; score pairs denote pre-/post-RoPE.

ModelNominal MiB/layerLLC r512 budgetLLC pre / postVECTOR budgetVECTOR pre / post
Qwen3-4B142K+6K45.45 / 44.522K+3K43.42 / 42.58
Qwen3-4B204K+4K46.52 / 46.534K+2K46.01 / 45.85
Qwen3-4B368K+4K47.34 / 47.328K+2K46.84 / 47.13
Llama-3.1-8B142K+6K37.28 / 37.192K+3K41.37 / 37.64
Llama-3.1-8B204K+4K40.59 / 43.804K+2K40.28 / 35.78
Llama-3.1-8B368K+4K46.05 / 47.938K+2K45.79 / 43.24

At 14 MiB/layer, both LLC variants outperform both VECTOR variants on Qwen3. On Llama, post-RoPE LLC retains nearly the same accuracy as pre-RoPE LLC while enabling direct attention over codes, although pre-RoPE VECTOR remains stronger at this budget.

These LongBench results show that pre-RoPE VECTOR does not consistently outperform post-RoPE VECTOR under our cache-management framework; the advantage depends on the model and budget combination.

Response to W2

The cited works establish important ideas in KV representation, retention and budget allocation. LLC's contribution is a cache-side design combining a calibration-fitted post-RoPE codec, bounded second-tier management and direct attention over codes on an unchanged pretrained model. MiKV and Palu are already cited; we will add LESS, MixedDimKV, TransMLA, MHA2MLA and xKV and clarify their relationship to our design.

LLC separates token importance from compressibility. The base eviction policy selects the exact tier; demoted K/V states are encoded using a fixed, calibration-fitted joint all-head post-RoPE basis; and a separately bounded second tier retains codes with the lowest reconstruction residuals. Attention combines exact states and codes through a joint softmax. The post-RoPE decoder is absorbed into attention, allowing query projection once per head and value decoding after code accumulation. This avoids per-token K/V reconstruction without gradient-based training or changes to model weights or RoPE.

These works overlap with both the codec and cache-management aspects of LLC. The relevant distinctions are:

Related workRelationship to LLC
LESS [1] / MiKVLESS retains discarded information in a learned recurrent summary; MiKV retains lower-importance K/V at lower precision. LLC retains individually addressable token codes in a bounded second tier and attends directly to those codes.
MixedDimKV [2]Allocates PCA dimensions among tokens, with separate K/V projections and a head-aware variant. LLC uses a fixed joint all-head K/V code for demoted tokens and separates exact-token selection from compressed-tier admission.
TransMLA [3] / MHA2MLA [4]Convert pretrained attention to MLA and fine-tune the converted model; MHA2MLA also modifies RoPE. LLC operates on already-computed cache states, preserving the pretrained attention weights and positional encoding.
Palu / xKV [5]Palu factorizes projection weights; xKV factorizes grouped-layer caches. LLC fits a per-layer activation-space joint K/V basis and applies it to the demoted tier. Low-rank sharing and efficient execution have clear precedent in these works.

The practical value of this design is evaluated across models, base eviction policies, cache budgets and calibration-length transfer, with an implemented attention kernel. Our separate baseline response reports code quantization, MiKV-style retention and matched-memory pre-/post-RoPE VECTOR comparisons. We will revise the contribution statement and related work to explain how the codec, compressed-tier management and attention implementation work together, and support this design with the measured memory–accuracy trade-offs.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.