acceptodds
Under review as a conference paper at ICLR 2027

CarveKV: Cross-Layer Adaptive Latent Vector Eviction for Efficient LLM Inference

Abstract

Multi-Head Latent Attention (MLA) reduces the per-token memory footprint of the KV cache, yet cache storage still grows linearly with context length. Existing token-eviction methods are largely formulated for expanded KV representations or make independent layer-wise decisions, leaving open whether eviction can be performed directly and consistently in MLA's compressed latent space. We show that latent-token importance rankings persist substantially across Transformer depth, even after controlling for causal eligibility and token position. Based on this observation, we propose CarveKV, a training-free MLA-native eviction framework that aggregates rank-normalized importance estimates across multiple layers and combines prompt-wide and final-window attention views through effective-support routing. CarveKV produces a single shared keep set across layers and scores compressed latent and decoupled-RoPE caches directly, without persistently reconstructing expanded per-head KV tensors. Across 80 natural-language and code windows, eligibility-normalized pairwise Spearman correlation reaches 0.651-0.654 and remains above 0.50 under within-position controls. On eight LongBench tasks at a matched 60% physical-cache budget, CarveKV retains 96.6% of full-cache quality. Under matched 8K-32K system settings, it also reduces prefill latency by 1.0-3.0%, with comparable decode throughput and peak memory.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multi-Head Latent Attention, KV cache compression, latent cache eviction, efficient LLM inference, long-context language models, cross-layer importance, model compression

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_idhDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors propose CarevKV, a KV-cache compression method designed for Multi-Head Latent Attention (MLA). They first show that token importance is correlated across layers and use this observation to motivate a shared token eviction strategy. They combine cross-layer importance scores with a query-dependent selection mechanism and soft eviction through token merging. Experiments show promising accuracy–memory trade-offs.

Critical strengths
  • MLA is gaining popularity, with recent state-of-the-art open models such as GLM and Kimi adopting variants of it. MLA-specific KV-cache eviction is relatively underexplored, making the problem timely and relevant.

  • The proposed method is motivated by an interesting empirical observation: token importance remains correlated across layers, even after controlling for positional effects.

  • The experimental evaluation is reasonably thorough, covering diverse tasks and model configurations, with carefully matched physical KV-cache budgets. The method achieves competitive accuracy against MLA-adapted baselines.

  • The method is training-free, which is practically appealing.

Critical weaknesses
  • The paper does not compare against existing MLA-specific KV-cache compression methods. Instead, the authors implement their own adaptations of conventional methods such as SnapKV and H2O. A comparison with an MLA-native compression approach would strengthen the evaluation.

  • A central motivation for using shared eviction decisions across layers is compatibility with Hugging Face's DynamicCache. I am more familiar with GQA, where different layers can retain different token positions without issues. I am not sure whether MLA introduces additional constraints here. Could the authors clarify why shared indices are necessary or particularly beneficial for MLA inference?

  • Some parts of the presentation seem tied to a specific model configuration without making this sufficiently explicit. For example, the key dimension of 128 in line 182 is presented without qualification. Similarly, Table 1 and the fixed layer committee in Section 3.3 seem model-specific.

  • Several design choices feel heuristic and would benefit from further ablations, particularly the effective-support-based tail allocation, the global score combining the mean and top-two layer ranks, and soft eviction through super tokens. While the tail-only baseline and single-layer comparisons provide some supporting evidence, they do not clearly isolate the contributions of these individual components.

Less important: -- Native MLA models tend to be very large, making them difficult to evaluate in an academic setting. Consequently, the authors evaluate only one native MLA model, while the other two model families are converted to MLA. This is understandable but somewhat limits the experimental evidence for generalization.

Questions for authors
  1. Can the authors clarify the practical benefits of shared indices ?

  2. Could the authors provide additional ablations to better isolate the contributions of each component of the method ?

  3. Are there some native MLA baselines to compare to ?

1 comment

Thank you for highlighting the controlled persistence results and for asking what shared indices contribute in practice.

C1. Why share token indices across layers?

As you note, DynamicCache appends independently by layer and allows different retained positions. Equal physical lengths suffice for the common decode-mask shape. Sharing indices lets one committee decision supply a common position-to-slot mapping, while each layer gathers its own latent and RoPE tensors. New tokens keep their logical positions for RoPE. The L12 and L16 shared masks yield lower PPL than independently selected layer-wise masks in this diagnostic. The execution paths differ, so the comparison is descriptive (Table 1, Appendix F).

CodeSearchNet/Python PPL at a 70% original-mid-token budget for the masked conditions

Mask selectionPPL
No eviction6.8466
Independent per layer8.2162
Shared from L127.3071
Shared from L167.6199

C2. What do the component ablations show?

At 60% physical retention, the full method has higher point estimates in all eight comparisons shown. The intervals exclude zero for tail-only, fixed 50/50 quota, L16-only and content-only scoring. The 50/50 control retains both score views and isolates input-dependent allocation.

Matched-budget ablations at 60% retention. Full CarveKV scores 22.645 on 384 examples and 23.638 on the nested 96. Gains are full minus variant. Intervals are paired and unadjusted.

VariantnScoreGain95% CI
Tail only38421.086+1.560[0.542, 2.735]
Global only38421.471+1.174[-0.111, 2.523]
Fixed 50/50 quota38421.592+1.054[0.070, 2.194]
L16 only38422.024+0.622[0.031, 1.301]
Mean ranks, no top-two38422.188+0.458[-0.167, 1.229]
No pooling38422.473+0.172[-0.767, 1.134]
Content-only scoring9621.931+1.707[0.227, 3.127]
Mean raw scores9623.416+0.223[-0.371, 0.871]

C3. Which MLA compression baselines are covered?

We use SnapKV-style and H₂O-style references to compare token-selection rules on the same MLA representation at matched physical bytes. xKV changes the cache representation by factorizing latents across layers. LCA uses learned latent condensation with query-aware pooling and positional anchors. CarveKV selects which positions to retain in a pretrained MLA cache and can merge evicted entries without retraining. We have no matched-checkpoint, matched-byte measurements against xKV or LCA. A direct comparison must count factors or condensed states and any retained RoPE keys. xKV · LCA

C4. Which settings are specific to DeepSeek-Coder-V2-Lite?

The content-key dimension of 128, the Table 1 diagnostic, and the five-layer committee refer to DeepSeek-Coder-V2-Lite. Sections 3.3 and 4.1 and Appendices O–Q specify the model-dependent settings. Table 6 separates the second native MLA backbone, MiniCPM-3-4B, from the two converted backbones.

Related Revisions

  1. Sections 3.3 and 4.1 clarify shared indices and model-specific settings.

  2. Section 4.4 and Appendices R–S report ablations. Sections 2 and 4.7 cover baselines.

Reviewer_SoA5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CarveKV, a training-free, post-prefill token eviction framework for MLA (Multi-Head Latent Attention) compressed latent caches. The authors first provide diagnostic evidence on 80 natural-language and code windows that latent-token importance persists across layers. They then design RoPE-complete MLA scoring, value-aware modulation, a rank-normalized cross-layer committee, effective-support tail routing, sink/recent protection, and super-token soft eviction.

Critical strengths
  1. CarveKV scores directly against cached latents c_j^KV and decoupled RoPE keys, using weight absorption to avoid persistently materializing expanded per-head KV. The shared mask is compatible with Hugging Face DynamicCache's layer-wise concatenation. This is a substantive MLA-native adaptation, unlike work that force-fits MHA-style eviction rules onto MLA.

  2. The authors go beyond raw Spearman correlation by adding causal-eligibility normalization and within-position-bin controls, ruling out the trivial explanation that early positions simply see more sampled queries. Within the committee, normalized Spearman reaches 0.731/0.737 and Jaccard 0.540/0.543, providing credible empirical support for a shared mask.

  3. At 60% retention it preserves 96.6% of full-cache quality; at 80% retention it reaches macro score 22.837 versus 20.372 for SnapKV-style, leading on 7 of 8 tasks while using 20% fewer physical bytes. System-wise, prefill latency drops by 1.0–4.0% versus SnapKV-style/H2O-style, with decode throughput differing by at most 0.3%.

Critical weaknesses
  1. At the fixed 60% budget, CarveKV scores 22.049 versus 22.147 for SnapKV-style MLA, i.e., 0.098 lower. The main selling point is closer to “comparable quality + cross-layer sharing + slightly better systems” rather than a quality lead, which may be insufficient for reviewers prioritizing SOTA gains.

  2. Normalized Spearman is around 0.65 and within-bin around 0.50, indicating non-trivial layer-wise ranking differences. The committee set {4,6,8,12,16}, decision layer D=16, Q_g=Q_t=512, N_s=4, N_r=16, and ρ_p=4 are all hand-chosen; the main text lacks sufficient hyperparameter sensitivity analysis, and a shared mask may lose information on some tasks or layers.

  3. The main quality conclusions rest on a fixed 60% physical budget. The \tau = 60, \tau = 90, and knee-and-coverage heuristic are explicitly labeled descriptive configurations, and \tau = 60 is not compared against an equal-average fixed budget. In practice, “how much to retain” remains heuristic, and the “adaptive” framing in the title/abstract may raise expectations that are not fully met.

  4. The main experiments concentrate on DeepSeek-Coder-V2-Lite (16B, 27 layers, d_c=512). Transfer experiments mention MiniCPM-3-4B, Qwen2.5-7B TransMLA, and Llama2-7B MHA2MLA, but detailed results in the main text are sparse. LongBench quality is evaluated with an 8192-token cap despite the model supporting 128K. Compressed NIAH recall drops from 92%/93% to 83–90%/82–91%, indicating perceptible long-context retrieval loss.

  5. The decision is made at D=16, so layers 0–16 temporarily retain the full prefix during prefill. Retained KV is the post-eviction physical size, not peak allocated GPU memory; peak prefill memory is not substantially reduced, and prefill latency drops only 1–3%. Moreover, SnapKV-style/H2O-style are in-house adaptations rather than official implementations, limiting external comparability, and there is no same-condition comparison or combination study with quantization, IndexMem, or LCA.

Questions for authors

Does CarveKV’s little complex “cross-layer committee effective-support routing” mechanism really offer necessary and robust incremental value?

1 comment

Thank you for highlighting the MLA-native scoring and positional controls, and for asking which parts of the design earn their complexity.

C1. What does adaptive routing add over a fixed split?

Effective support measures how broadly the final queries distribute attention. Concentrated attention calls for fewer tail slots, leaving more room for prompt-wide evidence. At 60% physical retention, the fixed 50/50 control keeps both score views and isolates the effect of adapting their allocation. The tail-only and global-only controls test the contribution of combining the two views (Table 3).

C2. What supports the multi-layer committee design?

The L16-only comparison supports using a committee. The evidence for specific memberships and mean-rank aggregation is weaker, with intervals that include zero. All eight ablations summarized in Tables 3 and 24–25 have lower point estimates than full CarveKV. See Appendix S for sensitivity.

Score gain of full CarveKV over selected ablations at 60%. Paired 95% intervals are unadjusted.

AblationnGain95% CI
Tail only384+1.560[0.542,2.735]
Fixed 50/50 quota384+1.054[0.070,2.194]
L16 only384+0.622[0.031,1.301]
Content-only logits96+1.707[0.227,3.127]

The 96 examples here are a subset of the 384 and separate from the confirmation set below.

C3. How does CarveKV compare with SnapKV across budgets?

At 60% on the original 800 examples, CarveKV scores 22.049 versus 22.147 for the original SnapKV-style reference. On the independent 96-example confirmation set (12 per task), default CarveKV leads smoothed SnapKV at four of five matched budgets by point estimate. Only the 80% interval excludes zero, and point estimates vary across cohorts (Table 2, Appendix T).

LongBench macro scores on the 96 confirmation examples. Intervals are paired and unadjusted.

BudgetCarveKVSmoothed SnapKVΔ95% CI
40%22.41022.284+0.125[-1.23,1.57]
50%23.06823.115-0.046[-1.85,1.63]
60%23.54222.696+0.846[-0.60,2.34]
70%22.64621.711+0.935[-1.28,2.73]
80%23.45222.176+1.276[0.31,2.31]

C4. How does the optional total-budget selector perform?

The optional selector sets total retention from prefill statistics without a task-specific ratio. On 80 test windows, suffix quality is close to a separately calibrated fixed ratio (Appendix V). C1 tests allocation within a fixed budget.

PolicyMean retainedSuffix PPL
Full cache100.000%2.786589
Adaptive76.041%2.833975
Calibrated fixed76.013%2.833166

C5. What do transfer and long-context evaluations show?

Table 6 reports transfer to native MiniCPM and two converted MLA models, with a larger PPL increase on MiniCPM. At 32K, CarveKV preserves all 80 full-cache NIAH hits, while quality drops on RepoBench-P (Section 4.5, Appendix U).

C6. How does CarveKV affect peak memory and latency?

At 16K, peak allocation falls modestly and end-to-end time rises slightly versus full cache (Appendix M).

16K systems measurements. End-to-end time includes 512 generated tokens.

MetricFull cacheCarveKV (60%)
Retained KV (MiB)486.00291.59
Peak prefill allocated (GiB)39.28939.106
End-to-end time (s)22.26622.853

Early layers hold the full prefix until the layer-16 decision. We plan to process chunks in causal order, with parallel work within each chunk. At each boundary the committee would rescore resident and incoming entries, then evict or merge entries back to budget. This bounds cache growth between decisions. Earlier eviction changes the history visible to later queries, while rescoring and compaction add overhead. We will measure peak allocation, latency and quality across chunk sizes and eviction frequencies (Section 5).

The matched-byte systems results cover the MLA-adapted baselines. Same-protocol comparisons with xKV, LCA and quantization remain open.

Related Revisions

  1. Sections 4.3–4.4 and Appendices R–T report matched-budget comparisons, ablations and sensitivity.

  2. Sections 4.5–4.7 and 5 and Appendix V cover longer inputs, transfer, systems, optional budget selection and chunked prefill.

Reviewer_DgBjOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

CarveKV proposes an eviction policy that scored compressed latents in MLA models (DS-Coder) directly and maintain the sink and latest tokens, but scored the intermediate tokens for eviction. CarveKV uses a single keep set across all the layers and scores the keep set which emphasized cross-layer persistence. The paper shows benefits of the approach for matched cache budget and also show retention with upto 60% retenion. The paper show marginal gains in some cases compared to exisiting Snap-KV style approaches.

Critical strengths

The paper motivates the use of cross layer importance persistence in token eviction and show cases high spearman correlation across layers to motivate this approach. The use of pair-wise spearman correlation to motivate identifying the committee for Training free approach is suitable for broad application. The key claim is interesting, but the results are limited to one model and short context lengths.

Critical weaknesses

The paper showcases results, mainly on a single model DS-coder 16B model which uses MLA and there is no clear evidence of scaling to other model classes and broader generalization. The results does not show improvement over baseline approach using SnapKV, and coupled with limited evaluation the claims are not strong. Overall, the paper lacks sufficient ablations and broad applicability to warrant a conference publication in its current form.

Questions for authors

Can the authors add some additional results on a larger class of MLA models to strengthen the claims. Additionally, ablation studies for longer context lengths, wider evals are needed for completeness. The Spearman coefficient approach to identify certain layers is reasonable, but it is not clear how well this scales with larger set of models and datasets, is this dataset dependent and can scale to real production usecases? How does the H2O-style and Snap-KV style implementations differ from the reference paper implementations? What are key modifications that were enable to make a fair comparison with CarveKV?

2 comments

Thank you for recognizing the motivation for training-free eviction based on cross-layer persistence and for asking where that evidence generalizes.

C1. How does CarveKV perform at longer contexts?

The same 20 RepoBench-P examples were evaluated at both lengths. At 16K, CarveKV retains 96.94% of full-cache quality on GovReport/QMSum and leads smoothed SnapKV by 2.110 points [−0.321, 4.543]. At 32K on RepoBench-P, it trails full cache by 5.600 points [−11.600, −0.499], even though it recovers all 80 full-cache NIAH hits. The synthetic probe therefore misses a loss in real code completion (Table 5, Appendix U). Brackets give paired 95% intervals.

Long-context quality at 60% physical retention for compressed methods (48 natural-language and 20 code examples)

EvaluationFull cacheCarveKVSmoothed SnapKV
16K GovReport/QMSum23.56822.84720.737
16K RepoBench-P46.55050.25046.550
32K RepoBench-P52.60047.00051.500

C2. How is the committee chosen, and does it generalize?

We find that cross-layer rank correlations remain about 0.50 in both natural language and code after controlling for causal eligibility and position. At matched bytes, CarveKV has higher point estimates than all eight variants summarized in the ablation table, including multi-layer versus L16 scoring. Four paired intervals exclude zero (Tables 3, 24–25). We set committee membership per backbone. Cross-workload calibration and production serving remain open (Section 3.3, Appendix S).

C3. How were SnapKV and H₂O adapted for a fair comparison?

We map SnapKV's per-head selection to a shared MLA token set using the final 512 prefill queries and also test width-five smoothing. Our H₂O-style adaptation replaces online heavy-hitter updates with one eviction decision based on 512 stratified prefill queries, retaining accumulation and recency protection. Matching the MLA checkpoint, inputs and post-prefill cache bytes isolates selection quality. At 60% retention on 800 examples, CarveKV and the original SnapKV-style adaptation score 22.049 and 22.147, respectively (Sections 4.1 and 4.7). SnapKV · H₂O

C4. What evidence supports transfer across MLA backbones?

The original experimental setup and appendix already covered native MiniCPM-3-4B and MLA-converted Qwen2.5-7B and Llama2-7B alongside DeepSeek-Coder-V2-Lite. Table 6 now summarizes these results. The converted models retain 83.48% and 61.97% of their caches with suffix PPL changes of −0.62% and +0.45%, respectively. MiniCPM shows a larger PPL increase. The policy transfers across these MLA backbones, with quality depending on the model and configuration.

Related Revisions

  1. Sections 4.5–4.6 and Tables 5–6 report longer-context quality and summarize cross-model transfer.

  2. Sections 3.3 and 4.7 and Appendix S clarify committee selection, sensitivity and baseline adaptation.

Thanks for the response to my questions. Some of my concerns were answered, still generalizability on datasets and real production usecases, models remains unanswered. I am inclined to maintain my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.