- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The authors propose CarevKV, a KV-cache compression method designed for Multi-Head Latent Attention (MLA). They first show that token importance is correlated across layers and use this observation to motivate a shared token eviction strategy. They combine cross-layer importance scores with a query-dependent selection mechanism and soft eviction through token merging. Experiments show promising accuracy–memory trade-offs.
Critical strengths
-
MLA is gaining popularity, with recent state-of-the-art open models such as GLM and Kimi adopting variants of it. MLA-specific KV-cache eviction is relatively underexplored, making the problem timely and relevant.
-
The proposed method is motivated by an interesting empirical observation: token importance remains correlated across layers, even after controlling for positional effects.
-
The experimental evaluation is reasonably thorough, covering diverse tasks and model configurations, with carefully matched physical KV-cache budgets. The method achieves competitive accuracy against MLA-adapted baselines.
-
The method is training-free, which is practically appealing.
Critical weaknesses
-
The paper does not compare against existing MLA-specific KV-cache compression methods. Instead, the authors implement their own adaptations of conventional methods such as SnapKV and H2O. A comparison with an MLA-native compression approach would strengthen the evaluation.
-
A central motivation for using shared eviction decisions across layers is compatibility with Hugging Face's DynamicCache. I am more familiar with GQA, where different layers can retain different token positions without issues. I am not sure whether MLA introduces additional constraints here. Could the authors clarify why shared indices are necessary or particularly beneficial for MLA inference?
-
Some parts of the presentation seem tied to a specific model configuration without making this sufficiently explicit. For example, the key dimension of 128 in line 182 is presented without qualification. Similarly, Table 1 and the fixed layer committee in Section 3.3 seem model-specific.
-
Several design choices feel heuristic and would benefit from further ablations, particularly the effective-support-based tail allocation, the global score combining the mean and top-two layer ranks, and soft eviction through super tokens. While the tail-only baseline and single-layer comparisons provide some supporting evidence, they do not clearly isolate the contributions of these individual components.
Less important: -- Native MLA models tend to be very large, making them difficult to evaluate in an academic setting. Consequently, the authors evaluate only one native MLA model, while the other two model families are converted to MLA. This is understandable but somewhat limits the experimental evidence for generalization.
Questions for authors
-
Can the authors clarify the practical benefits of shared indices ?
-
Could the authors provide additional ablations to better isolate the contributions of each component of the method ?
-
Are there some native MLA baselines to compare to ?
Thank you for highlighting the controlled persistence results and for asking what shared indices contribute in practice.
C1. Why share token indices across layers?
As you note, DynamicCache appends independently by layer and allows different retained positions. Equal physical lengths suffice for the common decode-mask shape. Sharing indices lets one committee decision supply a common position-to-slot mapping, while each layer gathers its own latent and RoPE tensors. New tokens keep their logical positions for RoPE. The L12 and L16 shared masks yield lower PPL than independently selected layer-wise masks in this diagnostic. The execution paths differ, so the comparison is descriptive (Table 1, Appendix F).
CodeSearchNet/Python PPL at a 70% original-mid-token budget for the masked conditions
C2. What do the component ablations show?
At 60% physical retention, the full method has higher point estimates in all eight comparisons shown. The intervals exclude zero for tail-only, fixed 50/50 quota, L16-only and content-only scoring. The 50/50 control retains both score views and isolates input-dependent allocation.
Matched-budget ablations at 60% retention. Full CarveKV scores 22.645 on 384 examples and 23.638 on the nested 96. Gains are full minus variant. Intervals are paired and unadjusted.
C3. Which MLA compression baselines are covered?
We use SnapKV-style and H₂O-style references to compare token-selection rules on the same MLA representation at matched physical bytes. xKV changes the cache representation by factorizing latents across layers. LCA uses learned latent condensation with query-aware pooling and positional anchors. CarveKV selects which positions to retain in a pretrained MLA cache and can merge evicted entries without retraining. We have no matched-checkpoint, matched-byte measurements against xKV or LCA. A direct comparison must count factors or condensed states and any retained RoPE keys. xKV · LCA
C4. Which settings are specific to DeepSeek-Coder-V2-Lite?
The content-key dimension of 128, the Table 1 diagnostic, and the five-layer committee refer to DeepSeek-Coder-V2-Lite. Sections 3.3 and 4.1 and Appendices O–Q specify the model-dependent settings. Table 6 separates the second native MLA backbone, MiniCPM-3-4B, from the two converted backbones.
Related Revisions
Sections 3.3 and 4.1 clarify shared indices and model-specific settings.
Section 4.4 and Appendices R–S report ablations. Sections 2 and 4.7 cover baselines.