- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
As LLM inference is increasingly constrained by the KV cache, the authors propose QEvict, a KV-cache compression method. They first observe three properties of KV utility: long-horizon historical importance is broad; this importance is persistent but evolves gradually; and relevance to an individual query is highly concentrated. Motivated by this, the authors combine eviction, low-bit quantization, and query-conditioned sparse attention so that they complement each other. Specifically, historical KV windows are first divided into full-precision, INT2-quantized, and evicted tiers according to accumulated attention, with periodic re-routing. A compact summary then enables fast querying of the quantized windows so that only the most relevant ones are read exactly, while unselected windows are compensated through calibrated estimates. Finally, experiments on LongBench, RULER, and GSM8K show that QEvict achieves a better quality–memory trade-off than eviction and quantization baselines, especially under tight budgets, and improves batched decoding throughput.
Critical strengths
- The method is well motivated: the authors verify the three observations through preliminary experiments, giving each design choice a clear rationale.
- The authors explain in detail how real speedups are achieved in practice, with a fused kernel that reads only the selected quantized windows without materializing the full compressed cache.
- Experiments show that the method achieves strong speed and memory improvements while preserving accuracy.
Critical weaknesses
- The method is largely combinational, which limits its novelty. Accumulated-attention eviction, INT2 quantization, and query-aware window selection have all been explored in prior work, and the contribution mainly lies in combining them.
- Limited model coverage. Experiments use only two models, Llama-3.1-8B and Mistral-7B, lacking validation on larger and more recent models, as well as on newer attention mechanisms and architectures (e.g., hybrid models, Multi-head Latent Attention (MLA), and DeepSeek Sparse Attention (DSA)).
- Limited datasets. The main experiments cover only LongBench, RULER, and GSM8K, and RULER is evaluated only on Llama at 32K, lacking validation on longer contexts and long-generation tasks such as long chain-of-thought reasoning.
Questions for authors
- Does QEvict remain effective on larger and more recent models, as well as on hybrid models or models with MLA or DSA?
- How does QEvict perform on longer contexts and long-generation tasks?
6 comments
Response to Reviewer qxVs (2/4): Q1(Contd.) & Q2, DSA and Experiments on LongGenBench-16K
Q1(c): Applicability to hybrid architectures and DeepSeek Sparse Attention (DSA)
The successful MLA adaptation also provides a foundation for extending QEvict to DeepSeek Sparse Attention (DSA). While DSA uses a learned indexer to select relevant historical tokens for attention computation, QEvict addresses the complementary problem of which historical KV states to retain and at what precision. Combining the two would require coordinating DSA's indexing mechanism with QEvict's tiered storage and selective-access policy.
Similarly, for hybrid architectures, QEvict's cache-management mechanisms can potentially be applied to attention layers that maintain growing KV caches, while leaving fixed-size recurrent or linear-attention states unchanged.
We acknowledge that direct evaluations on DSA and hybrid models remain future work. Nevertheless, the additional experiments provide concrete evidence of scalability to 70B models and portability to MLA, addressing the primary concern that QEvict's effectiveness may be limited to the original GQA-based backbones.
Q2: Evaluation on Longer Generation Workloads
We thank the reviewer for highlighting the importance of evaluating QEvict beyond long-input understanding tasks. We agree that long-generation workloads present a distinct challenge: the KV cache continues growing throughout decoding, while instructions introduced early in the prompt may need to be recalled and followed thousands of tokens later.
To directly evaluate this setting, we conducted additional experiments on LongGenBench-16K, which requires approximately 16K-token generated outputs containing multiple ordered subtasks and instructions distributed across different positions in the prompt. This complements our original LongBench, RULER, and GSM8K evaluations by explicitly measuring long-horizon instruction retention and generation quality.
Experimental setting. We evaluated Llama-3.1-8B-Instruct at a 10% budget. We compare QEvict against Full-KV, KVQuant-2bit, SnapKV, and Layer-DefensiveKV.
LongGenBench reports four metrics:
- CR (Completion Rate): Percentage of required subtasks generated.
- STIC-1: Instruction satisfaction among subtasks that were generated.
- STIC-2: Instruction satisfaction across all required subtasks, counting instructions associated with missing subtasks as failed.
- wAvg: The benchmark's combined score, computed as: .
Table: LongGenBench-16K results on Llama-3.1-8B-Instruct. QEvict, SnapKV, and Layer-DefensiveKV use a 10% KV-cache budget, and KVQuant uses approximately 15% equivalent storage. Best results are highlighted in bold.
| Method | CR | STIC-1 | STIC-2 | wAvg |
|---|---|---|---|---|
| QEvict (10%) (Ours) | 86.6 | 41.5 | 27.3 | 23.64 |
| KVQuant-2bit (~15%) | 98.8 | 26.2 | 26.0 | 25.7 |
| SnapKV (10%) | 99.9 | 5.2 | 5.2 | 5.2 |
| Layer-DefensiveKV (10%) | 99.3 | 5.5 | 5.5 | 5.4 |
Long-generation performance. QEvict achieves the highest instruction-satisfaction scores across all evaluated methods. Its STIC-1 is 1.58× higher than KVQuant and approximately 8× higher than SnapKV. Furthermore, QEvict achieves the highest STIC-2, outperforming both KVQuant and eviction method, even when instructions associated with missing subtasks are counted as failures.
These results are particularly relevant to long-generation settings, where information introduced early in the prompt may only become useful much later. QEvict's mixed-precision historical retention preserves a broader set of potentially relevant instructions for subsequent query-conditioned access, whereas eviction-only methods can permanently discard information before it becomes relevant.
KVQuant achieves a higher wAvg, primarily due to its higher subtask completion rate. However, QEvict achieves higher instruction-satisfaction scores on both STIC metrics while using approximately one-third less equivalent KV memory. This highlights a meaningful trade-off between subtask completion and instruction following rather than a uniform advantage for any single compression strategy.
We have updated the manuscript accordingly, with all additions highlighted in red. Specifically, Appendix J presents detailed evaluations on Llama-3.1-70B-Instruct and DeepSeek-V2-Lite-Chat (MLA), demonstrating scalability and architectural portability. Appendix K reports our LongGenBench-16K evaluation, examining long-horizon instruction retention during extended generation.
To further address the reviewer's concern regarding reasoning-intensive workloads, we additionally conducted experiments on MATH-500. We report these results in our final response (4/4), complementing LongGenBench-16K with an evaluation of mathematical reasoning performance on the MATH-500 dataset under restricted KV-cache memory.
Response to Reviewer qxVs (3/4): W1, Novelty Discussion
W1: Novelty Beyond Combining Existing Components
We thank the reviewer for this observation. We acknowledge that accumulated-attention scoring, low-bit quantization, and query-aware selection have individually been explored in prior work. QEvict's novelty lies in decoupling historical retention, storage precision, and query-time access, and in designing the mechanisms that allow these decisions to operate jointly under a fixed memory budget.
W1(a): Methodological contributions
The key insight behind QEvict is that historical KV states need not be accessed at every decoding step to remain useful for future queries. This leads to three contributions:
-
Decoupled retention and access. Unlike eviction methods that permanently discard historical information, QEvict retains broad context in mixed-precision storage while accessing only query-relevant windows during decoding. This separates the memory required to preserve history from the memory traffic incurred by attention.
-
Recoverable precision allocation. QEvict dynamically manages full-precision (F) and INT2 (Q) tiers under an explicit byte budget. Low-priority windows are demoted rather than immediately discarded and can be promoted when their importance increases. Write-once quantization avoids repeated quantization error during tier transitions.
-
Calibrated completion for sparse access. Skipped windows may still contribute to attention and must not be treated as irrelevant. QEvict approximates their contributions through calibrated window-level completion, preserving attention accuracy and supporting importance estimation without reading every retained window.
These mechanisms address a fundamental distinction overlooked by treating retention and access as the same decision: information that is unnecessary for the current query may remain important for future queries.
W1(b): Evidence from component-wise ablations
We further examine these design choices through controlled ablations of precision allocation, quantization resolution, selective access, and completion.
Table: Ablation of QEvict's key design choices. Precision-allocation and bit-width experiments use 12 LongBench tasks at 5% equivalent KV memory. Selective-access and completion analyses use three LongBench tasks under an 8× compressed-cache setting. Lower attention error is better.
| Design choice | Configurations compared | Key results |
|---|---|---|
| Precision allocation () | FP16-only (0) / Mixed precision (0.7) / INT2-only (1) | Average: 38.7 / 42 / 41.4. Mixed precision leads on 10/12 tasks. |
| Q-tier bit width | INT2 / INT4 / INT8, at matched memory | Average: 42 / 39.5 / 23.6. INT2 beats INT4 on 10/12 and INT8 on 12/12 tasks. |
| Query-conditioned access () | 25% versus 100% of Q-tier windows | Scores: 25.90 vs. 25.96, with 31% fewer KV bytes read per step. |
| Window-level completion | Enabled versus disabled | Attention error: 0.018 vs. 0.037, a 52% reduction. |
| Completion calibration | Calibrated versus uncalibrated | Worst-head error: 0.016 vs. 0.470, approximately 29× lower. |
The results support three conclusions:
-
First, mixed-precision allocation outperforms both single-tier extremes, improving the LongBench average by 3.21 points over FP16-only and 0.53 points over INT2-only at the same memory budget.
-
Second, INT2 consistently outperforms higher-precision alternatives at matched storage, supporting the importance of historical coverage over quantization resolution.
-
Third, query-conditioned access preserves nearly the same task performance while reducing memory reads, with calibrated completion substantially improving the approximation of omitted attention contributions.
Importantly, the endpoints isolate precision-allocation choices within QEvict, rather than reproducing complete standalone eviction or quantization methods. The ablations therefore establish the value of the proposed design decisions without attributing all gains to any single component.
W1(c): Generalizability of the design
As shown in our preceding response to Q1, the same cache-management principles transfer to DeepSeek-V2-Lite-Chat with Multi-head Latent Attention (MLA). This adaptation preserves QEvict's core retention, precision-allocation, selective-access, and completion mechanisms, providing further evidence that the methodology is not specific to conventional GQA-based caching.
Together, these ablations and architectural evaluations clarify QEvict's methodological contributions beyond combining existing compression techniques. Separately, to further address the concern regarding dataset coverage, we provide additional MATH-500 experiments, complementing our earlier long-generation evaluation.
Response to Reviewer qxVs (1/4): Q1, Experiments on Llama-3.1-70B-Instruct & DeepSeek-V2-Lite-Chat(MLA)
Q1: Generalization to Larger Models and Newer Attention Architectures
We thank the reviewer for highlighting the need to evaluate QEvict beyond the models reported. To directly address this concern, we conducted additional experiments on Llama-3.1-70B-Instruct, to investigate scalability to substantially larger models, and DeepSeek-V2-Lite-Chat, to evaluate portability to Multi-head Latent Attention (MLA). Both evaluations investigate QEvict under a 10% KV-cache budget.
Q1(a): Generalization to Llama-3.1-70B-Instruct
We evaluated QEvict on all 12 LongBench tasks using Llama-3.1-70B-Instruct, comparing it against Full-KV, SnapKV, Layer-DefensiveKV, and KVQuant-2bit.
Table: LongBench results on Llama-3.1-70B-Instruct. KVQuant-2bit uses approximately 15% memory. Best results among compressed methods are highlighted in bold.
QEvict achieves the highest average score among the compressed methods, retaining 96.7% of Full-KV performance with only 10% equivalent KV memory. QEvict achieves the best result on 8 of 12 tasks among compressed methods, outperforming Layer-DefensiveKV on 9/12 tasks and SnapKV on 11/12 tasks. Notably, it also surpasses KVQuant-2bit despite using approximately one-third less equivalent KV memory.
These results demonstrate that QEvict's effectiveness extends to 70B-scale models, preserving near-Full-KV quality under a restrictive memory budget.
Q1(b): Generalization to Multi-head Latent Attention (DeepSeek-V2-Lite-Chat)
We further extended QEvict to DeepSeek-V2-Lite-Chat, a Mixture-of-Experts model employing MLA, which differs fundamentally from conventional GQA-based caching. Instead of storing separate per-head KV states, MLA maintains compressed latent representations and decoupled positional keys.
We adapted QEvict through latent-aware cache management, RoPE-aware packing, and shared latent storage. Importantly, the underlying window-ranking, precision-tier routing, byte-budget allocation, query-conditioned selection, and calibrated completion mechanisms remain unchanged. QEvict therefore operates on top of MLA's native KV compression, demonstrating that its retention-and-access strategy complements architectural cache compression.
We evaluated the adaptation on 11 LongBench tasks at a 10% KV-cache budget, comparing against SnapKV, Layer-DefensiveKV, and KVQuant-2bit. NarrativeQA was excluded because DeepSeek-V2-Lite-Chat has 32K-token context limit, which ~50% of NarrativeQA examples exceed.
Table: LongBench results on DeepSeek-V2-Lite-Chat (MLA). Best results are highlighted in bold.
QEvict achieves the highest average score, outperforming the strongest baseline by 5.61 points, and leads on 8/11 tasks. Notably, QEvict achieves the best performance on all QA and summarization tasks.
These results establish that QEvict's advantages are not restricted to GQA-based models. Its central retention-and-access methodology remains effective when applied directly to MLA's already-compressed latent representations, demonstrating both architectural portability and complementarity with existing KV-compression methods.