acceptodds
Under review as a conference paper at ICLR 2027

QEvict: Retain Broadly, Read Selectively for Long-Context Decoding

17 views

Abstract

Long-context LLM inference is increasingly constrained by the memory and computation of the key-value (KV) cache. We find that KV utility exhibits two distinct timescales: long-horizon historical importance is broad and evolves gradually, while relevance to an individual query is much more concentrated. Motivated by this, we propose QEvict, a window-based KV-cache architecture that separates historical residency, storage representation, and current-query access. Historical windows are periodically routed under a byte budget among execution-datatype storage, compact INT2 storage, and permanent removal using accumulated attention. A write-once INT2 representation supports stable migration across storage priorities, while a compact query-conditioned representation selects only a fraction of retained low-bit windows for token-level attention. Unread windows remain represented through calibrated window-level completion. A fused mixed-precision kernel realizes selective INT2 access without materializing the full compressed cache. Across LongBench, RULER, and GSM8K with Llama and Mistral, QEvict improves the quality–memory trade-off over representative eviction, quantization, and mixed-precision baselines, with particularly strong gains under tight KV-cache budgets.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
KV Cache Compression, Large Language Model Inference, Recoverable Eviction, KV Cache Quantization, Long-Context Decoding, Attention Drift, Efficient Inference

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_qxVsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

As LLM inference is increasingly constrained by the KV cache, the authors propose QEvict, a KV-cache compression method. They first observe three properties of KV utility: long-horizon historical importance is broad; this importance is persistent but evolves gradually; and relevance to an individual query is highly concentrated. Motivated by this, the authors combine eviction, low-bit quantization, and query-conditioned sparse attention so that they complement each other. Specifically, historical KV windows are first divided into full-precision, INT2-quantized, and evicted tiers according to accumulated attention, with periodic re-routing. A compact summary then enables fast querying of the quantized windows so that only the most relevant ones are read exactly, while unselected windows are compensated through calibrated estimates. Finally, experiments on LongBench, RULER, and GSM8K show that QEvict achieves a better quality–memory trade-off than eviction and quantization baselines, especially under tight budgets, and improves batched decoding throughput.

Critical strengths
  1. The method is well motivated: the authors verify the three observations through preliminary experiments, giving each design choice a clear rationale.
  2. The authors explain in detail how real speedups are achieved in practice, with a fused kernel that reads only the selected quantized windows without materializing the full compressed cache.
  3. Experiments show that the method achieves strong speed and memory improvements while preserving accuracy.
Critical weaknesses
  1. The method is largely combinational, which limits its novelty. Accumulated-attention eviction, INT2 quantization, and query-aware window selection have all been explored in prior work, and the contribution mainly lies in combining them.
  2. Limited model coverage. Experiments use only two models, Llama-3.1-8B and Mistral-7B, lacking validation on larger and more recent models, as well as on newer attention mechanisms and architectures (e.g., hybrid models, Multi-head Latent Attention (MLA), and DeepSeek Sparse Attention (DSA)).
  3. Limited datasets. The main experiments cover only LongBench, RULER, and GSM8K, and RULER is evaluated only on Llama at 32K, lacking validation on longer contexts and long-generation tasks such as long chain-of-thought reasoning.
Questions for authors
  1. Does QEvict remain effective on larger and more recent models, as well as on hybrid models or models with MLA or DSA?
  2. How does QEvict perform on longer contexts and long-generation tasks?
6 comments
AuthorsOpenReview ↗

Response to Reviewer qxVs (1/4): Q1, Experiments on Llama-3.1-70B-Instruct & DeepSeek-V2-Lite-Chat(MLA)


Q1: Generalization to Larger Models and Newer Attention Architectures


We thank the reviewer for highlighting the need to evaluate QEvict beyond the models reported. To directly address this concern, we conducted additional experiments on Llama-3.1-70B-Instruct, to investigate scalability to substantially larger models, and DeepSeek-V2-Lite-Chat, to evaluate portability to Multi-head Latent Attention (MLA). Both evaluations investigate QEvict under a 10% KV-cache budget.


Q1(a): Generalization to Llama-3.1-70B-Instruct


We evaluated QEvict on all 12 LongBench tasks using Llama-3.1-70B-Instruct, comparing it against Full-KV, SnapKV, Layer-DefensiveKV, and KVQuant-2bit.

Table: LongBench results on Llama-3.1-70B-Instruct. KVQuant-2bit uses approximately 15% memory. Best results among compressed methods are highlighted in bold.

DatasetFull-KVQEvictSnapKVKVQuant-2bitLayer-DefensiveKV
Single-Document QA
NarrativeQA36.3142.3127.4430.9236.52
Qasper49.7340.2422.9643.5941.03
MultiFieldQA-en54.7949.1426.5051.6449.61
Multi-Document QA
HotpotQA65.7165.1147.0751.5262.87
2WikiMultihopQA68.6368.2735.5539.6461.09
MuSiQue45.9644.9730.6738.5044.54
Summarization
GovReport34.8532.5825.3631.8530.96
QMSum23.9324.1718.4923.4423.51
MultiNews26.7223.9721.2026.0923.75
Few-Shot Learning
TREC76.5074.0047.5065.5072.50
TriviaQA94.2193.7693.9089.4293.80
SAMSum47.0445.1244.8444.8744.84
Average52.0350.3036.7944.7548.75

QEvict achieves the highest average score among the compressed methods, retaining 96.7% of Full-KV performance with only 10% equivalent KV memory. QEvict achieves the best result on 8 of 12 tasks among compressed methods, outperforming Layer-DefensiveKV on 9/12 tasks and SnapKV on 11/12 tasks. Notably, it also surpasses KVQuant-2bit despite using approximately one-third less equivalent KV memory.

These results demonstrate that QEvict's effectiveness extends to 70B-scale models, preserving near-Full-KV quality under a restrictive memory budget.


Q1(b): Generalization to Multi-head Latent Attention (DeepSeek-V2-Lite-Chat)


We further extended QEvict to DeepSeek-V2-Lite-Chat, a Mixture-of-Experts model employing MLA, which differs fundamentally from conventional GQA-based caching. Instead of storing separate per-head KV states, MLA maintains compressed latent representations and decoupled positional keys.

We adapted QEvict through latent-aware cache management, RoPE-aware packing, and shared latent storage. Importantly, the underlying window-ranking, precision-tier routing, byte-budget allocation, query-conditioned selection, and calibrated completion mechanisms remain unchanged. QEvict therefore operates on top of MLA's native KV compression, demonstrating that its retention-and-access strategy complements architectural cache compression.

We evaluated the adaptation on 11 LongBench tasks at a 10% KV-cache budget, comparing against SnapKV, Layer-DefensiveKV, and KVQuant-2bit. NarrativeQA was excluded because DeepSeek-V2-Lite-Chat has 32K-token context limit, which ~50% of NarrativeQA examples exceed.

Table: LongBench results on DeepSeek-V2-Lite-Chat (MLA). Best results are highlighted in bold.

DatasetQEvictSnapKVKVQuant-2bitLayer-DefensiveKV
Single-Document QA
Qasper20.9614.3420.5412.25
MultiFieldQA-en36.2623.9530.5224.58
Multi-Document QA
HotpotQA25.7615.3917.4016.14
2WikiMultihopQA22.3321.2317.0221.38
MuSiQue14.977.865.747.20
Summarization
GovReport30.8315.8917.7715.79
QMSum22.4417.8016.6817.86
MultiNews22.7316.8422.6017.16
Few-Shot Learning
TREC64.0043.0070.5050.00
TriviaQA76.3877.0853.5477.58
SAMSum27.1725.9729.8330.21
Average33.0825.4027.4726.38

QEvict achieves the highest average score, outperforming the strongest baseline by 5.61 points, and leads on 8/11 tasks. Notably, QEvict achieves the best performance on all QA and summarization tasks.

These results establish that QEvict's advantages are not restricted to GQA-based models. Its central retention-and-access methodology remains effective when applied directly to MLA's already-compressed latent representations, demonstrating both architectural portability and complementarity with existing KV-compression methods.

AuthorsOpenReview ↗

Response to Reviewer qxVs (2/4): Q1(Contd.) & Q2, DSA and Experiments on LongGenBench-16K


Q1(c): Applicability to hybrid architectures and DeepSeek Sparse Attention (DSA)


The successful MLA adaptation also provides a foundation for extending QEvict to DeepSeek Sparse Attention (DSA). While DSA uses a learned indexer to select relevant historical tokens for attention computation, QEvict addresses the complementary problem of which historical KV states to retain and at what precision. Combining the two would require coordinating DSA's indexing mechanism with QEvict's tiered storage and selective-access policy.

Similarly, for hybrid architectures, QEvict's cache-management mechanisms can potentially be applied to attention layers that maintain growing KV caches, while leaving fixed-size recurrent or linear-attention states unchanged.

We acknowledge that direct evaluations on DSA and hybrid models remain future work. Nevertheless, the additional experiments provide concrete evidence of scalability to 70B models and portability to MLA, addressing the primary concern that QEvict's effectiveness may be limited to the original GQA-based backbones.


Q2: Evaluation on Longer Generation Workloads


We thank the reviewer for highlighting the importance of evaluating QEvict beyond long-input understanding tasks. We agree that long-generation workloads present a distinct challenge: the KV cache continues growing throughout decoding, while instructions introduced early in the prompt may need to be recalled and followed thousands of tokens later.

To directly evaluate this setting, we conducted additional experiments on LongGenBench-16K, which requires approximately 16K-token generated outputs containing multiple ordered subtasks and instructions distributed across different positions in the prompt. This complements our original LongBench, RULER, and GSM8K evaluations by explicitly measuring long-horizon instruction retention and generation quality.

Experimental setting. We evaluated Llama-3.1-8B-Instruct at a 10% budget. We compare QEvict against Full-KV, KVQuant-2bit, SnapKV, and Layer-DefensiveKV.

LongGenBench reports four metrics:

  • CR (Completion Rate): Percentage of required subtasks generated.
  • STIC-1: Instruction satisfaction among subtasks that were generated.
  • STIC-2: Instruction satisfaction across all required subtasks, counting instructions associated with missing subtasks as failed.
  • wAvg: The benchmark's combined score, computed as: CR×STIC-2/100\mathrm{CR}\times\mathrm{STIC\text{-}2}/100.

Table: LongGenBench-16K results on Llama-3.1-8B-Instruct. QEvict, SnapKV, and Layer-DefensiveKV use a 10% KV-cache budget, and KVQuant uses approximately 15% equivalent storage. Best results are highlighted in bold.

MethodCRSTIC-1STIC-2wAvg
QEvict (10%) (Ours)86.641.527.323.64
KVQuant-2bit (~15%)98.826.226.025.7
SnapKV (10%)99.95.25.25.2
Layer-DefensiveKV (10%)99.35.55.55.4

Long-generation performance. QEvict achieves the highest instruction-satisfaction scores across all evaluated methods. Its STIC-1 is 1.58× higher than KVQuant and approximately 8× higher than SnapKV. Furthermore, QEvict achieves the highest STIC-2, outperforming both KVQuant and eviction method, even when instructions associated with missing subtasks are counted as failures.

These results are particularly relevant to long-generation settings, where information introduced early in the prompt may only become useful much later. QEvict's mixed-precision historical retention preserves a broader set of potentially relevant instructions for subsequent query-conditioned access, whereas eviction-only methods can permanently discard information before it becomes relevant.

KVQuant achieves a higher wAvg, primarily due to its higher subtask completion rate. However, QEvict achieves higher instruction-satisfaction scores on both STIC metrics while using approximately one-third less equivalent KV memory. This highlights a meaningful trade-off between subtask completion and instruction following rather than a uniform advantage for any single compression strategy.

We have updated the manuscript accordingly, with all additions highlighted in red. Specifically, Appendix J presents detailed evaluations on Llama-3.1-70B-Instruct and DeepSeek-V2-Lite-Chat (MLA), demonstrating scalability and architectural portability. Appendix K reports our LongGenBench-16K evaluation, examining long-horizon instruction retention during extended generation.

To further address the reviewer's concern regarding reasoning-intensive workloads, we additionally conducted experiments on MATH-500. We report these results in our final response (4/4), complementing LongGenBench-16K with an evaluation of mathematical reasoning performance on the MATH-500 dataset under restricted KV-cache memory.

AuthorsOpenReview ↗

Response to Reviewer qxVs (3/4): W1, Novelty Discussion


W1: Novelty Beyond Combining Existing Components


We thank the reviewer for this observation. We acknowledge that accumulated-attention scoring, low-bit quantization, and query-aware selection have individually been explored in prior work. QEvict's novelty lies in decoupling historical retention, storage precision, and query-time access, and in designing the mechanisms that allow these decisions to operate jointly under a fixed memory budget.


W1(a): Methodological contributions


The key insight behind QEvict is that historical KV states need not be accessed at every decoding step to remain useful for future queries. This leads to three contributions:

  1. Decoupled retention and access. Unlike eviction methods that permanently discard historical information, QEvict retains broad context in mixed-precision storage while accessing only query-relevant windows during decoding. This separates the memory required to preserve history from the memory traffic incurred by attention.

  2. Recoverable precision allocation. QEvict dynamically manages full-precision (F) and INT2 (Q) tiers under an explicit byte budget. Low-priority windows are demoted rather than immediately discarded and can be promoted when their importance increases. Write-once quantization avoids repeated quantization error during tier transitions.

  3. Calibrated completion for sparse access. Skipped windows may still contribute to attention and must not be treated as irrelevant. QEvict approximates their contributions through calibrated window-level completion, preserving attention accuracy and supporting importance estimation without reading every retained window.

These mechanisms address a fundamental distinction overlooked by treating retention and access as the same decision: information that is unnecessary for the current query may remain important for future queries.


W1(b): Evidence from component-wise ablations


We further examine these design choices through controlled ablations of precision allocation, quantization resolution, selective access, and completion.

Table: Ablation of QEvict's key design choices. Precision-allocation and bit-width experiments use 12 LongBench tasks at 5% equivalent KV memory. Selective-access and completion analyses use three LongBench tasks under an 8× compressed-cache setting. Lower attention error is better.

Design choiceConfigurations comparedKey results
Precision allocation (α\alpha)FP16-only (0) / Mixed precision (0.7) / INT2-only (1)Average: 38.7 / 42 / 41.4. Mixed precision leads on 10/12 tasks.
Q-tier bit widthINT2 / INT4 / INT8, at matched memoryAverage: 42 / 39.5 / 23.6. INT2 beats INT4 on 10/12 and INT8 on 12/12 tasks.
Query-conditioned access (ρ\rho)25% versus 100% of Q-tier windowsScores: 25.90 vs. 25.96, with 31% fewer KV bytes read per step.
Window-level completionEnabled versus disabledAttention error: 0.018 vs. 0.037, a 52% reduction.
Completion calibrationCalibrated versus uncalibratedWorst-head error: 0.016 vs. 0.470, approximately 29× lower.

The results support three conclusions:

  • First, mixed-precision allocation outperforms both single-tier extremes, improving the LongBench average by 3.21 points over FP16-only and 0.53 points over INT2-only at the same memory budget.

  • Second, INT2 consistently outperforms higher-precision alternatives at matched storage, supporting the importance of historical coverage over quantization resolution.

  • Third, query-conditioned access preserves nearly the same task performance while reducing memory reads, with calibrated completion substantially improving the approximation of omitted attention contributions.

Importantly, the α\alpha endpoints isolate precision-allocation choices within QEvict, rather than reproducing complete standalone eviction or quantization methods. The ablations therefore establish the value of the proposed design decisions without attributing all gains to any single component.


W1(c): Generalizability of the design


As shown in our preceding response to Q1, the same cache-management principles transfer to DeepSeek-V2-Lite-Chat with Multi-head Latent Attention (MLA). This adaptation preserves QEvict's core retention, precision-allocation, selective-access, and completion mechanisms, providing further evidence that the methodology is not specific to conventional GQA-based caching.

Together, these ablations and architectural evaluations clarify QEvict's methodological contributions beyond combining existing compression techniques. Separately, to further address the concern regarding dataset coverage, we provide additional MATH-500 experiments, complementing our earlier long-generation evaluation.

Reviewer_YgSbOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper identifies two distinct timescales in KV-cache importance: historical importance is broadly distributed and evolves gradually over long horizons, whereas attention relevance is more concentrated at each decoding step. Based on this observation, QEvict decouples historical retention, storage representation, and query-time access, integrating KV eviction, low-bit quantization, mixed-precision storage, and query-conditioned sparse access into a unified cache-management framework. It organizes KV windows into three tiers, namely full-precision (F), quantized (Q), and evicted (E), and periodically reallocates them based on long-horizon importance statistics. To prevent cumulative quantization errors, QEvict uses a write-once INT2 scheme. It also maintains a compact read representation for each quantized window, enabling query-conditioned selection without reading and dequantizing the entire INT2 cache. The system further incorporates execution optimizations, including a Triton key-major kernel and fused mixed-precision decoding.

​Evaluation on LongBench, RULER, GSM8K, and other tasks demonstrates QEvict's quality–memory trade-off and inference performance. In batched decoding, it improves throughput while reducing memory usage; however, at batch size 1, query-conditioned selection overhead prevents it from improving decode latency. Overall, QEvict combines long-horizon cache management with query-time selective access, and demonstrates the benefits of this design through system-level optimizations.

Critical strengths
  1. Decoupled Retention and Query-Conditioned Access. QEvict separates long-term KV retention from query-time access, using a compact read representation to select relevant quantized windows without reading and dequantizing the entire INT2 cache. Selected windows undergo token-level attention, while unselected windows contribute through a calibrated window-level completion term and remain eligible for promotion based on updated long-horizon importance statistics. This design reduces cache-read overhead while preserving historical information that may become relevant to future queries, balancing historical coverage, attention efficiency, and memory footprint.

  2. System-Level Optimization. QEvict translates its sparse-access design into practical performance gains through a Triton key-major prefill kernel, fused mixed-precision decode kernel, and layer-major cache migration. At batch size 32 and an 8K decode length, it achieves 1.33× the decode throughput of the Full-KV FlashAttention-2 baseline while reducing peak GPU memory usage by 28.5%. These results demonstrate substantial implementation effort and strengthen the practical case for the proposed design.

Critical weaknesses

QEvict combines KV eviction, low-bit quantization, mixed-precision storage, and query-conditioned sparse access in a unified cache-management framework. Although this co-design yields a favorable quality–memory trade-off, it significantly increases system complexity. Its implementation relies on the coordination of multiple mechanisms and specialized operators and fused kernels, potentially raising the costs of implementation, maintenance, and porting to other models or inference frameworks. Furthermore, the paper's contributions primarily lie in the system-level integration and optimization of existing compression and sparse-access techniques. Although the design is complete and the engineering implementation is solid, there is still room to further clarify the general methodology and design principles that can be independently transferred or generalized for future research.

Questions for authors

Effectiveness of window-level completion. How sensitive is QEvict's model quality to the accuracy of the calibrated window-level completion term for windows that are not selected for token-level attention? Could the authors clarify how accurately it captures the contribution of skipped windows and how this approximation affects the quality–memory trade-off?

6 comments
AuthorsOpenReview ↗

Response to Reviewer YgSb (1/4): Q, Window level completion sensitivty and accuracy


Q: Accuracy and Sensitivity of Window-Level Completion


We thank the reviewer for this insightful question. To directly assess the effectiveness of window-level completion, we conducted additional experiments on Llama-3.1-8B-Instruct across three LongBench tasks (GovReport, QMSum, and MultiNews), examining:

(i) approximation accuracy,

(ii) calibration sensitivity, and

(iii) the resulting quality–memory-access trade-off.


Q(i): Accuracy of approximating skipped-window contributions


We measure the attention-output error introduced by selective reading, using exact attention over all retained KV windows (ρ=1.0\rho=1.0) as the reference. We compare selective attention with and without calibrated window-level completion at the default read ratio (ρ=0.25\rho=0.25) and a more restrictive setting (ρ=0.10\rho=0.10).

Attention-output error ↓ρ=0.25\rho=0.25 (ours)ρ=0.10\rho=0.10
Without completion0.0370.064
With calibrated completion (QEvict)0.0180.030
Error reduction52%53%

Completion reduces attention-output error by 52–53% across both read ratios. Notably, this benefit persists at ρ=0.10\rho=0.10, where 90% of Q-tier windows are skipped during token-level attention. The absolute error reduction also increases from 0.019 to 0.034 as fewer windows are read, demonstrating that completion becomes particularly valuable under more aggressive selective access.


Q(ii): Effectiveness of calibration and robustness to estimation errors


We first ablate the calibration factor to determine whether the raw window-level estimate accurately represents the attention mass of skipped windows. The uncalibrated estimate systematically underestimates this mass by approximately 32%. Our one-time calibration corrects this bias, reducing the worst single-head error (approximately 30×) and improving GovReport from 28.46 to 29.87.

We then directly assess sensitivity by deliberately scaling the calibrated completion mass or injecting noise proportional to the observed estimation error.

Completion-mass perturbationChange in LongBench score
×0.5−0.2%
×0.8+0.2%
×1.25+0.3%
×2−1.5%
×4−2.5%
Noise at 2× actual estimation error−1.0%
Noise at 4× actual estimation error−1.2%

QEvict is robust to substantial estimation inaccuracies. Scaling the completion mass between 0.5× and 1.25× changes model quality by at most 0.3%. Even when random noise is increased to 4× the actual estimation error, the score decreases by only 1.2%. These results show that calibration improves approximation accuracy, while downstream model quality remains stable under moderate estimation errors.


Q(iii): Impact on the quality–memory-access trade-off


Finally, we compare QEvict with two controlled variants: reading all retained windows (ρ=1.0\rho=1.0), which eliminates selective-attention approximation error, and selective reading without completion (ρ=0.25\rho=0.25). All three configurations use the same KV-cache storage budget, isolating the effect of selective reading and completion.

ConfigurationKV memoryGovReportQMSumMultiNewsAverage
Full-KV693.5 MB33.1023.3224.8527.09
Read all retained windows (ρ=1.0\rho=1.0)87.2 MB30.0922.9724.8225.96
QEvict (ρ=0.25\rho=0.25)87.2 MB29.8723.0624.7725.90
Without completion (ρ=0.25\rho=0.25)87.2 MB29.8322.2624.6525.58

As evident, QEvict closely matches the quality of reading every retained window while reducing KV-cache reads per decoding step. Separately, our attention-kernel profiling shows that QEvict halves attention execution latency (0.23 ms vs. 0.46 ms) compared with reading all retained windows. Moreover, completion improves all three tasks relative to selective reading without completion, increasing the average score from 25.58 to 25.90. Compared with uncompressed Full-KV, QEvict retains 95.6% of the average score using approximately 8× less KV-cache storage.

Overall, these results directly demonstrate that calibrated window-level completion (i) eliminates over half of the attention approximation error from skipped windows, (ii) remains robust to substantial estimation inaccuracies, and (iii) enables selective access to only 25% of Q-tier windows while preserving nearly the same quality as reading all retained windows, with significantly lower memory traffic and attention latency.

We have updated the manuscript accordingly, with all additions highlighted in red. Specifically, we have added a new Appendix I, with detailed experimental analyses reported in Tables 13–17. These additions evaluate the accuracy of approximating skipped-window contributions, the effectiveness of calibration, robustness to completion-mass estimation errors, and the resulting quality–memory-access trade-off.

AuthorsOpenReview ↗

Response to Reviewer YgSb (2/4): W, Design Principles and DeepSeek-V2-Lite-Chat (MLA)


W: Generalizable Design Principles, Architectural Portability, and System Complexity


We thank the reviewer for raising these important concerns. We clarify QEvict's transferable methodological contributions, generalization across architectures and model scales, and implementation portability, supported by additional experiments on DeepSeek-V2-Lite-Chat (MLA) and Llama-3.1-70B-Instruct.


W(i): What are QEvict's generalizable methodological contributions beyond system-level integration?


The central insight of QEvict is that retaining historical KV information and accessing it during decoding are distinct optimization problems. A retained KV state need not participate in attention at every decoding step. QEvict builds on this distinction through three design principles:

  • Decoupled retention and access. QEvict preserves broad historical context in compressed storage while reading only query-relevant windows, separating KV-cache capacity from decoding-time memory traffic.

  • Coverage-aware precision allocation. Under a fixed byte budget, QEvict prioritizes high-importance states for full-precision storage while retaining broader history at low precision, balancing historical coverage and numerical fidelity.

  • Calibrated completion for skipped windows. QEvict approximates the aggregate attention contributions of unselected windows without reading their token-level KV states, improving attention accuracy under selective access.

Thus, QEvict's methodological contribution is not merely combining eviction, quantization, and sparse attention, but decoupling historical retention from query-time computation while accounting for omitted attention contributions. These principles are transferable across KV-cache representations and attention architectures, as supported by our additional MLA experiments.


W(ii): Does QEvict generalize across different attention architectures and model scales?


To evaluate whether QEvict's methodology extends beyond the original model configurations, we conducted two additional experiments: porting QEvict to Multi-head Latent Attention (MLA) and evaluating its effectiveness on a 70B-parameter model.

(a) Portability to Multi-head Latent Attention: DeepSeek-V2-Lite-Chat.

We extended QEvict to DeepSeek-V2-Lite-Chat, a 15.7B-parameter Mixture-of-Experts model. Unlike Llama's Grouped-Query Attention (GQA), DeepSeek-V2-Lite-Chat employs Multi-head Latent Attention (MLA), which stores compressed latent representations and decoupled RoPE keys rather than conventional per-head KV states. This presents a substantially different cache representation and provides a direct test of QEvict's architectural portability. We adapted QEvict to operate on these latent representations through latent-aware cache management, RoPE-aware packing, and shared latent storage, while keeping the core QEvict logic unchanged, that is window ranking, tier routing, budget allocation, query-conditioned selection, and calibrated completion.

Thus, QEvict compresses and selectively accesses the cache on top of MLA's existing architectural compression, rather than replacing or bypassing it.

We evaluate this adaptation on 11 LongBench tasks at a 10% KV-cache budget. NarrativeQA is excluded because DeepSeek-V2-Lite-Chat has a native context limit of 32K tokens, while approximately 50% of NarrativeQA samples exceed this limit.

Table: LongBench results on DeepSeek-V2-Lite-Chat (MLA). Best results are highlighted in bold.

DatasetQEvict (10%)SnapKV(10%)KVQuant-2bit(~15%)Layer-DefensiveKV(10%)
Single-Document QA
Qasper20.9614.3420.5412.25
MultiFieldQA-en36.2623.9530.5224.58
Multi-Document QA
HotpotQA25.7615.3917.4016.14
2WikiMultihopQA22.3321.2317.0221.38
MuSiQue14.977.865.747.20
Summarization
GovReport30.8315.8917.7715.79
QMSum22.4417.8016.6817.86
MultiNews22.7316.8422.6017.16
Few-Shot Learning
TREC64.0043.0070.5050.00
TriviaQA76.3877.0853.5477.58
SAMSum27.1725.9729.8330.21

QEvict achieves the highest average score of, outperforming the strongest baseline (KVQuant-2bit) by 5.61 points, and leads on 8 of 11 tasks. Notably, QEvict achieves the best results on all five QA tasks and all three summarization tasks.

These results are important because MLA already compresses the KV representation architecturally. QEvict's consistent gains demonstrate that broad retention and query-conditioned access provide benefits even when applied to an already-compressed latent cache. This directly supports the generality and complementarity of our design principles beyond conventional GQA-based models.

AuthorsOpenReview ↗

Response to Reviewer YgSb (3/4): W(Contd.), Llama-3.1-70B-Instruct and System complexity


(b) Scalability to Llama-3.1-70B-Instruct.

We further evaluated QEvict on Llama-3.1-70B-Instruct to investigate whether its retention-and-access strategy remains effective at substantially larger model scale. Compared with the 8B model, the 70B backbone introduces significantly greater model depth and a different query to KV-head grouping, increasing the importance of carefully managing historical KV states under constrained memory.

We evaluated all 12 LongBench tasks at a 10% equivalent KV-cache budget, comparing QEvict against Layer-DefensiveKV, SnapKV and KVQuant-2bit.

Table: LongBench results on Llama-3.1-70B-Instruct. QEvict, SnapKV, and Layer-DefensiveKV use 10% equivalent KV-cache memory, while KVQuant-2bit uses approximately 15%. Full-KV uses 100% memory. Best results among compressed methods are highlighted in bold.

DatasetFull-KV (100%)QEvict (Ours)SnapKVKVQuant-2bitLayer-DefensiveKV (10%)
Single-Document QA
NarrativeQA36.3142.3127.4430.9236.52
Qasper49.7340.2422.9643.5941.03
MultiFieldQA-en54.7949.1426.5051.6449.61
Multi-Document QA
HotpotQA65.7165.1147.0751.5262.87
2WikiMultihopQA68.6368.2735.5539.6461.09
MuSiQue45.9644.9730.6738.5044.54
Summarization
GovReport34.8532.5825.3631.8530.96
QMSum23.9324.1718.4923.4423.51
MultiNews26.7223.9721.2026.0923.75
Few-Shot Learning
TREC76.5074.0047.5065.5072.50
TriviaQA94.2193.7693.9089.4293.80
SAMSum47.0445.1244.8444.8744.84
Average52.0350.3036.7944.7548.75

QEvict achieves the highest average score of 50.30 among the compressed methods, outperforming Layer-DefensiveKV by 1.55 points, KVQuant-2bit by 5.55 points, and SnapKV by 13.51 points. It achieves the best result on 8 of 12 tasks among the four compressed methods and outperforms Layer-DefensiveKV on 9 of 12 tasks and SnapKV on 11 of 12 tasks. Notably, QEvict achieves these improvements using only 10% equivalent KV memory, compared with approximately 15% for KVQuant-2bit, while retaining 96.7% of Full-KV performance.

Together, the DeepSeek-V2-Lite-Chat and Llama-3.1-70B-Instruct experiments provide complementary evidence of QEvict's generalizability. The former demonstrates its effectiveness on MLA's compressed latent-attention architecture, while the latter confirms that its advantages over eviction- and quantization-based baselines persist at 70B model scale.


W(iii): How does QEvict's complexity affect implementation and portability?


Although QEvict coordinates multiple mechanisms, its implementation is modular and builds on established inference-engine abstractions. Window metadata maps to paged KV blocks, mixed-precision tiers to quantized cache pools, tier migration to block management, query-conditioned selection to indexed sparse access, and calibrated completion to an attention-kernel epilogue. The fused decode operator is implemented in Triton.

Thus, the additional complexity is primarily localized to cache management and the decode-attention kernel, without requiring changes to model training or the underlying transformer architecture. These abstractions provide a practical integration path for inference frameworks such as vLLM and SGLang.

Furthermore, our anonymized repository has been updated with the complete MLA adaptation, including the latent-aware cache interface, fused kernel, evaluation scripts, and correctness tests.

We have updated the manuscript accordingly, with all additions highlighted in red. Specifically, Appendix J presents QEvict's generalizable design principles, detailed evaluations on Llama-3.1-70B-Instruct and DeepSeek-V2-Lite-Chat (MLA), and implementation complexity.

To further complement these evaluations, we conducted additional experiments on MATH-500 to assess mathematical reasoning performance under restricted KV-cache memory. We present these results in our next response, providing additional evidence of QEvict's effectiveness across different downstream workloads.

Reviewer_pqJYOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

QEvict is a window-based KV-cache architecture that combines eviction and quantization methods to improve the quality-memory trade-off. Historical context is partitioned into fixed-size windows (W=8 tokens by default) and can be categorized into 3 tiers: full-precision tier (F), INT2 quantized tier (Q), or permanently evicted (E). Ranking and routing is triggered after prefill and every R decode steps (R=8 by default) based on accumulated attention mass. Note that the contribution from both selected and unselected windows is taken into account during importance scoring.

Experiments cover 3 benchmarks (LongBench, RULER, and GSM8K) on 2 models (Llama-3.1-8B-Instruct and Mistral-7B-Instruct-v0.2) against 6 eviction baselines (StreamingLLM, SnapKV, AdaKV, CriticalKV, DefensiveKV, Layer-DefensiveKV), 2 quantization baselines (KIVI, KVQuant), and 3 mixed-precision baselines (ZipCache, MiKV, ResQ). When compared with eviction methods at matched budget, QEvict stands out among others (Table 1 and 2) because ~70% of the budget is used to store INT2 windows, which allows QEvict to retain more history than other pure eviction methods. Similarly, when compared with pure quantization methods, because QEvict preserves some windows in full precision, it can outperform most of the competitors at a fixed KV budget. A fused mixed-precision Triton kernel implements the proposed method end-to-end without materializing the full attention matrix or a dequantized Q tier. QEvict demonstrated a 1.33× decode-throughput speedup over Full-KV with FlashAttention, using a batch size of 32, a 256-token prefill, and 8K decode.

Critical strengths
  1. Broad experimental coverage. 3 benchmarks (LongBench, RULER, GSM8K), 2 backbones, and 11 baselines across both compression families the paper studies (eviction, and quantization/mixed-precision), with rigorous byte-level budget accounting.
  2. Stable hyperparameters, confirmed by ablation. Sweeping window size, Q-tier allocation, and read ratio each produces only small accuracy swings, and the paper reports the negative result that increasing read ratio beyond 0.25 gives no consistent gain — a reasonable, non-brittle default configuration.
  3. Real kernel implementation with hardware-measured speedup. A fused mixed-precision Triton kernel realizes the method without materializing the full attention matrix or a dequantized Q tier, backed by measured batch-32 throughput/memory (429.3 tok/s, 34.70 GiB) and explicit OOM comparisons against baselines, not simulated savings.
Critical weaknesses
  1. Untested design extremes, and a loss to most quantization baselines at the default. As a hybrid of eviction and quantization, QEvict should approach the best of each world at its own design's extremes — but these extremes are never tested, and at the interior default it loses to most quantization baselines on GSM8K. At α=1, QEvict's historical budget would go entirely to the INT2 Q tier, reducing to something close to a pure quantization method; one would expect it to then match top quantization baselines like KIVI or KVQuant closely. But the α ablation (Table 9) only sweeps {0.10, 0.30, 0.50, 0.70, 0.90} — neither α=0 (pure full-precision retention) nor α=1 (pure quantization) is ever run, so this expected degenerate-case behavior is untested. At the actual default (α=0.70), GSM8K results (Tables 6b/7b) already show the gap this predicts: QEvict trails four of the six quantization/mixed-precision baselines on both backbones, beating only the two weakest. The main text's framing ("quantization methods remain strong... illustrating a complementary trade-off between broad adaptive retention and uniformly quantized storage," p.9) does not explain this discrepancy, and no experiment or discussion is given to clarify whether QEvict recovers competitive quantization-only performance at that extreme.
  2. No engagement with compaction, a third popular family of KV-cache compression. The paper frames its comparisons around only two families — eviction, and quantization/mixed-precision (line 1965) — leaving out compaction/context-reconstruction methods (e.g., KVzip), which synthesize new representations that themselves participate in attention. (This differs from QEvict's own "compact representation," a routing signal discarded after selection, never a substitute for real token-level K/V.) No baseline from this family is compared against, and the paper never discusses whether compaction is a competing or complementary option to its own design.
  3. The Q tier is fixed at INT2 with no stated reasoning, and no exploration of the resolution-vs-coverage trade-off this implies. The paper adopts INT2 for the Q tier throughout — the quantization grid, the write-once migration record, and the entire W/α/ρ ablation (Appendix G) — without discussing why this precision was chosen over a higher-bit alternative. This becomes a natural question once KVQuant-3b is seen beating QEvict's INT2-based default on GSM8K (Tables 6b/7b): since QEvict's byte budget trades window count against per-window precision (Eq. 2), would storing fewer windows at higher resolution help or hurt the overall quality-memory trade-off? The paper never tests or discusses this.
  4. The paper's own description of its batch-1 result ("does not improve latency") understates an actual ~50% slowdown. Section 5.1 states that "at batch size 1, the compact Q-tier scan and selection overhead are not fully amortized, so QEvict does not improve latency over optimized Full-KV" — phrasing that reads as a neutral wash. The actual numbers (Table 11) show QEvict at 0.0678s TPOT versus 0.0451s for Full-KV-flash at batch 1, 8K decode: QEvict is roughly 50% slower per token, not merely flat. No intermediate batch sizes are reported, so the onset point at which the QEvict kernel becomes advantageous over Full-KV remains unidentified.
Questions for authors

please see Weaknesses above

9 comments
AuthorsOpenReview ↗

Response to Reviewer pqJY (1/4): W1 & W2, Boundary configuration and Compaction


W1: Evaluation of Boundary Configurations (α=0\alpha=0 and α=1\alpha=1)


We thank the reviewer for the constructive suggestion to evaluate the boundary allocations of the Q-tier budget and for raising the comparison with quantization baselines on GSM8K. We address both points below.


W1(a): Boundary-case evaluation at α=0\alpha=0 and α=1\alpha=1


To directly address the missing extremes in Table 9, we performed additional experiments at α=0\alpha=0 (FP16-only historical tier) and α=1\alpha=1 (INT2-only historical tier) on 12 LongBench tasks using Llama-3.1-8B-Instruct at a 5% equivalent KV-cache memory budget. We compare these runs with the reported default α=0.70\alpha=0.70 results from Table 2, under the same memory budget. The best score in each row is highlighted in bold.

Datasetα=0\alpha=0 (FP16 only)α=0.70\alpha=0.70 (Ours)α=1\alpha=1 (INT2 only)
NarrativeQA27.7527.8027.67
Qasper23.3330.9029.17
MultiFieldQA-en36.7147.4044.63
HotpotQA53.3657.3056.43
2WikiMultihopQA42.9445.7045.39
MuSiQue29.4332.5031.48
GovReport25.1826.0026.66
QMSum22.8623.7023.36
MultiNews18.5723.1019.75
TREC59.5067.5064.50
TriviaQA87.7689.8089.17
SAMSum38.1332.3039.39

Our hybrid allocation (α=0.70\alpha=0.70) achieves the highest score on 10 of the 12 tasks, outperforming both boundary configurations on those tasks. In particular, it leads on all six single- and multi-document QA tasks, as well as QMSum, MultiNews, TREC, and TriviaQA. Relative to the all-INT2 extreme (α=1\alpha=1), the default gains 3.35 points on MultiNews, 3.00 on TREC, 2.77 on MultiFieldQA-en, and 1.73 on Qasper. It also outperforms the FP16-only extreme (α=0\alpha=0) on 11 of 12 tasks.

These results answer the asked boundary-case question directly, that neither pure FP16-tier retention nor an entirely INT2 historical tier is uniformly optimal.


W1(b): Why α=1\alpha=1 is not a KIVI/KVQuant-equivalent mode?


As Eq. (2) specifies, α\alpha controls only the division of the historical byte budget between the F and Q tiers. At α=1\alpha=1, the F-tier historical allocation becomes zero, but QEvict still performs attention-based window ranking, budget-constrained eviction, query-conditioned selection of the retained INT2 windows, and calibrated completion for windows not selected for token-level attention, also protected sink and recent states are also handled separately.

Thus, the α=1\alpha=1 endpoint is an all-INT2 historical-storage configuration of QEvict, not a reproduction of KIVI or KVQuant. Those methods use different quantization and cache-access mechanisms, so matching their GSM8K scores is not a degeneracy property implied by setting α=1\alpha=1.


W1(c): Clarification of the GSM8K performance gap


We agree that some quantization baselines achieve higher GSM8K accuracy at approximately 20% KV memory. Importantly, QEvict still outperforms every evaluated eviction baseline at that budget on both backbones, while its design additionally limits token-level reads to a selected fraction of the Q tier (ρ=0.25\rho=0.25 by default).

This makes the GSM8K result a task-dependent accuracy versus selective-access trade-off, rather than evidence that the α=1\alpha=1 endpoint ought to reproduce a particular quantizer. We will clarify this distinction in the paper and avoid implying universal accuracy dominance over quantization baselines.


W2: Comparison with KV-Cache Compaction Methods


We thank the reviewer for highlighting the importance of KV-cache compaction methods. We agree that the original manuscript did not sufficiently discuss this family of approaches. To address this concern, we conducted additional experiments with four representative methods: GRKV (EMNLP 2026) [1], CaM (ICML 2024) [2], D2O (ICLR 2025) [3], and AsymKV (NeurIPS 2025) [4].


W2(a): Clarification regarding KVzip and baseline selection


While KVzip [5] uses context reconstruction to estimate KV importance, it ultimately performs KV-pair eviction rather than merging or reconstructing replacement KV representations. Therefore, to directly address the reviewer's concern regarding representation compaction, we selected GRKV, CaM, D2O, and AsymKV, which incorporate KV merging, compensation, or representation transformation.


W2(b): Additional comparison with compaction methods


We evaluated these four methods on Llama-3.1-8B-Instruct across 12 LongBench tasks, under 10% and 20% KV-cache budgets. For a consistent comparison, the evaluated compaction variants use CriticalKV as the underlying eviction backbone. The results cover single-document QA, multi-document QA, summarization, and few-shot learning.

AuthorsOpenReview ↗

Response to Reviewer pqJY (2/4): W2(contd.), Compaction Baselines


Table: LongBench results at 10% KV-cache budget (Llama-3.1-8B-Instruct). The best result in each row is highlighted in bold.

DatasetQEvict(Ours)GRKVCaMD2OAsymKV
Single-Document QA
Nar.QA28.9027.1119.6420.7819.07
Qasper35.0024.5220.9416.2123.18
Mul.QA50.9024.0329.6822.8327.39
Multi-Document QA
Hot.QA58.2050.8449.7336.2538.82
2Wi.QA47.5034.0324.7625.5625.21
MuSiQue32.6024.7825.0817.6710.16
Summarization
Gov.Re.32.2023.6021.6420.0727.47
QMSum25.4018.4823.5015.2020.07
M.News24.2021.3717.9220.2017.56
Few-Shot Learning
TREC68.0037.4427.0227.8916.92
Tri.QA90.6080.8583.3273.2378.78
SAMSum40.8042.0635.1835.6726.31

Table: LongBench results at 20% KV-cache budget (Llama-3.1-8B-Instruct). The best result in each row is highlighted in bold.

DatasetQEvict(Ours)GRKVCaMD2OAsymKV
Single-Document QA
Nar.QA29.9028.9429.0629.5223.51
Qasper40.7028.3533.1326.7632.46
Mul.QA55.5038.5038.3437.5833.63
Multi-Document QA
Hot.QA58.0052.4555.9149.5243.67
2Wi.QA48.8040.3538.5532.8639.54
MuSiQue32.6021.0925.2011.9210.30
Summarization
Gov.Re.33.6028.6532.8918.3129.92
QMSum25.8025.4824.8117.7019.36
M.News26.9019.3523.7115.3925.86
Few-Shot Learning
TREC71.5052.0519.5335.4710.72
Tri.QA91.0078.8177.9580.7287.97
SAMSum42.0041.0639.5136.1921.16

The results demonstrate that QEvict outperforms all four evaluated compaction variants on 11 of 12 tasks at 10% memory and all 12 tasks at 20% memory, achieving the best performance in 23 of 24 task–budget comparisons.


W2(c): Relationship to compaction methods


The two approaches are architecturally complementary. Compaction methods primarily reduce information loss by merging or transforming KV representations, whereas QEvict decouples historical retention from query-time access, maintaining broad historical coverage through mixed-precision storage while selectively reading query-relevant INT2 windows. Unlike reconstructed KV entries, QEvict's compact window representations serve as selection summaries rather than substitutes for token-level K/V.

This distinction also suggests a potential extension in which merged KV representations could be incorporated into QEvict's historical storage hierarchy while retaining its query-conditioned selective-access mechanism.

We have updated the manuscript accordingly, with all additions highlighted in blue. Specifically, we have expanded the Related Work section(Section-2) to discuss GRKV, CaM, D2O, and AsymKV, highlighting their architectural distinctions and complementarity with QEvict. We have also added a new comparison table(Table-3) and accompanying discussion in the Experiments section(Section-5), reporting results across all 12 LongBench tasks at 10% and 20% KV-cache budgets. These additions provide a more comprehensive evaluation against representation-compaction methods and further substantiate QEvict's effectiveness.


References

[1] Junjie Peng et al. GRKV: Global Regression for Training-Free KV Cache Compression in Long-Context LLMs. Proceedings of EMNLP, 2026.

[2] Yuxin Zhang et al. CaM: Cache Merging for Memory-efficient LLMs Inference. Proceedings of the 41st International Conference on Machine Learning (ICML),2024.

[3] Zhongwei Wan et al. D2O: Dynamic Discriminative Operations for Efficient Long-Context Inference of Large Language Models. International Conference on Learning Representations (ICLR), 2025.

[4] Wanyun Cui et al. Homogeneous Keys, Heterogeneous Values: Exploiting Local KV Cache Asymmetry for Long-Context LLMs. Advances in Neural Information Processing Systems (NeurIPS), 2025.

[5] Kim et al. KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction. Advances in Neural Information Processing Systems (NeurIPS), 2025.

AuthorsOpenReview ↗

Response to Reviewer pqJY (3/4): W3, INT4 and INT8 precision ablation


W3: Justification of INT2 Precision and Resolution–Coverage Trade-off


We thank the reviewer for raising this important point. Our choice of INT2 is motivated by QEvict's central observation that historical KV importance is broadly distributed. Under a fixed memory budget, higher-bit quantization improves numerical precision but reduces the number of historical windows that can be retained (Eq. 2). We selected INT2 to prioritize historical coverage while preserving high-importance windows in full precision.


W3(a): Additional precision ablation (INT2 vs. INT4 vs. INT8)


To directly investigate this trade-off, we conducted additional experiments using INT4 and INT8 Q-tier storage on Llama-3.1-8B-Instruct across 12 LongBench tasks, under the 5% equivalent KV-cache memory budget. We compare these results with the default INT2 configuration reported in Table 2.

Table: Effect of Q-tier precision on LongBench at 5% equivalent KV-cache memory. The best result in each row is highlighted in bold.

DatasetINT2 (Ours)INT4INT8
NarrativeQA27.8027.5615.54
Qasper30.9026.8623.93
MultiFieldQA-en47.4039.6627.95
HotpotQA57.3055.8730.47
2WikiMultihopQA45.7043.9236.54
MuSiQue32.5029.5511.91
GovReport26.0021.933.14
QMSum23.7023.7714.78
MultiNews23.1016.0218.15
TREC67.5062.0037.50
TriviaQA89.8088.4744.35
SAMSum32.3038.6019.54
Average42.0039.5223.65

W3(b): Resolution–coverage trade-off


The results show that INT2 outperforms INT4 on 10/12 tasks and INT8 on all 12 tasks, achieving the highest average score of 42.00, compared with 39.52 (INT4) and 23.65 (INT8). Although INT4 performs better on SAMSum and marginally on QMSum, these gains do not generalize across the benchmark.

These findings directly support our design choice, that retaining broader historical context with INT2 generally provides greater benefits than storing fewer windows at higher precision. Furthermore, QEvict's mixed-precision hierarchy already preserves high-priority KV states in full precision, reducing the need for higher-bit storage throughout the Q tier.


W3(c): Relation to GSM8K


The observation regarding KVQuant-3b on GSM8K is valid, however, KVQuant and QEvict differ in their quantization, retention, and query-time access mechanisms. Consequently, this comparison does not isolate the effect of Q-tier bit width. Our additional experiments directly evaluate the precision–coverage trade-off within QEvict at matched memory, establishing the advantage of INT2 in the evaluated LongBench setting.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.