acceptodds
Under review as a conference paper at ICLR 2027

Recent KV Cache Optimization Strategies for Efficient Large Language Model Inference

Abstract

Autoregressive inference in Large Language Models (LLMs) is bottle necked by the memory capacity and bandwidth demands of the Key-Value (KV) cache. Although many KV cache optimization strategies have been proposed, making optimal deployment choices is difficult due to inconsistent evaluation practices and difficult to make direct comparisons. We present a systematic review and quantitative meta analysis of KV cache optimization strategies. We organized these approaches families of strategies, including quantization, token eviction, architectural reductions, and paged memory management. By extracting and normalizing the reported data from existing studies, we examined the impact of memory footprint, latency, throughput, and model quality on KV cache compression strategies. Our findings show that there is no single method that is universally superior; rather, the optimal strategy selection is highly dependent on the specific deployment constraints. We provide an evidence based recommendation matrix to guide future deployments and highlight gaps in current evaluation methods.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
KV Cache, AI, LLM, NLP, ML

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_tYdeOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper sorts out the kV cache optimization strategy and analyzes the memory, speed, and deployment constraints through published results and public experimental logs. The main work includes checking the quantization storage overhead, correcting the context length caliber, conducting pairing performance comparison, and analyzing the impact of aggregation rules and candidate configurations on the comparison results.

Critical strengths
  1. The quantitative metadata, residual cache, and input length fields are carefully checked to remind readers that the actual storage demand cannot be estimated only by the nominal number of bits.
  2. The author distinguishes between request, configuration, and independent research, and clearly states that some log indicators are not equal to end-to-end throughput, which helps to avoid improper performance interpretation.
Critical weaknesses
  1. This paper is positioned as a systematic review and quantitative meta-analysis, but the starting corpus is 17 reports. The in-depth request-level analysis mainly relies on an existing experimental product. Literature selection and independent evidence coverage are insufficient to support a broad generalization of the entire optimization field.
  2. It mainly compares the task quality that lacks synchronous measurement with the system performance, and it is unable to establish sufficient joint tradeoffs of quality, memory, and speed. Part of the findings are the further check of the existing implementation costs and existing data, and the depth of the current new knowledge is still insufficient.
  3. There are differences in the experimental conditions and index caliber of the results, and the reliability of cross-study comparison still needs to be strengthened. Different models, hardware, context length, and batch processing settings may affect performance. It is recommended to clarify which results can be directly compared, and test whether the main conclusions are robust through group analysis or sensitivity analysis.
  4. There is still a gap between the indicators used in the request-level analysis and the actual deployment performance. Cache size or local operation time-consuming cannot directly replace the first token delay and end-to-end throughput. It is recommended to clarify the conclusion range that each indicator can support, and avoid directly interpreting the change of local overhead as the improvement of overall reasoning efficiency.
Questions for authors
  1. In addition to the reports already listed, has a reproducible search and screening process been adopted? How to evaluate the coverage of the current sample for the main optimization categories?
  2. Which deployment recommendations are simultaneously supported by quality and system performance evidence under the same configuration? Which ones are merely conditional suggestions based on background literature?
Reviewer_zEcBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper surveys KV cache optimization strategies and performs a quantitative meta-analysis. It builds a taxonomy of method families, reanalyzes comparable numerical results, and audits storage accounting, paired comparisons, and aggregation sensitivity. Using KIVI, SnapKV, CaM, and related configurations, it shows that budget eligibility and method rankings depend on how storage and latency are measured. It concludes that deployment-specific trade-offs matter more than universal superiority.

Critical strengths
  1. Useful storage accounting: KIVI’s actual storage includes metadata and residuals, which affect budget eligibility.
  2. Careful pairing: model, dataset, prompt hash, run ID, and hardware/software versions are matched.
Critical weaknesses
  1. Unclear normalization: "Hardware-aware normalization" is central to the meta-analysis but is not fully described. It is unclear how A100 measurements are mapped to other hardware, or how mapping error is accounted for.
  2. Lack of theoretical contribution: The paper is an empirical review without new optimization methods or theoretical analysis. For ICLR, this may be viewed as a weakness.
Questions for authors
  1. How exactly is hardware-aware normalization performed, and how is its validity verified?
  2. Given the small number of reports, how robust are the meta-analytic conclusions?
  3. Can the meta-analysis be extended to include more artifacts, such as vLLM or TensorRT-LLM benchmark data?
  4. How sensitive are the results to the choice of ε?
Reviewer_G7mmOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a systematic review and quantitative meta-analysis of KV cache optimization strategies (quantization, token eviction, architectural reductions, paged memory management). The corpus contains 17 primary reports (9 with usable tables; 433 observations, 377 retained, 28 meeting full metadata completeness), plus a reanalysis of one public benchmark artifact (Agrawal & Mayer 2026: 28 run files, Llama-3.1-8B / Mistral-7B, KIVI-2/4, SnapKV, CaM). Findings: effective KIVI storage including metadata and residuals exceeds nominal bit-width fractions (KIVI-4 at 31.3–32.2% fails a 25% budget on all 1,200 instances); SnapKV dominates KIVI-4 on recorded cached-forward rates; aggregation-rule changes rarely alter rankings; candidate-set identity can change frontier membership. The paper concludes there is no universally superior strategy and offers a deployment decision matrix.

Critical strengths
  1. The storage accounting is a genuinely useful contribution: showing that KIVI-4’s realized storage fraction (31.3–32.2% of FP16) exceeds the nominal 25%, so that budget eligibility must be judged on total storage rather than nominal bit width, is the kind of precise deployment-relevant finding surveys rarely produce.
  2. Methodological discipline in the reanalysis: pairing only exactly matched requests, distinguishing what different aggregation rules estimate, and enumerating candidate-set sensitivity rather than hiding ambiguous blocks inside a frontier.
Critical weaknesses
  1. Title-versus-evidence mismatch. A “systematic review and quantitative meta-analysis” of the field built on 17 reports (7 contextual), 9 data-bearing papers, and 28 fully comparable observations cannot support field-level conclusions; the quantitative results are, in effect, a secondary analysis of a single existing benchmark. No systematic screening numbers (search → dedup → inclusion) or bias assessment are reported.
  2. The conclusions contradict the body’s own discipline: Section 5.1 issues three fairly universal deployment recommendations, including an unqualified “ensuring zero quality degradation” claim for paging/prefix reuse that the body’s own text conditions; the reproducibility statement mentions random-effects-model scripts while the methods state I² is not estimated.
  3. Coverage gaps: architectural reductions (MLA and variants) are absent from the quantitative analysis, and the many 2026 methods cited in the introduction never enter the evidence base, so the “recommendation matrix” rests on a narrow slice of the field.
Questions for authors
  1. Please report the full screening pipeline with numbers (records identified → screened → excluded by reason → included), and justify the 17-report corpus relative to the very large literature your own introduction cites.
  2. Which paired evidence supports each of the three Section 5.1 recommendations? Would you agree the “zero quality degradation” phrasing should be weakened to the conditional form used in Section 4.6?
  3. Equation 2 matching all 4,800 KIVI measurements to 1e-15: since the same packing implementation defines both the formula and the logs, is this anything more than a consistency check? Please clarify its evidential value.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.