acceptodds
Under review as a conference paper at ICLR 2027

LazyRecall: Decision-Risk-Aware Online KV Cache Management for Long-Form Generation

Abstract

Long-form generation and extended reasoning continuously expand the key–value (KV) cache, making GPU memory a bottleneck for scalable inference. Managing a fixed GPU KV budget requires three coupled decisions: what to evict, when to evict, and what to do if an evicted KV becomes relevant again. Existing methods often select KVs using attention statistics or local removal effects, which may not provide a globally comparable measure of their impact on the final model decision. Even with a precomputed eviction order, frequent rescoring incurs overhead, while bulk eviction removes states before their capacity is needed. Moreover, once evicted states are permanently discarded, subsequent rescoring cannot recover them if their relevance returns. We introduce LazyRecall, an online KV-cache management framework that jointly addresses selection, execution, and recovery. Decision-Risk uses a shared vector–Jacobian product (VJP) to approximate each KV’s effect on the final model decision, enabling model-wide selection on a common risk scale. On-Demand Removal decouples periodic planning from on-demand execution, retaining KVs until their capacity is required. Asynchronous Recall keeps evicted states recoverable on the CPU and retrieves relevant history for subsequent re-evaluation without blocking current decoding. Experiments on long-form writing and mathematical reasoning show that LazyRecall reduces the active GPU KV cache to as little as 15.5% of FullCache while maintaining comparable generation quality, with negligible additional per-request decoding latency. Anonymous code is available at https://anonymous.4open.science/r/LazyRecall/.

Primary area:
generative models
Keywords:
KV Cache Compression, Efficient LLM Inference, Long-Context Inference

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_G6AXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents LazyRecall, an online KV-cache management framework for long-form generation under a fixed active GPU cache budget. It combines three mechanisms: (1) Decision-Risk, which uses one shared vector-Jacobian product to estimate how deleting each physical KV entry would change the confidence margin of the current top-1 token and thereby permits model-wide ranking; (2) On-Demand Removal, which periodically plans a global eviction queue but consumes it only as new cache slots are needed; and (3) Asynchronous Recall, which archives evicted KVs on CPU, retrieves candidates with per-layer/head MIPS, and re-evaluates ready candidates at later audits. Evaluation covers LongGenBench, AIME 2024, MATH-500 ablations, PG-19 diagnostics, and a single-GPU implementation with custom CUDA kernels.

Critical strengths
  1. Coherent integration of three cache-management decisions. The paper combines model-wide selection, delayed execution, and reversible eviction in one implementable framework.

  2. Substantial systems implementation. The custom CUDA path and asynchronous CPU/GPU coordination go beyond an algorithm-only study. The long-generation stress test is also informative.

Critical weaknesses
  1. The evidence does not isolate the claimed advantage of the central mechanisms. Decision-Risk uses a top-1, first-order approximation and shows only modest agreement with measured deletion risk. I could not fully interpret the restricted-oracle result because its setup is under-specified, while the PG-19 diagnostic is small and lacks uncertainty. I am also concerned by the MATH-500 result: LazyRecall exceeds FullCache by 5.6 points with one fixed seed. The progressive ablation then removes several components cumulatively, so the reported drops cannot be cleanly assigned to individual mechanisms. The complete system appears to work, but the paper does not convincingly show why.

  2. The efficiency claim uses incomplete resource and serving-level accounting. The appendix shows that recall increasingly shifts state to CPU memory and that CPU search becomes slower on longer generations. Yet the main comparison emphasizes GPU memory and average single-request throughput. I could not determine how the method behaves under concurrent requests or whether periodic audits create latency spikes. Host-memory cost and communication contention are also missing from the main comparison. In my view, the quality-efficiency trade-off is therefore only partially characterized.

Questions for authors
  1. Could the authors clarify the experimental setup behind Figure 3 and the restricted-oracle comparison? The model, candidate construction, deletion setting, and variation across examples are needed to interpret this result.

  2. How does LazyRecall compare end to end with the closest output-sensitive and delayed-eviction methods under the same GPU and CPU budgets? Please clarify any adaptations needed to place these baselines in the common setting.

  3. Do the quality and ablation conclusions hold across independent seeds? A non-progressive ablation that removes each component separately from the full system would make the attribution clearer.

  4. Do the authors have concurrent-serving results that include CPU memory and the latency distribution? It would also help to know how often retrieved states are readmitted and how long recovery takes.

Reviewer_fdB6OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3(+1)
Presentation
2
Soundness
3(+1)
Read review
Paper summary

LazyRecall manages an online KV cache under a fixed active GPU budget. A shared vector–Jacobian product estimates each physical KV slot's effect on the current top-1 decision; periodic audits form an eviction queue, while actual eviction is delayed until capacity is needed. Evicted KVs remain in a CPU archive and can be recalled asynchronously. The paper often leads the compressed-cache baselines on LongGenBench and AIME, with quality varying by metric and budget; it also reports near-FullCache single-request throughput at 10K tokens and lower peak GPU allocation on a 100K-token run.

Critical strengths
  1. Selection, eviction timing, and recovery form a coherent system. The paper distinguishes exact local attention perturbation from its approximate downstream decision-risk score.
  2. Fixed-order Bulk-versus-Lazy controls help isolate the timing decision. End-to-end comparisons cover multiple models, tasks, and active-cache budgets.
  3. The paper reports total GPU allocation, throughput, and CPU archive cost. At 100K tokens, peak GPU allocation is about 15.5 GiB versus 27.4 GiB for FullCache, while recall uses about 29.0 GiB of CPU memory. It also reports a recall-off ablation (88.2% versus 86.0% MATH-500 accuracy) and per-operation timing, so those measurements are already available.
  4. Appendix diagnostics test larger decision supports and higher-order approximations: they improve risk-ranking metrics but greatly increase scoring time, explaining the chosen top-1 first-order approximation.
Critical weaknesses
  1. The risk score has only moderate agreement with actual full-vocabulary single-KV deletion KL (Spearman ρ = 0.304). The restricted joint-deletion result does not show how often the proxy misses rare, high-impact deletions or whether those misses cause task failures.
  2. LongGenBench Once/Range accuracy is conditional on generated check positions; missing outputs appear separately under Completion. Across Tables A1–A2, LazyRecall is best or tied among compression methods in 14/16 conditional Once/Range comparisons but only 3/8 Completion comparisons. For Llama-3.1-8B at budget 2,048, its Completion is 65.26% versus 66.09% for FullCache and 69.81% for BeaconKV, despite higher conditional Once/Range scores. The source benchmark distinguishes conditional STIC-1 from STIC-2, which counts requirements for missing outputs in its denominator. An equivalent unconditional score is needed to judge complete instruction following.
  3. Being the best compression method does not always mean preserving FullCache quality. In Table A3's Qwen3.5-9B thinking runs at budget 4,096, LazyRecall scores 40.83% versus 73.33% FullCache on AIME I, and 39.17% versus 66.67% on AIME II; the gap nearly disappears at budget 8,192. Thus its quality claim is sensitive to the active-cache budget for this model and long reasoning output.
  4. The long-run GPU saving has a substantial CPU-memory cost, and the evaluation uses a single request. At 10K tokens, total GPU allocation drops from about 16.27 to 15.39 GiB, while throughput is 23.66 versus 24.62 tokens/s. CPU search and packing average about 768 ms per triggered call at 100K tokens, though much of this overlaps GPU decoding. Concurrent serving remains outside the demonstrated scope.
  5. AIME uses eight stochastic responses per problem without paired uncertainty. The MATH-500 ablation removes components cumulatively, so its rows do not identify each component's independent effect.
Questions for authors
  1. From the existing LongGenBench generations, can you report an STIC-2-style score with every required check in the denominator, as well as the fraction of tasks satisfying all requested checks? This should clarify whether the Once/Range gains survive the Completion differences.
  2. From the existing single-KV diagnostic samples, can you report high quantiles of measured deletion KL and top-1 flips among slots scored low-risk? The paper already reports mean dispersion and rank agreement; the missing tail view would test whether rare severe misses are hidden by those summaries. Any link to downstream task failures would further help interpret the proxy.
  3. Can you provide paired uncertainty for AIME, especially for the large Qwen3.5-9B budget-4,096 deficit to FullCache in Table A3, and clarify which conclusions Table D1's cumulative ablation supports directly? The recall-off row already measures recall's incremental contribution under that ablation order.
Reviewer_3eC3OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents an intelligent KV cache management strategy that improves performance vs budget trade-off. Three key points are considered, namely, the eviction criterion, the eviction timing, and the strategy for handling cache misses. Intelligent strategies are suggested for each of the above steps.

The eviction criteria is decided in terms of the decision risk measuring the effect of the token on the final decision. This is computed using the vectorised Jacobian product. The criteria is global and helps evaluate the salience of a token for the generation decision. The criteria is computed in a batch wise manner and the tokens are sorted in occasional rounds.

Eviction is performed in asynchronous intervals. This process is decoupled form the eviction score computation rounds, even though the eviction decision is taken based on the global decision risk. The timing of the eviction is planned based on the occupancy status of the cache as decided by the cache budget.

The most interesting innovation of the paper is perhaps the decoupling of the decoding in GPU with recall of the evicted tokens in the CPU memory. The evicted tokens are marked as future cache members and are recalled whenever necessary. The decoding process in GPU and fetching the evicted candidates from the CPU runs in parallel. This lazy recall strategy is an elegant method of handling cache misses in an predictive pipeline manner.

Detailed experimental results on the LongGenBench and the AIME benchmarks requiring multihop reasoning are presented. The proposed method is found to be competitive with the state of art while having a less management budget.

Critical strengths

The main strengths are:

  1. Asynchronicity of the victim identification, eviction timing and cache recall process. This provides an useful insight to the cache management strategy design.
  2. Use of the global Jacobian as salience in decision risk measure.
  3. Delayed eviction based on the cache occupancy, removing the need for frequent score recomputation and sorting and thus reducing the management overhead
  4. Decoupling the process of decoding of the current cached tokens in GPU with the prefetching of candidate cache members using predicting pipelining in the CPU.
  5. Strong experimental evidence of the effectiveness of the proposed scheme.
Critical weaknesses
  1. The global decision risk score computed using the Jacobian product has a higher overhead as compared to existing cache eviction scores. Its effectiveness across multiple instances and tasks is not guaranteed.
  2. The exploitation of the CPU memory for the LazyRecall step may effect other operations in a serving system.
  3. The effectiveness of the proposed method for other inference tasks besides long form generation is not evident from the results.
Questions for authors
  1. The rationale for using the Jacobian product score as a decision risk measure should be justified in more detail. Particularly, the reason for it being effective as a global measure.
  2. There are several hyperparameters that are used in many of the cache management steps. Justification for choosing their value may be elaborated.
  3. Nature of the performance of the individual steps of the cache management scheme under various cache budgets may be experimentally studied to get a better understanding of the trade-offs in the proposed methodology.
Reviewer_kDMeOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents a method for how the KV cache should be handled to preserve LLM performance when doing long-form generation or long reasoning under a fixed active GPU KV budget. It presents Decision-Risk, which first-order approximates how much the attention output change when a single KV is deleted changes the final prediction and turns it into a comparable score; On-Demand Removal, which, based on this, builds an eviction queue at each periodic risk audit and removes as many as needed from the queue whenever space is needed; and Asynchronous Recall, which stores evicted KVs on the CPU, retrieves them asynchronously, and adds some candidates to the evaluation candidates at the next audit; and, combining these, it proposes LazyRecall. Through this, it claims to have achieved decent performance on long-generation benchmarks and math reasoning benchmarks while reducing active KV.

Critical strengths
  1. The structure that explicitly divides KV cache management into the three decisions of what to delete, when to delete, and how to restore deleted KVs, and designs them together by appropriately combining them, is clean.
  2. The idea of measuring the effect of KV removal by the change in the final prediction rather than the local change in attention output, and estimating this for all KVs with only a single backward, is sound, and in fact, in Figure 3, the base Decision-Risk matches the measured deletion risk better than LAVa and OBCache.
Critical weaknesses
  1. The justification for the score actually used is lacking. The paper’s error analysis (Appendix C.2) and the diagnostics in Figure 3 are about the score computed with a single query at audit time. However, in practice it uses temporal risk, which multiplies the gradient of the current token by the output changes of recent queries, and what this value approximates is not explained (Appendix C.3 also mentions it only as a surrogate). For example, for a head that has almost no effect on the prediction of the audit-time token, the scores of all KVs become close to 0 even if recent queries referenced it heavily. Contrary to the intent of temporal risk (Section 3.2) to protect KVs that are important to even one of the recent queries, this causes the KVs heavily referenced by recent queries to be removed first. If such KVs are removed in large numbers within a single audit period due to global sorting, then when later tokens need to refer back to earlier content through that head, the information is already gone. Also, since recall finds candidates by the similarity between the query mean and keys but decides whether to keep them by Decision-Risk, in long documents where similar formats repeat, only duplicate KVs may be retrieved and the KVs that are actually needed may not be restored. Evidence is needed that temporal risk and recall work as intended even in such situations.
  2. I am not sure whether the gains claimed in the experiments come from the paper’s design. In Table D1, using only Decision-Risk and lazy execution gives lower performance than FullCache, and the largest performance gains come from heuristics such as max over recent queries and neighbor pooling. Although LazyRecall consistently leads across multiple settings in the main results, it is hard to trust this advantage as is with the current experimental setup. On LongGenBench, LazyRecall mostly leads in Once/Range accuracy but often falls behind in Completion, and since accuracy is scored only on the generated output, generating less can be more favorable. Also, the resource conditions relative to the baselines and the generation length settings are not clear either (I left these as questions below). Since it pays the cost of a backward during inference and CPU memory of more than twice the GPU savings, it should be clearly shown that LazyRecall has a clear performance advantage over other methods.
Questions for authors
  1. The FullCache AIME 2024 score of Qwen3.5-9B seems considerably lower than the public evaluation result of the same model (MathArena AIME 2026 92.5%), so what is the maximum generation length in the AIME and MATH-500 experiments, and what is the proportion of responses truncated by reaching the maximum length for each method?
  2. What are the detailed settings of the baselines? In particular, I am curious whether the prompt KV was included in the budget, and based on which layers the budget was computed in the hybrid architecture of Qwen3.5.
  3. The settings of the released code and the paper are quite different, and some of them seem important to the results but are not stated in the paper. For example, the code always preserves the prompt KV outside the budget, excludes the most recent 32 positions from removal, and limits the size of the CPU archive per head. With this limit, it seems difficult to get the CPU memory numbers in Table E1. Also, the admission method of recall and the default hyperparameters seem to differ from the paper as well. Please state what setting each result reported in the paper was actually obtained with.

Comments on the paper

1 comment

Thank you to the authors for the detailed response, which has addressed all of my concerns. I will raise my rating accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.