- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents LazyRecall, an online KV-cache management framework for long-form generation under a fixed active GPU cache budget. It combines three mechanisms: (1) Decision-Risk, which uses one shared vector-Jacobian product to estimate how deleting each physical KV entry would change the confidence margin of the current top-1 token and thereby permits model-wide ranking; (2) On-Demand Removal, which periodically plans a global eviction queue but consumes it only as new cache slots are needed; and (3) Asynchronous Recall, which archives evicted KVs on CPU, retrieves candidates with per-layer/head MIPS, and re-evaluates ready candidates at later audits. Evaluation covers LongGenBench, AIME 2024, MATH-500 ablations, PG-19 diagnostics, and a single-GPU implementation with custom CUDA kernels.
Critical strengths
-
Coherent integration of three cache-management decisions. The paper combines model-wide selection, delayed execution, and reversible eviction in one implementable framework.
-
Substantial systems implementation. The custom CUDA path and asynchronous CPU/GPU coordination go beyond an algorithm-only study. The long-generation stress test is also informative.
Critical weaknesses
-
The evidence does not isolate the claimed advantage of the central mechanisms. Decision-Risk uses a top-1, first-order approximation and shows only modest agreement with measured deletion risk. I could not fully interpret the restricted-oracle result because its setup is under-specified, while the PG-19 diagnostic is small and lacks uncertainty. I am also concerned by the MATH-500 result: LazyRecall exceeds FullCache by 5.6 points with one fixed seed. The progressive ablation then removes several components cumulatively, so the reported drops cannot be cleanly assigned to individual mechanisms. The complete system appears to work, but the paper does not convincingly show why.
-
The efficiency claim uses incomplete resource and serving-level accounting. The appendix shows that recall increasingly shifts state to CPU memory and that CPU search becomes slower on longer generations. Yet the main comparison emphasizes GPU memory and average single-request throughput. I could not determine how the method behaves under concurrent requests or whether periodic audits create latency spikes. Host-memory cost and communication contention are also missing from the main comparison. In my view, the quality-efficiency trade-off is therefore only partially characterized.
Questions for authors
-
Could the authors clarify the experimental setup behind Figure 3 and the restricted-oracle comparison? The model, candidate construction, deletion setting, and variation across examples are needed to interpret this result.
-
How does LazyRecall compare end to end with the closest output-sensitive and delayed-eviction methods under the same GPU and CPU budgets? Please clarify any adaptations needed to place these baselines in the common setting.
-
Do the quality and ablation conclusions hold across independent seeds? A non-progressive ablation that removes each component separately from the full system would make the attribution clearer.
-
Do the authors have concurrent-serving results that include CPU memory and the latency distribution? It would also help to know how often retrieved states are readmitted and how long recovery takes.
Thank you to the authors for the detailed response, which has addressed all of my concerns. I will raise my rating accordingly.