acceptodds
Under review as a conference paper at ICLR 2027

KVReclaim: Recompute-Aware Placement and Migration-Free Reclamation for KV Caches on ZNS SSDs

Abstract

Large language model (LLM) inference increasingly relies on long contexts and persistent conversation histories, causing KV caches to grow rapidly and become a major bottleneck for scalable serving. SSD-backed KV caching provides a cost-effective means of extending scarce GPU and host memory, but conventional SSD accesses incur substantial KV recovery latency, especially as device-managed garbage collection (GC) induces data migration that can interfere with foreground KV I/O. Zoned Namespace (ZNS) SSDs provide an opportunity to eliminate such unnecessary migration by exposing data placement and space reclamation to the host. Based on this insight, we present KVReclaim, a host–SSD coordinated KV-cache management framework for ZNS SSDs that exploits KV semantics across the storage hierarchy. KVReclaim manages KV states at conversation granularity and flushes them into zones according to their runtime access behavior and recomputation overhead. During garbage collection (GC), it directly discards selected KV states instead of migrating them, thereby reducing GC-induced data movement. KVReclaim further dynamically selects between SSD-backed KV recovery and recomputation by comparing their estimated costs, thereby reducing recovery overhead under varying SSD I/O conditions. Experiments on long-context and multi-turn LLM workloads show that KVReclaim reduces E2E latency by 74.2% on average compared with state-of-the-art SSD-backed KV-cache management schemes.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Large Language Models, KV Cache, LLM Serving, SSD, AI Infrastruction

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_rMbeOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The main claim of this paper is to use SSDs to improve the performance of LLM systems.

To avoid the garbage collection operations may interfere with LLM workload, Zoned Namespace (ZNS) SSDs was propose to reduce migration by exposing data placement and space reclamation to users.

The authors proposed KVReclaim, a host–SSD coordinated KV-cache management framework for ZNS SSDs that takes advantage of KV semantics across the storage system. In particular, KVReclaim manages KV states at the conversation granularity. It flushes the data into zones according to their runtime access behavior and recomputation overhead. Due to these operations, during garbage collection instant, the system can directly discardsselected KV states instead of migrating so to reduce GC-induced overheads.

Critical strengths

The strength of this paper is in presenting a simple algorithm to improve performance of LLM systems by reducing the amount of GC's overheads during garbage collection time.

Critical weaknesses

The major weaknesses include:

  • The proposed idea of using zone-based SSDs and techniques in reducing GC's overhead is quite simple and easily done. So there is a lack of novelty in the work since there was no clever algorithm or smart data-structures were proposed.
  • The performance of KVReclaim hinges on the precision of state conversation granularity. The authors didn't illustrate the impact and pitfalls of such.
  • The evaluation is poor since the authors only compared with some simple SSD's GC algorithms, while there are many clever GC algorithms out there.
  • What is the relationship between the ZNS backend and layout gains?
Questions for authors

Please address the following:

  • The proposed idea of using zone-based SSDs and techniques in reducing GC's overhead is quite simple and easily done. So there is a lack of novelty in the work since there was no clever algorithm or smart data-structures were proposed.
  • The performance of KVReclaim hinges on the precision of state conversation granularity. The authors didn't illustrate the impact and pitfalls of such.
  • The evaluation is poor since the authors only compared with some simple SSD's GC algorithms, while there are many clever GC algorithms out there.
1 comment
AuthorsOpenReview ↗

Response to Reviewer rMbe for Novelty, Conversation-Level Granularity, and Advanced GC Comparison

Concluding remarks: The main claim of this paper is to use SSDs to improve the performance of LLM systems. To avoid the garbage collection operations may interfere with LLM workload, Zoned Namespace (ZNS) SSDs was propose to reduce migration by exposing data placement and space reclamation to users. The authors proposed KVReclaim, a host–SSD coordinated KV-cache management framework for ZNS SSDs that takes advantage of KV semantics across the storage system. In particular, KVReclaim manages KV states at the conversation granularity. It flushes the data into zones according to their runtime access behavior and recomputation overhead. Due to these operations, during garbage collection instant, the system can directly discardsselected KV states instead of migrating so to reduce GC-induced overheads.

Thank you very much for your time and effort in reviewing our manuscript and for your valuable comments and suggestions.

We have submitted a revised version of the manuscript incorporating our responses to the reviewers' comments.

Question 1: What is the novelty of KVReclaim beyond using zone-based SSDs to reduce GC overhead? What new algorithms or data structures distinguish it from a straightforward implementation?

Thank you for this insightful question. ZNS provides host-side control over SSD data placement and space reclamation, while leaving application-specific data management and optimization to the host software. It does not itself provide KV-aware management policies. A straightforward approach is to organize KV cache data by conversation to reduce data migration during subsequent garbage collection.

Compared to straightforward ZNS implementation, we emphasize that KVReclaim contributes coordinated placement, eviction, and recovery policies Its DropCost-aware placement groups conversations with similar eviction costs into the same placement cohort. Then, it couples garbage collection with conversation-level KV eviction, directly discarding low-cost KV states to reclaim space while reducing page migration. Furthermore, its adaptive recovery policy selects between SSD reads and KV recomputation based on their estimated runtime costs, reducing KV recovery overhead. These policies jointly exploit the regenerability of KV states to improve storage management. The results of ablation study verities the efficitiveness of KVReclaim compared to the naive ZNS implementation. We have revised the manuscript by adding more description to help reader better understand our contribution.

Please refer to Sections 3.2–3.4 and 4.3.3 at pp. 5–8 and pp. 9–10, respectively.

Question 2: How does the precision of conversation-level state granularity affect KVReclaim's performance? What are the impacts and potential pitfalls of this granularity?

Thank you for this insightful question. Conversation-level granularity defines the unit of KV eviction and invalidation. Invalidating a selected conversation’s cached KV across its occupied zones can make one or more zones directly resettable without page migration. The trade-off is reduced flexibility in retaining individual KV states: evicting a long conversation may discard KV that will soon be reused, increasing subsequent recomputation. KVReclaim accounts for this cost through DropCost, which combines access activity and recomputation overhead. Besides, DropCost-aware placement groups conversations with similar eviction costs, and ReclaimScore balances the aggregate eviction cost against the reclamation benefit. Recovery operates on the required KV range, so conversation-level invalidation does not necessarily require recomputing the entire conversation. These mechanisms balance migration savings against the cost of regenerating discarded KV.

Question 3: Why is KVReclaim compared only with simple SSD garbage-collection algorithms? How does it perform against more advanced GC algorithms?

Thank you for this helpful suggestion. Advanced GC optimizations, such as low-I/O-intensity-aware partial GC scheduling [1], reduce interference with foreground I/O by scheduling GC operations during periods of low I/O activity. In contrast, KVReclaim incorporates KV eviction and recomputation costs into reclamation decisions, directly discarding selected KV states and avoiding valid KV migration during GC.

Following your suggestion, we have extended our experiments to include this advanced GC scheduling scheme, integrating it into both CachedAttention and FlashGen. The results show that KVReclaim achieves the lowest mean E2E recovery latency in 11 of the 12 evaluated configurations, demonstrating the effectiveness of KV-aware reclamation beyond GC scheduling optimizations. We have revised the manuscript to include these comparisons.

Please refer to Appendix A.3.4 and Figure 4 for more details.

[1] Z. Sha et al. Low I/O Intensity-aware Partial GC Scheduling to Reduce Long-tail Latency in SSDs. ACM Transactions on Architecture and Code Optimization, 18(4).

Reviewer_ymZWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

KVReclaim is a KV-cache management framework that coordinates conversation-level eviction with host-controlled space reclamation on ZNS SSDs. It groups conversations with similar estimated eviction costs, reclaims zones by discarding regenerable KV states without migrating valid data, and dynamically chooses between SSD reads and recomputation. The evaluation uses H200-generated traces replayed through FEMU, covering two models and LoCoMo and LongMemEval workloads.

Critical strengths
  1. KV states can be regenerated, so preserving every valid entry during garbage collection may waste bandwidth. The paper turns this observation into a coherent design connecting placement, reclamation, and recovery.
  2. Beyond latency, the paper reports forced-recomputation GPU time, erase counts, and recovery-path breakdowns. These measurements help assess whether reducing storage traffic merely shifts excessive work to the GPU.
Critical weaknesses
  1. The evaluation does not clearly separate policy benefits from ZNS backend and layout gains. Table 3 leaves the baseline configuration unclear and does not independently isolate cost-aware recovery.
  2. Replay fidelity is not validated against a live system, particularly for GPU contention and computation–I/O overlap. Latency boundaries and the discrepancy between the abstract’s 74.2% reduction and Section 4.3.1 also remain unclear.
Questions for authors
  1. What mechanisms do the Table 3 variants share, and what benefit does cost-aware recovery provide with placement and eviction fixed?
  2. How does replay capture GPU contention and computation–I/O overlap, and can hardware measurements validate its accuracy?
  3. How are latency metrics defined, and how is the 74.2% reduction calculated?
1 comment
AuthorsOpenReview ↗

Response to Reviewer ymZW for Component Ablation, GPU–SSD Simulation, and Recovery Latency Metrics

Concluding remarks: KVReclaim is a KV-cache management framework that coordinates conversation-level eviction with host-controlled space reclamation on ZNS SSDs. It groups conversations with similar estimated eviction costs, reclaims zones by discarding regenerable KV states without migrating valid data, and dynamically chooses between SSD reads and recomputation. The evaluation uses H200-generated traces replayed through FEMU, covering two models and LoCoMo and LongMemEval workloads.

Thank you very much for your time and effort in reviewing our manuscript and for your valuable comments and suggestions.

We have submitted a revised version of the manuscript incorporating our responses to the reviewers' comments.

Question 1: What mechanisms do the Table 3 variants share, and what benefit does cost-aware recovery provide with placement and eviction fixed?

Thank you for your question. Table 3 presents our ablation study of KVReclaim’s components, with all variants sharing the same ZNS SSD backend. With placement and eviction held fixed, the results show that enabling cost-aware recovery further improves overall performance compared with the variant without this component, demonstrating its effectiveness.

Question 2: How does replay capture GPU contention and computation–I/O overlap, and can hardware measurements validate its accuracy?

Thank you for this insightful question. In our prior manuscript, we focused on storage-induced performance differences and estimated computation time using a least-squares fit based on KV-cache depth. Given our limited access to enterprise-oriented ZNS SSD hardware, we used FEMU for storage emulation. To better capture GPU execution behavior and resource contention, we have further integrated TokenSim with FEMU and repeated the experiments.The relative results show that KVReclaim remains effective under this combined simulation setup with GPU contention.

Please refer to Sections 4.2 and 4.3.1 at pp. 8 and 9 for more details.

Question 3: How are latency metrics defined, and how is the 74.2% reduction calculated?

Thank you for pointing this out. Thank you for requesting this clarification. We measure E2E recovery latency from the arrival of a KV recovery event until the corresponding SSD read or GPU recomputation completes. We summarize these measurements using the mean, P50, and P95. The original 74.2% represents the arithmetic average of the P50 and P95 latency reductions against CachedAttention and FlashGen, with each reduction derived from geometric-mean latency ratios across configurations. Averaging the four reported reductions of 90.1%, 89.0%, 56.4%, and 61.3% yields 74.2%. Following our updated experimental evaluation, this aggregate reduction is now 72.4%.

Please refer to Section 4.3.1 at pp. 8 and 9 for more details.

Reviewer_VtFGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

KVReclaim coordinates historical KV-cache placement, eviction, and recovery on ZNS SSDs. It groups conversations by estimated eviction cost, invalidates conversation-level KV to reclaim zones without migration, and chooses SSD reads or recomputation based on storage and GPU delays. Evaluation uses H200-generated traces replayed with FEMU, two models, and LoCoMo and LongMemEval workloads. Section 4.3.1 reports normalized geometric-mean latency reductions of approximately 65% over CachedAttention and FlashGen.

Critical strengths
  1. The paper identifies a concrete mismatch between application-level KV eviction and device-level garbage collection. Regenerable KV states make recomputation-aware reclamation a useful way to reduce storage traffic.

  2. Placement, reclamation, and recovery form a coherent design. Grouping by eviction cost can improve zone selection, while considering GPU queueing makes recovery decisions more relevant than comparing isolated read and recomputation times.

  3. Latency distributions, forced recomputation GPU time, erase counts, and recovery-path breakdowns help characterize the trade-off between discarding cached states and regenerating them.

Critical weaknesses
  1. Policy benefits are not sufficiently separated from backend and layout gains. Table 3 does not fully define the placement, eviction, and recovery policies in Base, Evict, Cohort, and Full, or which backend and I/O optimizations they share. Cost-aware recovery is not independently isolated. Section 4.3.1 attributes some gains to ZNS parallelism, and Qwen-LoCoMo improves substantially without erases or recomputation. These results support the complete system but do not isolate reclamation and recovery policies. Explicit configurations and a simple-policy control on the same ZNS backend with comparable I/O optimizations are needed.

  2. The replay framework lacks end-to-end hardware validation. H200 traces and FEMU replay do not by themselves establish fidelity to a live GPU-and-SSD system. The manuscript provides neither complete-system hardware measurements nor matched validation of the replayed CachedAttention and FlashGen baselines. It is unclear how replay preserves their scheduling, prefetching, and computation-I/O overlap. Proactive recomputation can also change GPU occupancy, queueing, and subsequent recovery decisions; how replay captures this feedback and execution costs under contention is unspecified. Without hardware evidence or quantified replay errors, the reported gains' applicability to real serving remains uncertain.

  3. Latency boundaries and headline aggregation are unclear. The abstract reports a 74.2% average E2E reduction, versus 65.1% and 65.6% in Section 4.3.1. Table 1 refers to "end-to-end recovery latency," while the text also describes request-visible latency. Precise boundaries and aggregation rules are needed to distinguish KV recovery gains from complete-request gains.

Questions for authors
  1. What storage backend and KV placement, eviction, and recovery policies does Base use? Does it share the same ZNS backend and I/O optimizations as Full? With placement and eviction fixed, what additional latency benefit does cost-aware recovery provide?

  2. Can a small live-system experiment validate KVReclaim and the replayed baselines under matched conditions, quantifying latency errors and relative trends while addressing the scheduling and contention concerns above?

  3. What events delimit each latency metric, and how is the abstract's 74.2% reduction calculated?

1 comment
AuthorsOpenReview ↗

Response to Reviewer VtFG for Baseline Configuration, Component Ablation, Simulation Fidelity, and Latency Metrics

Concluding remarks: KVReclaim coordinates historical KV-cache placement, eviction, and recovery on ZNS SSDs. It groups conversations by estimated eviction cost, invalidates conversation-level KV to reclaim zones without migration, and chooses SSD reads or recomputation based on storage and GPU delays. Evaluation uses H200-generated traces replayed with FEMU, two models, and LoCoMo and LongMemEval workloads. Section 4.3.1 reports normalized geometric-mean latency reductions of approximately 65% over CachedAttention and FlashGen.

Thank you very much for your time and effort in reviewing our manuscript and for your valuable comments and suggestions.

We have submitted a revised version of the manuscript incorporating our responses to the reviewers' comments.

Question 1: What storage backend and KV placement, eviction, and recovery policies does Base use? Does it share the same ZNS backend and I/O optimizations as Full?

We apologize for the unclear description of Base. Base uses the same ZNS SSD backend as Full, with the proposed optimizations disabled. KV data are placed across different zones in a round-robin manner according to conversation IDs.

With placement and eviction fixed, what additional latency benefit does cost-aware recovery provide?

Furthermore, we also conducted ablation experiments to evaluate the contribution of individual KVReclaim components, including cost-aware recovery. With placement and eviction held fixed, we compared Full with a variant that disables cost-aware recovery. The results show that enabling cost-aware recovery further improves overall performance.

Please refer to Section 4.3.3 at pp. 9 and 10 for more details.

Question 2: Can a small live-system experiment validate KVReclaim and the replayed baselines under matched conditions, quantifying latency errors and relative trends while addressing the scheduling and contention concerns above?

Thank you for this helpful suggestion. Given our limited access to enterprise-oriented ZNS SSD hardware, we use FEMU, a high-fidelity, full-system SSD emulator widely used in storage research. To better capture GPU execution behavior, we have adopted a TokenSim-derived GPU simulator coupled with FEMU and repeated the experiments. The results show that KVReclaim remains effective in this combined simulation environment.

Please refer to Sections 4.2 and 4.3.1 at pp. 8 and 9 for more details.

To facilitate reproducibility, the code and relevant scripts will be publicly available after acceptance.

Question 3: What events delimit each latency metric, and how is the abstract's 74.2% reduction calculated?

Thank you for pointing this out. We measure E2E recovery latency from the arrival of a KV recovery event until the corresponding SSD read or GPU recomputation completes. We report the mean, P50, and P95 of these durations. The original 74.2% was obtained by averaging the four P50 and P95 latency reductions relative to CachedAttention and FlashGen, following geometric-mean aggregation across configurations:

(90.1% + 89.0% + 56.4% + 61.3%) / 4 = 74.2%

We have now updated all experiments and revised the corresponding reduction in the abstract from 74.2% to 72.4%, based on the average reduction in mean E2E recovery latency.

Please refer to Section 4.3.1 at pp. 8 and 9.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.