acceptodds
Withdrawn from ICLR 2027

Rethinking KV Cache Offloading: Bandwidth Scaling and Decode Continuity

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

KV cache offloading trades data movement for memory capacity in language model serving. As offload bandwidth increases, should systems continue to prioritize minimizing transferred bytes? We study this question using a storage-only NVLink peer as a controlled surrogate for a high-bandwidth remote tier, with inference resources held fixed. A bandwidth sweep reveals a reversal: request-granular migration begins slower than two fine-grained policies but becomes faster as transfer cost falls. Load sweeps and runtime analysis connect this reversal to a broader design concern: preserving useful decoding while recovering memory capacity. We formulate three design rules and instantiate them in CargpKV: amortize migration interruptions, recover headroom with few logical actions, and bound migration activity. Across three dense models and six workload regimes, comparisons with LayerKV, NEO, ZeRO-Inference, and FlexGen reveal both gains and boundaries. At the highest tested static latency loads, CargpKV reduces P99 time per output token by 24.3–49.3% relative to SOTA. The resulting guideline is conditional and portable: a fast offload path should be designed around the exposed cost of interrupting decoding, alongside the bytes it moves.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Efficient LLM inference, KV-cache offloading, GPU memory management, high-bandwidth interconnects, migration granularity, execution continuity

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_KTVaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks how KV-cache offloading should be organized as offload bandwidth grows. The authors use an NVLink-connected GPU that only stores KV data as a stand-in for a fast remote memory tier. They found that whole-request migration is slower than layer- and tensor-level migration at low bandwidth, but it overtakes both at high bandwidth. The authors explain that once transfers are cheap, coordination overhead and interruptions to decoding become the main cost. Based on this, the paper proposes three design rules and builds CARGOKV on them. The rules are to amortize migration interruptions, to free memory with few logical actions, and to limit migration activity. CARGOKV moves whole requests, offloads the request with the largest KV state first, reloads requests by remaining generation budget, and bounds asynchronous migration. The evaluation covers three Qwen models, five static workloads, one burst workload, and four baselines. At the highest static latency loads, CARGOKV lowers request-level P99 TPOT by 24.3–49.3% compared with LayerKV. NEO still reaches higher goodput in some settings, which the paper acknowledges. The scale-out results come from simulation, not from a measured multi-node cluster.

Critical strengths
  1. Well-motivated, controlled study of an important design question

The main strength is the controlled study of one clear question. Does the best way to organize KV offloading change as bandwidth grows? The paper shows that it does. At 32 GB/s, request migration is 13.9–18.1% slower than the two finer-grained policies. At the highest bandwidth setting, it is 5.3–8.1% faster. Inference compute and the remote path are the same for all policies, so this result is easier to interpret than a comparison where hardware and scheduling change together. The authors also say that batching or overlapping fine-grained transfers could reach the same goal. They do not claim that request-level movement is always better.

  1. Evaluation goes beyond a single benchmark

The paper combines a three-model comparison, runtime measurements, ablations, and controller sensitivity tests. In the ablations, removing request granularity raises P99 TPOT by 23–38%. Removing minimum-action selection raises it by 9–15%. Removing the guard raises it by 8–18%. Together, these results suggest that the gains do not come from one lucky parameter choice. The paper also reports results that are not favorable to its own method. For example, NEO reaches higher goodput than CARGOKV on Qwen2.5-72B/M-M at concurrency 27. This makes the claims more believable, because they are presented as advantages under certain workloads and pressures, not as universal wins.

Critical weaknesses
  1. The setup does not let readers check the memory pressure or the fairness of the baselines.

Although all methods get the same GPU and link budget, the paper does not report the effective KV capacity per GPU, the memory reserved for weights and runtime, or the native memory-fit concurrency c* for the main model–workload pairs. So it is hard to tell where the experiments become memory-constrained. Appendix A also does not give the baseline versions, tuning settings, or how the baselines were adapted to the storage-only NVLink peer. This is especially unclear for NEO’s CPU-attention path. Without these details, I cannot be sure that the gains come from the migration policy and not from differences in memory allocation or execution setup.

  1. The evidence does not isolate the continuity mechanism

The mechanism is not tested against the strongest baselines. The runtime breakdown compares CARGOKV only with ZeRO-Inference and FlexGen. The primary baselines are LayerKV and NEO. So the explanation holds for the large gains over the conventional references, but it is not shown for the more important comparisons. Also, CARGOKV issues fewer logical actions partly because it counts a whole request as one action instead of one layer. The paper itself notes that one request action can involve several physical copies. To show a real coordination benefit, it should report physical copies, synchronization events, and exposed time.

Questions for authors
  1. For each main model–workload pair, please report the KV datatype, the effective KV capacity per GPU, the memory reserved for weights and runtime, the native memory-fit concurrency c*, and the absolute SLO threshold. Was the KV pool deliberately limited below the available GPU memory? Please also show how much KV is on the GPU and how much is offloaded at a few representative loads, especially for Qwen3-32B/S-S. This would let readers check that these experiments are really under memory pressure.

  2. Which versions and settings of LayerKV, NEO, ZeRO-Inference, and FlexGen were used? What changes were needed to make them work with the storage-only NVLink peer? For NEO, please explain how the CPU-attention path was kept, how many CPU resources it had, and where its KV state was stored. For ZeRO-Inference and FlexGen, were weights and activations also offloaded, or only the KV cache?

  3. At one high-pressure operating point, please give the same runtime breakdown for CARGOKV, LayerKV, and NEO. In the bandwidth sweep, does the reversal still appear if the fine-grained transfers are batched or coalesced while keeping their residency decisions? Also, how much do the results vary across runs near the crossover?

Reviewer_ax3zOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how KV-cache migration policies should change as offload bandwidth increases. A controlled bandwidth sweep shows that request-level migration can overtake finer-grained alternatives as transfer costs decrease. The authors implement CARGOKV using whole-request migration, largest-first victim selection, reload ordering based on remaining generation budget, and bounded migration activity. Experiments across multiple models and workloads show benefits under substantial memory pressure, although comparisons with NEO depend on the setting. My main concerns are whether the evaluation adequately captures decoding continuity and how the scheduling policy affects waiting times across different request groups.

Critical strengths
  1. The controlled bandwidth sweep reveals a useful change in policy ordering while holding inference resources fixed. This provides concrete evidence that migration coordination deserves attention as transfer costs decrease.
  2. The evaluation covers multiple models and workload regimes, with ablations supporting the contributions of migration granularity, victim selection, and pacing. The paper also acknowledges settings where NEO performs better and clearly states the limitations of its peer-GPU surrogate.
Critical weaknesses
  1. The reported P99 TPOT is computed from each request’s average token time. Although suspension time is included, individual long pauses can be diluted by many shorter token intervals. SLO-goodput also excludes TTFT. These metrics demonstrate improvements in request-average decoding performance, but do not fully establish smoother output delivery. Inter-token latency tails, maximum output gaps, and end-to-end latency would strengthen the continuity claim.
  2. The scheduling policy’s impact on suspended requests is insufficiently evaluated. Largest-first offloading selects requests with large KV footprints, while budget-based reloading may delay those with large remaining generation budgets. The paper acknowledges that the reload policy provides no starvation bound, but does not report suspension durations or performance across request groups. It therefore remains unclear whether aggregate gains come with substantial delays for particular types of requests.
Questions for authors
  1. Do inter-token latency tails, maximum output gaps, and end-to-end latency support the same conclusions as request-average TPOT?
  2. How do suspension duration and completion latency vary with KV footprint and remaining generation budget, especially under bursty arrivals?
  3. Have you compared budget-based reloading with FCFS or age-based ordering in terms of request waiting time and overall goodput?
Reviewer_8zsLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates whether KV-cache offloading should use different policies as the bandwidth of the offload link increases. It uses an NVLink-connected peer GPU to hold remote KV state and compares request-level migration with two finer-grained approaches. At lower bandwidth, request-level migration is slower, but its performance improves as bandwidth increases. This is mainly attributed to the lower coordination overhead and fewer interruptions during decoding. Based on these observations, the authors propose CargoKV, which combines request-level migration with largest-first offload selection and limits on concurrent migration activity. Experiments on three Qwen models under both static and bursty workloads show 24.3–49.3% lower P99 time per output token than LayerKV at the highest static latency loads. Its comparison with NEO is less consistent, particularly when measured by goodput.

Critical strengths

The experimental setup is reasonable, and the results generally support the authors' explanation. Keeping the inference resources fixed during the bandwidth sweep is a good control and the ablations provide some evidence for the individual design choices. However, I found it difficult to judge how general the conclusions are because the fine-grained policies and some of the baseline configurations are not described in enough detail. I also would have liked to see variation across runs, particularly for the smaller performance differences.

The paper is generally easy to follow. The bandwidth study leads naturally into the CargoKV design and then the evaluation, and the distinction between the broader design idea and the specific implementation is clear. The discussion of the limitations of using a peer GPU and simulation is also helpful. My main presentation concern is that the primary results figure (figure 5) is quite dense. Can some of it move to ablation or appendix? A few more details about the experimental setup would also make the results easier to interpret.

The paper provides a useful perspective on KV-cache offloading over faster links. Since request migration and preemption themselves are not new, I see the main contribution as the empirical analysis of how their tradeoffs change as bandwidth increases. That is potentially useful, but I am not yet convinced that the experiments establish the robustness of the reported crossover or show a clear advantage over a straightforward request-level offloading policy.

Strengths-

  1. The motivation is clear. As offload bandwidth increases, it makes sense to examine the scheduling overhead left on the decode path. The paper gives this issue a concrete setting in LLM serving.
  2. The controlled bandwidth sweep is a good starting point. It shows a change in policy ordering with inference compute held fixed, which makes the argument more convincing than a comparison at only one bandwidth.
  3. The evaluation goes beyond the initial observation to cover several model sizes and workload regimes. The ablations suggest that granularity, victim selection, and migration guards all contribute to CargoKV's performance.
  4. I appreciate the discussion of limitations. The authors are clear about what the peer-GPU surrogate represents, label the scale-out results as simulations, and report cases where NEO performs better.
Critical weaknesses
  1. The fine-grained comparison needs more detail. FG-1 and FG-2 are described mainly in terms of migration granularity. I could not tell exactly how partially offloaded requests continue or suspend decoding, or how transfers overlap with computation. These choices could affect the observed reversal. The sweep also uses only one model and workload, so it is unclear how much the crossover would move in another setting. Section 2.3 and Figure 2 compare only FG-1 and FG-2 under load; adding ReqOffload would help connect the initial reversal to performance under serving load. Since the paper itself discusses batching finer transfers, I would also like to see whether that changes the result. This matters because the bandwidth-dependent observation carries much of the contribution.
  2. The baseline setup is not clear enough. I would like more information about how the existing systems were adapted to the peer-GPU setup, including where they stored offloaded state and which compute resources they used. This is especially relevant for NEO, with its CPU attention, and for FlexGen and ZeRO-Inference, which were developed for different operating conditions. The detailed runtime breakdown compares CargoKV with the latter two, leaving the explanation of the gains over LayerKV less directly supported. A simple request-level swap/resume policy on the same transfer path would be a useful baseline too. It would help show how much the selection and pacing choices add beyond request migration. The ablations partly address this, but leave that overall comparison open.
  3. There is little information about variability. The results average two runs, but no spread is reported. At high bandwidth, the TPOT improvement in the controlled sweep is about 5-8%, and the differences around the crossover are smaller. It is hard to judge those differences without seeing how consistent they are across runs. For the P99 results, the number of measured requests and the trace duration matter as well. Reporting these along with the run-to-run variation would make the results easier to assess.
  4. The abstract is broader than the reported comparison. The 24.3-49.3% P99 TPOT reduction "relative to SOTA" is specifically against LayerKV. NEO has slightly lower TPOT on S-S and higher goodput at the reported Qwen2.5-72B/M-M operating point. The abstract should name LayerKV and briefly acknowledge the mixed comparison with NEO. This is a smaller concern and does not drive my rating.
  5. Figure 1 appears on page 2 but is referred to on page 3.
  6. “FG-1 (coarser fine-grained)” is not clear and even confusing. Better to call it LayerGranularity or something.
  7. ReqOffload word used in Figure 1 is not used in text.
  8. In figure 1 and 2: some text is written in so small and light gray shade that it is not visible when printed.
  9. Even in left and right of figure 2: the marker styles for FG-1 and FG-2 are different. This is confusing. The marker styles are also different from that in figure 1.
  10. Experiments are conducted on a single GPU from NVIDIA. What about scalability to other GPUs, say from AMD?
Questions for authors
  1. Do partially offloaded requests remain runnable under FG-1 and FG-2? How are their transfers scheduled and overlapped with decoding?
  2. Would batching fine-grained transfers change the observed bandwidth-dependent reversal and how does ReqOffload perform in Figure 2's load study?
  3. How were the baselines configured for the peer-GPU setup and how does CargoKV compare with a simple request-level swap/resume policy?
  4. How consistent are the main results across runs and how many requests contribute to the reported tail-latency estimates? I find the questions posed by this paper interesting, and the explanation is plausible. The results under high memory pressure are promising, and the paper makes a reasonable case for paying more attention to interruptions during decoding. My main hesitation is with the comparisons. I would like a clearer picture of how the fine-grained policies and baselines are implemented, and of what CargoKV adds over a simple request swapping policy. These gaps make it difficult to judge how strong the empirical contribution really is.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.