- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper asks how KV-cache offloading should be organized as offload bandwidth grows. The authors use an NVLink-connected GPU that only stores KV data as a stand-in for a fast remote memory tier. They found that whole-request migration is slower than layer- and tensor-level migration at low bandwidth, but it overtakes both at high bandwidth. The authors explain that once transfers are cheap, coordination overhead and interruptions to decoding become the main cost. Based on this, the paper proposes three design rules and builds CARGOKV on them. The rules are to amortize migration interruptions, to free memory with few logical actions, and to limit migration activity. CARGOKV moves whole requests, offloads the request with the largest KV state first, reloads requests by remaining generation budget, and bounds asynchronous migration. The evaluation covers three Qwen models, five static workloads, one burst workload, and four baselines. At the highest static latency loads, CARGOKV lowers request-level P99 TPOT by 24.3–49.3% compared with LayerKV. NEO still reaches higher goodput in some settings, which the paper acknowledges. The scale-out results come from simulation, not from a measured multi-node cluster.
Critical strengths
- Well-motivated, controlled study of an important design question
The main strength is the controlled study of one clear question. Does the best way to organize KV offloading change as bandwidth grows? The paper shows that it does. At 32 GB/s, request migration is 13.9–18.1% slower than the two finer-grained policies. At the highest bandwidth setting, it is 5.3–8.1% faster. Inference compute and the remote path are the same for all policies, so this result is easier to interpret than a comparison where hardware and scheduling change together. The authors also say that batching or overlapping fine-grained transfers could reach the same goal. They do not claim that request-level movement is always better.
- Evaluation goes beyond a single benchmark
The paper combines a three-model comparison, runtime measurements, ablations, and controller sensitivity tests. In the ablations, removing request granularity raises P99 TPOT by 23–38%. Removing minimum-action selection raises it by 9–15%. Removing the guard raises it by 8–18%. Together, these results suggest that the gains do not come from one lucky parameter choice. The paper also reports results that are not favorable to its own method. For example, NEO reaches higher goodput than CARGOKV on Qwen2.5-72B/M-M at concurrency 27. This makes the claims more believable, because they are presented as advantages under certain workloads and pressures, not as universal wins.
Critical weaknesses
- The setup does not let readers check the memory pressure or the fairness of the baselines.
Although all methods get the same GPU and link budget, the paper does not report the effective KV capacity per GPU, the memory reserved for weights and runtime, or the native memory-fit concurrency c* for the main model–workload pairs. So it is hard to tell where the experiments become memory-constrained. Appendix A also does not give the baseline versions, tuning settings, or how the baselines were adapted to the storage-only NVLink peer. This is especially unclear for NEO’s CPU-attention path. Without these details, I cannot be sure that the gains come from the migration policy and not from differences in memory allocation or execution setup.
- The evidence does not isolate the continuity mechanism
The mechanism is not tested against the strongest baselines. The runtime breakdown compares CARGOKV only with ZeRO-Inference and FlexGen. The primary baselines are LayerKV and NEO. So the explanation holds for the large gains over the conventional references, but it is not shown for the more important comparisons. Also, CARGOKV issues fewer logical actions partly because it counts a whole request as one action instead of one layer. The paper itself notes that one request action can involve several physical copies. To show a real coordination benefit, it should report physical copies, synchronization events, and exposed time.
Questions for authors
-
For each main model–workload pair, please report the KV datatype, the effective KV capacity per GPU, the memory reserved for weights and runtime, the native memory-fit concurrency c*, and the absolute SLO threshold. Was the KV pool deliberately limited below the available GPU memory? Please also show how much KV is on the GPU and how much is offloaded at a few representative loads, especially for Qwen3-32B/S-S. This would let readers check that these experiments are really under memory pressure.
-
Which versions and settings of LayerKV, NEO, ZeRO-Inference, and FlexGen were used? What changes were needed to make them work with the storage-only NVLink peer? For NEO, please explain how the CPU-attention path was kept, how many CPU resources it had, and where its KV state was stored. For ZeRO-Inference and FlexGen, were weights and activations also offloaded, or only the KV cache?
-
At one high-pressure operating point, please give the same runtime breakdown for CARGOKV, LayerKV, and NEO. In the bandwidth sweep, does the reversal still appear if the fine-grained transfers are batched or coalesced while keeping their residency decisions? Also, how much do the results vary across runs near the crossover?