- Rating
- 4
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how to keep LLM requests running when GPU KV memory becomes tight. Instead of requiring the full 16-bit KV state before execution, ElasticKV allows attention to directly use an 8-bit BASE state and promotes it back when there is enough memory. The system is built on vLLM, with changes to page layout, attention kernels, and fidelity control. Evaluation shows that it avoids many preemptions and reduces serving latency, while keeping average generation quality close to FP16 in the tested settings.
Critical strengths
-
I like the observation that reaching the target KV fidelity does not have to block execution. The paper also does the work needed to make this useful: packing the pages, returning real capacity to the allocator, and letting attention consume the compact state. These parts fit together quite well.
-
The evaluation is fairly complete, including quality, realistic request traces, different model sizes, TP/DP, and NVIDIA/AMD GPUs. I also appreciate Table 3. Turning off promotion helps startup latency a little but nearly doubles BASE exposure, which explains why promotion is needed.
Critical weaknesses
My main question is how to choose between ElasticKV and static FP8. Appendix C.5 shows better speed with static FP8, while Table 1 shows better average quality with ElasticKV. This looks like a reasonable tradeoff, but I would like the paper to put these results together so readers can see when to use which. Also, average quality being close to FP16 does not mean every task or request sees the same behavior.
Some memory-management details are still unclear to me. I am not clear when is the exact CPU copy created, and whether that copying cost is included in the reported numbers. The per-rank GPU and host budgets would also help. These are mostly clarification questions; overall I like the design and the amount of evaluation.
Questions for authors
-
From the existing FP8 and quality results, when would you recommend ElasticKV over static FP8? It would help to discuss the quality and latency together, and say whether BASE exposure is concentrated in a small set of requests.
-
When is the exact CPU backing created or updated? Please also clarify the GPU/host memory budgets and where the copy overhead is counted.
4 comments
Response to Reviewer NvLx (Part 2/2)
W1.2: Also, average quality being close to FP16 does not mean every task or request sees the same behavior.
Q1.2: It would help to …, and say whether BASE exposure is concentrated in a small set of requests.
We agree that near-FP16 averages do not imply uniform quality preservation across tasks or requests. For quality across tasks, we have strengthened the LongBench evaluation and added a discussion of task-dependent losses in App. C.1, covering MuSiQue, HotpotQA, and 2WikiMQA, explicitly acknowledging task-dependent quality losses. For example, MuSiQue loses 0.78 and 2.02 F1 points on Llama-3.1-8B-Instruct and Qwen3-8B, respectively. These tasks require connected reasoning across multiple documents, where reduced KV fidelity may affect evidence selection or integration and thereby alter the final answer.
To examine whether BASE exposure is concentrated in a small set of requests, we have added measurements at three representative serving settings, including the highest-concurrency synthetic setting for each model and the A100 Mooncake Window C replay. We have reported and discussed the results in App. C.9 and also summarized them below.
| Serving setting | Requests | E = 0 | 0 < E ≤ 10% | 10% < E ≤ 20% | 20% < E ≤ 40% | E > 40% | P99 exposure |
|---|---|---|---|---|---|---|---|
| Synthetic, Llama-3.1-8B, c = 40 | 600 | 6.00% | 9.50% | 44.17% | 40.33% | 0.00% | 27.36% |
| Synthetic, Qwen3-8B, c = 38 | 600 | 4.00% | 6.17% | 47.50% | 42.33% | 0.00% | 29.97% |
| Mooncake, A100 Window C, d = 16 | 1,122 | 33.07% | 31.46% | 22.99% | 11.94% | 0.53% | 35.01% |
For the synthetic workloads, 84.50% of Llama requests and 89.83% of Qwen requests have 10-40% exposure, with P99 below 30% on both models. Thus, BASE exposure is spread across most requests. Mooncake is more uneven, with 33.07% of requests having no exposure, but 87.52% of requests remain at or below 20%, and only six requests (i.e., 0.53%) exceed 40%. This variation reflects the fluctuating traffic and variable request lengths in the realistic production trace. Requests executing entirely during low-pressure intervals can avoid demotion, while those spending more of their execution under sustained pressure may incur higher exposure. Longer-running requests may also span multiple pressure episodes.
W2: I am not clear when is the exact CPU copy created, and whether that copying cost is included in the reported numbers. The per-rank GPU and host budgets would also help.
Q2: When is the exact CPU backing created or updated? Please also clarify the GPU/host memory budgets and where the copy overhead is counted.
Exact CPU backing is created asynchronously through vLLM’s native offloading path as KV blocks become complete. A pair becomes demotion-eligible only after both blocks have valid CPU copies. Therefore, demotion uses previously established backing and requires no new D2H backup. The cost of these transfers is reflected in our end-to-end serving results. We have clarified the CPU backing mechanism in App. B.2.
Thanks for the suggestion. We have clarified the host memory budgets in Sec. 4.1 and added any previously missing GPU budget details to the corresponding experimental sections. We use host memory budgets of 128GB on A100 and 96GB per rank on MI355X. The GPU KV budget is fixed at 20GB for the concurrency, FP8, and system analysis experiments, and at 10GB for ShareGPT. We set gpu_memory_utilization to 0.9 for the A100 Mooncake replay and context length experiments, and to 0.75 for the MI355X Mooncake replay.
Thanks for the added FP8 experiments
Thanks for adding the comparison with static FP8 on the same Triton backend and the per-request BASE exposure results. The comparison makes the tradeoff clearer: FP8 can still give a faster first token at high concurrency, while ElasticKV generally has lower time per output token and higher throughput in these runs. The explanation of when the CPU copies are made is helpful too.
Thanks for the positive feedback and support
We sincerely thank the reviewer for the positive assessment and for taking the time to review the additional results and clarifications. We are glad that the comparison with static FP8 has made the trade-offs clearer, and that the per-request BASE exposure results and explanation of CPU backing have been helpful.
Thanks again for the thoughtful engagement and continued support for this work!
Response to Reviewer NvLx (Part 1/2)
We sincerely thank the reviewer for the careful reading and constructive feedback, and for recognizing the value of our core insight, the coherence of our system design, and the breadth of our evaluation. We have uploaded a revised manuscript and provide responses below.
Thanks for the suggestion. We have added a dedicated section in the main text, Sec. 4.4, to compare ElasticKV with static FP8 and discuss the deployment trade-offs.
In this new section, we have implemented an A100-compatible Triton FP8 reference with optimized conversion operations to enable backend-matched comparison on A100 [1, 2]. Compared with the optimized Triton FP8 reference, ElasticKV generally achieves lower TPOT and higher throughput. At low concurrency, ElasticKV has lower mean TTFT, while at higher concurrency, static FP8 achieves lower TTFT, including a 28% reduction on Llama-3.1-8B-Instruct at concurrency 40. Static FP8’s uniformly compressed KV provides more capacity headroom, while ElasticKV retains FP16 where possible and reclaims capacity on demand. This can favor FP8’s request startup under heavy memory pressure. However, its TTFT advantage does not extend to TPOT or throughput because the required conversion adds decoding work on A100 [2]. We retain the original FlashInfer [3] results in App. C.6.2 as a practical deployment reference.
Based on these results, we recommend ElasticKV for quality-sensitive services that aim to maximize generation quality subject to latency and throughput SLOs. This goal is relevant to common applications such as question answering (QA) and code completion, where generation quality directly affects the usefulness of the service. Its TPOT and throughput advantages in the matched Triton comparison also matter for streaming generation and longer responses. Even when static FP8 offers lower TTFT, its quality loss compared with FP16 can be substantial on certain tasks. For example, on MuSiQue, static FP8 reduces QA F1 by 3.89 and 4.69 points on Llama-3.1-8B-Instruct and Qwen3-8B, respectively, while ElasticKV limits these reductions to 0.78 and 2.02 at around 19% BASE exposure (see App. C.1). Therefore, when ElasticKV meets the required serving SLOs, these performance and quality benefits can favor its deployment. Static FP8 remains attractive only when its quality cost is acceptable.
[1] vLLM Project. "triton_attn". GitHub source code. https://github.com/vllm-project/vllm/blob/v0.15.0/vllm/v1/attention/backends/triton_attn.py#L472. Accessed 2026.
[2] NVIDIA. "Parallel Thread Execution ISA, Version 8.8 (Data Movement and Conversion Instructions: cvt)". CUDA documentation. https://docs.nvidia.com/cuda/archive/12.9.1/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-cvt. Accessed 2026.
[3] Ye et al. "FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving." MLSys, 2025.