acceptodds
Under review as a conference paper at ICLR 2027

Does Execution Require Target KV Fidelity? A Mixed-Fidelity KV Runtime for LLM Serving

Abstract

Large language model (LLM) serving is increasingly constrained by the GPU memory consumed by key-value (KV) caches. Existing compression, eviction, and offloading techniques alleviate this pressure, but serving runtimes typically treat only the configured target KV representation as execution-ready. Under memory pressure, this target-only contract can turn KV shortage into request stalls and preemptions. We present ElasticKV, a mixed-fidelity KV runtime built on the observation that reaching target KV fidelity need not gate execution. ElasticKV introduces a compact intermediate KV state, making fidelity a runtime-managed execution property. To realize this state in a paged serving runtime, ElasticKV combines (i) a pair-structured layout that turns fidelity reduction into reusable GPU capacity, (ii) a dual-mode attention backend that directly consumes the compact state while preserving the native target-only path, and (iii) pressure-aware fidelity control that adapts KV fidelity to memory pressure. Our extensive evaluation across diverse workloads, model families and scales, and GPU platforms demonstrates the effectiveness and generality of ElasticKV. Under high concurrency, ElasticKV achieves 3.8-4.0 lower time-to-first-token (TTFT) and 9.1 lower P90 TTFT than vLLM while preserving generation quality.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Large Language Model Serving, KV Cache, Runtime

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_NvLxOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to keep LLM requests running when GPU KV memory becomes tight. Instead of requiring the full 16-bit KV state before execution, ElasticKV allows attention to directly use an 8-bit BASE state and promotes it back when there is enough memory. The system is built on vLLM, with changes to page layout, attention kernels, and fidelity control. Evaluation shows that it avoids many preemptions and reduces serving latency, while keeping average generation quality close to FP16 in the tested settings.

Critical strengths
  • I like the observation that reaching the target KV fidelity does not have to block execution. The paper also does the work needed to make this useful: packing the pages, returning real capacity to the allocator, and letting attention consume the compact state. These parts fit together quite well.

  • The evaluation is fairly complete, including quality, realistic request traces, different model sizes, TP/DP, and NVIDIA/AMD GPUs. I also appreciate Table 3. Turning off promotion helps startup latency a little but nearly doubles BASE exposure, which explains why promotion is needed.

Critical weaknesses

My main question is how to choose between ElasticKV and static FP8. Appendix C.5 shows better speed with static FP8, while Table 1 shows better average quality with ElasticKV. This looks like a reasonable tradeoff, but I would like the paper to put these results together so readers can see when to use which. Also, average quality being close to FP16 does not mean every task or request sees the same behavior.

Some memory-management details are still unclear to me. I am not clear when is the exact CPU copy created, and whether that copying cost is included in the reported numbers. The per-rank GPU and host budgets would also help. These are mostly clarification questions; overall I like the design and the amount of evaluation.

Questions for authors
  1. From the existing FP8 and quality results, when would you recommend ElasticKV over static FP8? It would help to discuss the quality and latency together, and say whether BASE exposure is concentrated in a small set of requests.

  2. When is the exact CPU backing created or updated? Please also clarify the GPU/host memory budgets and where the copy overhead is counted.

4 comments
AuthorsOpenReview ↗

Response to Reviewer NvLx (Part 1/2)

We sincerely thank the reviewer for the careful reading and constructive feedback, and for recognizing the value of our core insight, the coherence of our system design, and the breadth of our evaluation. We have uploaded a revised manuscript and provide responses below.

W1.1: My main question is how to choose between ElasticKV and static FP8. … This looks like a reasonable tradeoff, but I would like the paper to put these results together so readers can see when to use which.

Q1.1: From the existing FP8 and quality results, when would you recommend ElasticKV over static FP8? It would help to discuss the quality and latency together…

Thanks for the suggestion. We have added a dedicated section in the main text, Sec. 4.4, to compare ElasticKV with static FP8 and discuss the deployment trade-offs.

In this new section, we have implemented an A100-compatible Triton FP8 reference with optimized conversion operations to enable backend-matched comparison on A100 [1, 2]. Compared with the optimized Triton FP8 reference, ElasticKV generally achieves lower TPOT and higher throughput. At low concurrency, ElasticKV has lower mean TTFT, while at higher concurrency, static FP8 achieves lower TTFT, including a 28% reduction on Llama-3.1-8B-Instruct at concurrency 40. Static FP8’s uniformly compressed KV provides more capacity headroom, while ElasticKV retains FP16 where possible and reclaims capacity on demand. This can favor FP8’s request startup under heavy memory pressure. However, its TTFT advantage does not extend to TPOT or throughput because the required conversion adds decoding work on A100 [2]. We retain the original FlashInfer [3] results in App. C.6.2 as a practical deployment reference.

Based on these results, we recommend ElasticKV for quality-sensitive services that aim to maximize generation quality subject to latency and throughput SLOs. This goal is relevant to common applications such as question answering (QA) and code completion, where generation quality directly affects the usefulness of the service. Its TPOT and throughput advantages in the matched Triton comparison also matter for streaming generation and longer responses. Even when static FP8 offers lower TTFT, its quality loss compared with FP16 can be substantial on certain tasks. For example, on MuSiQue, static FP8 reduces QA F1 by 3.89 and 4.69 points on Llama-3.1-8B-Instruct and Qwen3-8B, respectively, while ElasticKV limits these reductions to 0.78 and 2.02 at around 19% BASE exposure (see App. C.1). Therefore, when ElasticKV meets the required serving SLOs, these performance and quality benefits can favor its deployment. Static FP8 remains attractive only when its quality cost is acceptable.

[1] vLLM Project. "triton_attn". GitHub source code. https://github.com/vllm-project/vllm/blob/v0.15.0/vllm/v1/attention/backends/triton_attn.py#L472. Accessed 2026.

[2] NVIDIA. "Parallel Thread Execution ISA, Version 8.8 (Data Movement and Conversion Instructions: cvt)". CUDA documentation. https://docs.nvidia.com/cuda/archive/12.9.1/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-cvt. Accessed 2026.

[3] Ye et al. "FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving." MLSys, 2025.

AuthorsOpenReview ↗

Response to Reviewer NvLx (Part 2/2)

W1.2: Also, average quality being close to FP16 does not mean every task or request sees the same behavior.

Q1.2: It would help to …, and say whether BASE exposure is concentrated in a small set of requests.

We agree that near-FP16 averages do not imply uniform quality preservation across tasks or requests. For quality across tasks, we have strengthened the LongBench evaluation and added a discussion of task-dependent losses in App. C.1, covering MuSiQue, HotpotQA, and 2WikiMQA, explicitly acknowledging task-dependent quality losses. For example, MuSiQue loses 0.78 and 2.02 F1 points on Llama-3.1-8B-Instruct and Qwen3-8B, respectively. These tasks require connected reasoning across multiple documents, where reduced KV fidelity may affect evidence selection or integration and thereby alter the final answer.

To examine whether BASE exposure is concentrated in a small set of requests, we have added measurements at three representative serving settings, including the highest-concurrency synthetic setting for each model and the A100 Mooncake Window C replay. We have reported and discussed the results in App. C.9 and also summarized them below.

Serving settingRequestsE = 00 < E ≤ 10%10% < E ≤ 20%20% < E ≤ 40%E > 40%P99 exposure
Synthetic, Llama-3.1-8B, c = 406006.00%9.50%44.17%40.33%0.00%27.36%
Synthetic, Qwen3-8B, c = 386004.00%6.17%47.50%42.33%0.00%29.97%
Mooncake, A100 Window C, d = 161,12233.07%31.46%22.99%11.94%0.53%35.01%

For the synthetic workloads, 84.50% of Llama requests and 89.83% of Qwen requests have 10-40% exposure, with P99 below 30% on both models. Thus, BASE exposure is spread across most requests. Mooncake is more uneven, with 33.07% of requests having no exposure, but 87.52% of requests remain at or below 20%, and only six requests (i.e., 0.53%) exceed 40%. This variation reflects the fluctuating traffic and variable request lengths in the realistic production trace. Requests executing entirely during low-pressure intervals can avoid demotion, while those spending more of their execution under sustained pressure may incur higher exposure. Longer-running requests may also span multiple pressure episodes.

W2: I am not clear when is the exact CPU copy created, and whether that copying cost is included in the reported numbers. The per-rank GPU and host budgets would also help.

Q2: When is the exact CPU backing created or updated? Please also clarify the GPU/host memory budgets and where the copy overhead is counted.

Exact CPU backing is created asynchronously through vLLM’s native offloading path as KV blocks become complete. A pair becomes demotion-eligible only after both blocks have valid CPU copies. Therefore, demotion uses previously established backing and requires no new D2H backup. The cost of these transfers is reflected in our end-to-end serving results. We have clarified the CPU backing mechanism in App. B.2.

Thanks for the suggestion. We have clarified the host memory budgets in Sec. 4.1 and added any previously missing GPU budget details to the corresponding experimental sections. We use host memory budgets of 128GB on A100 and 96GB per rank on MI355X. The GPU KV budget is fixed at 20GB for the concurrency, FP8, and system analysis experiments, and at 10GB for ShareGPT. We set gpu_memory_utilization to 0.9 for the A100 Mooncake replay and context length experiments, and to 0.75 for the MI355X Mooncake replay.

Reviewer_NvLxOpenReview ↗

Thanks for the added FP8 experiments

Thanks for adding the comparison with static FP8 on the same Triton backend and the per-request BASE exposure results. The comparison makes the tradeoff clearer: FP8 can still give a faster first token at high concurrency, while ElasticKV generally has lower time per output token and higher throughput in these runs. The explanation of when the CPU copies are made is helpful too.

AuthorsOpenReview ↗

Thanks for the positive feedback and support

We sincerely thank the reviewer for the positive assessment and for taking the time to review the additional results and clarifications. We are glad that the comparison with static FP8 has made the trade-offs clearer, and that the per-request BASE exposure results and explanation of CPU backing have been helpful.

Thanks again for the thoughtful engagement and continued support for this work!

Reviewer_MeZZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ElasticKV reduces KV memory pressure by letting requests execute with an 8-bit BASE representation and restoring 16-bit KV when capacity permits. It combines paired-block storage, mixed-fidelity attention, and pressure-driven fidelity control to reduce preemptions and serving latency.

Critical strengths
  • The design turns reduced precision into usable GPU capacity: packing two BASE blocks frees a physical block, while attention directly consumes the packed representation.
  • The evaluation extends beyond synthetic workloads to ShareGPT and production traces, including a 70B model on AMD GPUs. The ablations also demonstrate the benefits of reserve-aware promotion and structured attention.
Critical weaknesses
  • Static FP8 already outperforms ElasticKV in Table 10: P90 TTFT is 1.47 vs. 2.87 seconds on Llama and 1.50 vs. 1.98 seconds on Qwen, with higher throughput. Different backends complicate attribution, but the paper still needs to establish when its quality advantage justifies this performance gap.
  • Average quality scores hide task-specific losses. At the 20% exposure cap, MuSiQue drops from 32.72 to 31.04 on Llama and from 36.30 to 34.26 on Qwen (Tables 4–5). These losses deserve discussion alongside the near-FP16 averages.
  • The quality study selects runs within exposure caps, but Algorithm 1 does not enforce those caps. Moreover, exposure is aggregated across requests and steps, so a low average can hide heavy BASE use for individual requests. The paper needs per-request exposure and quality results under the actual serving policy.
  • Demotion requires recoverable CPU backing, yet the implementation does not clearly explain when that backing is created or account separately for its transfer cost (§3.3; Appendix B). This matters because blocks without backing cannot be demoted precisely when memory relief is needed.
Questions for authors

Please see the weaknesses.

5 comments
AuthorsOpenReview ↗

Response to Reviewer MeZZ (Part 1/2)

We sincerely thank the reviewer for the careful reading and constructive feedback, and for recognizing the effectiveness of our system design and the breadth of our evaluation. We have uploaded a revised manuscript and provide esponses below.

W1: Static FP8 already outperforms ElasticKV in Table 10 ... Different backends complicate attribution, but the paper still needs to establish when its quality advantage justifies this performance gap.

We agree that static FP8 is an important deployment alternative. The native FP8 path in vLLM’s Triton backend requires FP8 conversions unsupported on A100 (SM80) [1, 2], which is why our original comparison used FlashInfer [3]. To address the backend mismatch, we have implemented an A100-compatible Triton FP8 reference using uint8 cache accesses and additional GPU instructions for FP8 conversion. We have further optimized conversion through bitwise decoding and FlashInfer-inspired packed SIMD conversion [3], while retaining the native Triton attention structure.

We have repeated the Fig. 4 sweep for both models and report the results in Sec. 4.4 of the revised manuscript. Compared with the optimized Triton FP8 reference, ElasticKV generally achieves 11%-17% lower mean TPOT and 15%-22% higher throughput. ElasticKV has lower mean TTFT at low concurrency, while static FP8 has lower mean TTFT at higher concurrency. Static FP8’s uniformly compressed KV provides more capacity headroom, while ElasticKV retains FP16 where possible and reclaims capacity on demand. This can favor FP8’s request startup under heavy memory pressure, even without preemption. However, static FP8’s TTFT advantage does not extend to TPOT or throughput in this experiment because the required conversion adds decoding work on A100 [2]. We retain the original FlashInfer results in App. C.6.2 as a practical deployment reference.

Regarding the practical choice, ElasticKV is particularly valuable for quality-sensitive services that aim to maximize generation quality subject to latency and throughput SLOs. This goal is relevant to common applications such as question answering (QA) and code completion, where generation quality directly affects the usefulness of the service. Its TPOT and throughput advantages in the matched Triton comparison also matter for streaming generation and longer responses. Even when static FP8 offers lower TTFT, its quality loss compared with FP16 can be substantial on certain tasks. For example, on MuSiQue, static FP8 reduces QA F1 by 3.89 and 4.69 points on Llama-3.1-8B-Instruct and Qwen3-8B, respectively, while ElasticKV limits these reductions to 0.78 and 2.02 at around 19% BASE exposure (see App. C.1). Therefore, when ElasticKV meets the required serving SLOs, its smaller quality loss can justify choosing it even where static FP8 offers better performance. We have added a dedicated discussion of the deployment trade-offs in Sec. 4.4 of the revised paper.

[1] vLLM Project. "triton_attn". GitHub source code. https://github.com/vllm-project/vllm/blob/v0.15.0/vllm/v1/attention/backends/triton_attn.py#L472. Accessed 2026.

[2] NVIDIA. "Parallel Thread Execution ISA, Version 8.8 (Data Movement and Conversion Instructions: cvt)". CUDA documentation. https://docs.nvidia.com/cuda/archive/12.9.1/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-cvt. Accessed 2026.

[3] Ye et al. "FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving." MLSys, 2025.

W2: Average quality scores hide task-specific losses. At the 20% exposure cap, MuSiQue drops from 32.72 to 31.04 on Llama and from 36.30 to 34.26 on Qwen (Tables 4–5). These losses deserve discussion alongside the near-FP16 averages.

Thanks for the suggestion. We have now strengthened the LongBench evaluation to better characterize task-level quality costs. We relaxed the highest exposure cap from 20% to 25% and repeated this setting three times with different request orders. The updated results have been incorporated into Sec. 4.2 and App. C.1, Tables 1, 4, and 5. The revised MuSiQue results show F1 losses of 0.78 points on Llama-3.1-8B-Instruct and 2.02 points on Qwen3-8B relative to FP16.

We agree that these losses deserve discussion alongside the near-FP16 aggregate averages. We have added a discussion of MuSiQue, HotpotQA, and 2WikiMQA in App. C.1, explicitly acknowledging task-dependent quality losses. These tasks require connected reasoning across multiple documents, where reduced KV fidelity may affect evidence selection or integration and thereby alter the final answer.

AuthorsOpenReview ↗

Response to Reviewer MeZZ (Part 2/2)

W3: The quality study selects runs within exposure caps, but Algorithm 1 does not enforce those caps. Moreover, exposure is aggregated across requests and steps, so a low average can hide heavy BASE use for individual requests. The paper needs per-request exposure and quality results under the actual serving policy.

We agree that per-request BASE exposure is important. We have added measurements under the actual serving policy at three representative serving settings, including the highest-concurrency synthetic setting for each model and the A100 Mooncake Window C replay. We have reported and discussed the results in App. C.9 and also summarized them below.

Serving settingRequestsE = 00 < E ≤ 10%10% < E ≤ 20%20% < E ≤ 40%E > 40%P99 exposure
Synthetic, Llama-3.1-8B, c = 406006.00%9.50%44.17%40.33%0.00%27.36%
Synthetic, Qwen3-8B, c = 386004.00%6.17%47.50%42.33%0.00%29.97%
Mooncake, A100 Window C, d = 161,12233.07%31.46%22.99%11.94%0.53%35.01%

For the synthetic workloads, BASE execution is spread across most requests, and P99 exposure remains below 30% on both models. Mooncake exhibits a longer right tail, but 87.52% of requests remain at or below 20%, and only six requests (i.e., 0.53%) exceed 40%. This variation reflects the fluctuating traffic and variable request lengths in the realistic production trace. Requests executing entirely during low-pressure intervals can avoid demotion, while those spending more of their execution under sustained pressure may incur higher exposure. Longer-running requests may also span multiple pressure episodes.

We evaluate quality on LongBench using the same pressure-driven fidelity-management policy as in the serving experiments (see Sec. 4.2 and App. C.1). The reported exposure caps are evaluation-side selection thresholds and are not enforced by Alg. 1.

W4: Demotion requires recoverable CPU backing, yet the implementation does not clearly explain when that backing is created or account separately for its transfer cost (§3.3; Appendix B). This matters because blocks without backing cannot be demoted precisely when memory relief is needed.

Exact CPU backing is created asynchronously through vLLM’s native offloading path as KV blocks become complete. A pair becomes demotion-eligible only after both blocks have valid CPU copies. Therefore, demotion uses previously established backing and requires no new D2H backup. The cost of these transfers is reflected in our end-to-end serving results. We have clarified the CPU backing mechanism in App. B.2.

AuthorsOpenReview ↗

Looking forward to further feedback

We thank the reviewer again for the careful review and constructive feedback. With the discussion period nearing its end, we would be grateful if the reviewer could take a look at our responses and let us know whether the concerns have been addressed. This would allow us to address any remaining concerns before the discussion closes.

For convenience, we briefly summarize the main additions already reported in our responses and incorporated into the revised manuscript.

  • FP8 comparison (W1). We implemented an optimized Triton FP8 reference for A100 and ran the full concurrency sweep on both models. In this comparison, ElasticKV generally achieves lower TPOT and higher throughput, while FP8 has lower TTFT at high concurrency.
    Manuscript updates: We added a dedicated section (Sec. 4.4, Fig. 5) to present these new results and discuss the deployment trade-offs.

  • Task-level quality evaluation (W2). We added an explicit discussion of task-level quality losses alongside the near-FP16 averages. We also strengthened the LongBench evaluation with three repetitions of the 25%-cap setting using different request orders.
    Manuscript updates: The updated results and analysis are now included in Sec. 4.2 and App. C.1 (Tables 1, 4, and 5).

  • Per-request exposure (W3). We added per-request exposure measurements under the actual serving policy for three representative settings. BASE exposure is spread across most synthetic requests. In realistic A100 Mooncake Window C, most requests still have limited exposure and only a small fraction experience high exposure.
    Manuscript updates: We reported these results in App. C.9 (Tab. 14).

  • CPU backing and transfer cost (W4). We clarified that exact CPU copies are created asynchronously as KV blocks become complete. Demotion requires no additional D2H backup, and transfer cost is included in our serving results.
    Manuscript updates: We clarified the mechanism in App. B.2.

Our earlier responses and the revised manuscript provide the supporting results and detailed explanations. We hope these additions and clarifications address the main concerns, and we would be very grateful if the reviewer would consider updating the rating in light of these additions.

Reviewer_Zc31OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that paged serving runtimes like vLLM only treat KV as executable in its target precision, so running short of KV memory turns directly into stalls, preemptions and recomputation. ElasticKV (built on vLLM 0.15.0) instead demotes eligible 16-bit KV blocks to a BASE form, the high byte of each value, keeps executing on it, and later restores the exact values from a CPU-side copy. Two demoted blocks are packed into one physical block, a dual-mode Triton backend handles TARGET-only and mixed batches (zero-filling the low byte in registers), and a controller demotes ahead of need using a dry run of the scheduler and promotes asynchronously. Quality is evaluated on LongBench with Llama-3.1-8B and Qwen3-8B under 5/10/20% exposure caps. Serving is evaluated on one A100 with a 20 GB KV budget (synthetic, ShareGPT and Mooncake traces) and on 8×MI355X with Llama-3.1-70B. Against FP16 vLLM without offloading, the paper reports 3.8–4.0× lower mean and 9.1× lower P90 TTFT at the highest concurrency (2.0–2.6× and 2.5–4.7× against vLLM with KV offloading), zero preemptions, LongBench within about 0.2 points of FP16, and 4.37 ms/step of overhead.

Critical strengths
  1. This is a complete, carefully instrumented implementation inside vLLM rather than a simulation. The pair layout is a clean fix for fragmentation, the high-byte BASE makes restoration exact with no scales, the native path is untouched for TARGET-only batches, the overhead is broken down per component (0.32 + 2.82 + 1.23 = 4.37 ms/step), and each control mechanism is ablated (e.g., eager promotion raises P90 TTFT from 2.8 to 4.4 s).

  2. The benefit shows up across many settings in the preemption regime: the synthetic sweep, ShareGPT (1297–1594 baseline preemptions vs. 0), Mooncake replays, KV-budget and context-length sweeps, and a 70B model on MI355X. The headline ratios I spot-checked against Fig. 4 and Table 9 are consistent.

Critical weaknesses
  1. Static FP8 KV, which vLLM already supports, is the obvious competitor and only appears in App. C.5. In the same headline setting it is faster: P90 TTFT 1.47 vs. 2.87 s (Llama) and 1.50 vs. 1.98 s (Qwen), with higher throughput (Table 10), at a LongBench cost of 1.17/0.52 points. The backend mismatch (FlashInfer vs. Triton) is acknowledged but not resolved. The abstract's 3.8–4.0× / 9.1× is also measured against FP16 vLLM with no host memory, while ElasticKV itself relies on host backing.

  2. The baselines seem to run in a thrashing regime that ordinary configuration would avoid (e.g., 4989 preemptions for 600 homogeneous requests on Qwen3-8B at concurrency 36). Scheduler settings are not reported and there is no admission-control or free-block-reserve baseline. On MI355X windows B and C, ElasticKV roughly halves mean TTFT and cuts drain time by 3.7–8.7× with only 0.01–0.02% BASE exposure, which suggests most of the gain comes from avoiding preemption cascades rather than from extra capacity.

  3. The quality evidence does not match the serving operating points. Exposure is a run-level average; per-request exposure in the serving runs is never reported, and the A100 Mooncake runs stay at 25–50% resident BASE for long stretches. The tasks most sensitive to BASE never reached high exposure in the LongBench runs: for Qwen3-8B, the 8 tasks outside the ≥18% subset lose 3.12 points under All-BASE vs. 0.86 for the 13 inside it (Table 5). The results are single runs on a mostly short-output benchmark.

  4. The method is effectively tied to FP16 KV. The high byte of an FP16 value is a truncated E5M2 number (All-BASE is below static FP8 on both models), and for BF16 the high byte carries no mantissa bits at all. The paper does not say whether the models were served in FP16 or BF16 and does not discuss this limitation.

Questions for authors
  1. Could you compare against static FP8 on the same attention backend across the Fig. 4 sweep, together with quality? If FP8 still dominates, in which regime is ElasticKV the better choice?

  2. What were the vLLM scheduler settings (max_num_seqs, max_num_batched_tokens, chunked prefill, preemption mode, block size), and how do Full-GPU and Exact-offload behave with a capped running-request limit or a small free-block reserve?

  3. Were the KV caches FP16 or BF16, and how would BASE work for BF16? Which FP8 format and scaling does the reference use?

  4. Can you report the per-request BASE-exposure distribution for the serving runs? Also, §4.6 states 21.7%/21.8% lower TTFT at 32K, but the Fig. 8 bars read closer to 43%/38%. Which is correct?

7 comments
AuthorsOpenReview ↗

Response to Reviewer Zc31 (Part 1/3)

We sincerely thank the reviewer for the careful reading and constructive feedback, and for recognizing the completeness of our implementation and its consistent benefits under memory pressure. We have uploaded a revised manuscript and provide responses below.

W1.1: Static FP8 KV, which vLLM already supports, is the obvious competitor … The backend mismatch (FlashInfer vs. Triton) is acknowledged but not resolved.

Q1: Could you compare against static FP8 on the same attention backend across the Fig. 4 sweep, together with quality? If FP8 still dominates, in which regime is ElasticKV the better choice?

We agree that static FP8 is an important deployment alternative. The native FP8 E4M3 path in vLLM’s Triton backend requires FP8 conversions unsupported on A100 (SM80) [1, 2], which is why our original comparison used FlashInfer [3]. Following the reviewer’s suggestion, we have implemented an A100-compatible Triton FP8 reference using uint8 cache accesses and additional GPU instructions for FP8 conversion. We have further optimized conversion through bitwise decoding and FlashInfer-inspired packed SIMD conversion [3], while retaining the native Triton attention structure.

We have repeated the Fig. 4 sweep for both models and report the results in Sec. 4.4 of the updated manuscript. Compared with the optimized Triton FP8 reference, ElasticKV generally achieves 11%-17% lower TPOT and 15%-22% higher throughput. At low concurrency, ElasticKV reduces mean TTFT by 10% and 19%, respectively. At higher concurrency, static FP8 achieves lower TTFT, including a 28% reduction on Llama-3.1-8B-Instruct at concurrency 40. Static FP8’s uniformly compressed KV provides more capacity headroom, while ElasticKV retains FP16 where possible and reclaims capacity on demand. This can favor FP8’s request startup under heavy memory pressure, even without preemption. However, static FP8’s TTFT advantage does not extend to TPOT or throughput due to its additional conversion overhead during decoding on A100 [2].

Regarding the practical choice, ElasticKV is particularly valuable for quality-sensitive services that aim to maximize generation quality subject to latency and throughput SLOs. This goal is relevant to common applications such as question answering (QA) and code completion, where generation quality directly affects the usefulness of the service. Its TPOT and throughput advantages also matter for streaming generation and longer responses. Even when static FP8 offers lower TTFT, its quality loss compared with FP16 can be substantial on certain tasks. For example, on MuSiQue, static FP8 reduces QA F1 by 3.89 and 4.69 points on Llama-3.1-8B-Instruct and Qwen3-8B, respectively, while ElasticKV limits these reductions to 0.78 and 2.02 at around 19% BASE exposure (see App. C.1). Therefore, when ElasticKV meets the required serving SLOs, these performance and quality benefits can favor its deployment. We have added a dedicated discussion of the deployment trade-offs in Sec. 4.4 of the revised paper.

[1] vLLM Project. "triton_attn". GitHub source code. https://github.com/vllm-project/vllm/blob/v0.15.0/vllm/v1/attention/backends/triton_attn.py#L472. Accessed 2026.

[2] NVIDIA. "Parallel Thread Execution ISA, Version 8.8 (Data Movement and Conversion Instructions: cvt)". CUDA documentation. https://docs.nvidia.com/cuda/archive/12.9.1/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-cvt. Accessed 2026.

[3] Ye et al. "FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving." MLSys, 2025.

W1.2: The abstract's 3.8–4.0$\times$ / 9.1$\times$ is also measured against FP16 vLLM with no host memory, while ElasticKV itself relies on host backing.

The original abstract’s ratios are relative to the default FP16 vLLM. Against FP16 vLLM with host backing under the same memory budgets, ElasticKV achieves 2.0-2.6$\times$ lower mean TTFT and 2.5-4.7$\times$ lower P90 TTFT. We have revised the abstract to report this comparison.

W2.1: Scheduler settings are not reported.

Q2.1: What were the vLLM scheduler settings (max_num_seqs, max_num_batched_tokens, chunked prefill, preemption mode, block size)?

We used the default vLLM scheduler settings in all the serving experiments. On A100, max_num_seqs=256 and max_num_batched_tokens=2048. On MI355X, these values were 1024 and 8192, respectively. Both platforms used chunked prefill, recomputation-based preemption, and 16-token KV blocks. We have explicitly reported these settings in Sec. 4.1.

AuthorsOpenReview ↗

Response to Reviewer Zc31 (Part 2/3)

W2.2: The baselines seem to run in a thrashing regime that ordinary configuration would avoid … and there is no admission-control or free-block-reserve baseline.

Q2.2: How do Full-GPU and Exact-offload behave with a capped running-request limit or a small free-block reserve?

We have added experiments with both suggested admission controls for Full-GPU and Exact-offload on Llama-3.1-8B-Instruct and Qwen3-8B. We separately test running-request caps of 32 and 28 and free-block reserves of 10% and 20% at the concurrency points where the original baselines incur preemptions, keeping all other settings unchanged from Fig. 4.

The new results have been incorporated into App. C.2, Figs. 10, 11. Overall, both strategies can substantially reduce preemptions, but these reductions rarely translate into lower TTFT. Many settings even increase TTFT. As an example, the table below shows the results on Qwen3-8B at concurrency 36. In this case, request cap 28 eliminates all Full-GPU preemptions and largely improves its throughput, but its mean TTFT remains 4.41$\times$ that of ElasticKV and its throughput is lower. For Exact-offload, all four tuning configurations reduce preemptions but increase mean TTFT and slightly decrease throughput. These results suggest that limiting request admission relieves KV pressure, but the benefits of fewer preemptions can be offset by additional queueing and reduced concurrency.

SystemSettingPreemptionsMean TTFT (s)Mean TPOT (ms)Throughput (tok/s)
Full-GPUOriginal498912.8697.49325.22
Full-GPURequest cap 32309613.1074.02405.53
Full-GPURequest cap 28011.7637.45711.60
Full-GPUReserve 10%166912.9357.13504.73
Full-GPUReserve 20%8013.5836.41699.55
Exact-offloadOriginal579.37.0541.63733.01
Exact-offloadRequest cap 323658.1840.73716.49
Exact-offloadRequest cap 282112.1637.04711.50
Exact-offloadReserve 10%8510.3438.01724.22
Exact-offloadReserve 20%813.6535.16715.93
ElasticKVOriginal02.6738.85854.92

W3.1: Exposure is a run-level average; per-request exposure in the serving runs is never reported, and the A100 Mooncake runs stay at 25–50% resident BASE for long stretches.

Q4.1: Can you report the per-request BASE-exposure distribution for the serving runs?

We agree that per-request BASE exposure is important. We have added measurements at three representative serving settings, including the highest-concurrency synthetic setting for each model and the A100 Mooncake Window C replay. We have reported and discussed the results in App. C.9 and also summarized them below.

Serving settingRequestsE = 00 < E ≤ 10%10% < E ≤ 20%20% < E ≤ 40%E > 40%P99 exposure
Synthetic, Llama-3.1-8B, c = 406006.00%9.50%44.17%40.33%0.00%27.36%
Synthetic, Qwen3-8B, c = 386004.00%6.17%47.50%42.33%0.00%29.97%
Mooncake, A100 Window C, d = 161,12233.07%31.46%22.99%11.94%0.53%35.01%

For the synthetic workloads, BASE execution is spread across most requests, and P99 exposure remains below 30% on both models. Mooncake exhibits a longer right tail, but 87.52% of requests remain at or below 20%, and only six requests (i.e., 0.53%) exceed 40%. This variation reflects the fluctuating traffic and variable request lengths in the realistic production trace. Requests executing entirely during low-pressure intervals can avoid demotion, while those spending more of their execution under sustained pressure may incur higher exposure. Longer-running requests may also span multiple pressure episodes. The 25-50% resident BASE ratio in Window C is an instantaneous residency statistic and cannot be directly interpreted as cumulative execution exposure for individual requests.

AuthorsOpenReview ↗

Response to Reviewer Zc31 (Part 3/3)

W3.2: The tasks most sensitive to BASE never reached high exposure in the LongBench runs: for Qwen3-8B, the 8 tasks outside the ≥18% subset lose 3.12 points under All-BASE vs. 0.86 for the 13 inside it (Table 5). The results are single runs on a mostly short-output benchmark.

We have now strengthened the LongBench evaluation by (i) repeating the highest-cap setting (25%) three times and (ii) expanding coverage at ≥18% BASE exposure from 13/21 to 19/21 tasks on both models. The repetitions use different request orders while keeping other settings fixed. These results have been incorporated into Sec. 4.2 and App. C.1, Tabs 1, 4, and 5. Across all 21 tasks, the highest-cap average differs from FP16 by +0.03+0.03 points on Llama-3.1-8B-Instruct and −0.32-0.32 points on Qwen3-8B. Meanwhile, six of the eight Qwen3-8B tasks mentioned in the review now reach this 18% threshold. On these six tasks, ElasticKV loses 0.90 points relative to FP16, compared with 3.74 points under All-BASE.

W4.1: The method is effectively tied to FP16 KV…. The paper does not say whether the models were served in FP16 or BF16 and does not discuss this limitation.

Q3: Were the KV caches FP16 or BF16, and how would BASE work for BF16? Which FP8 format and scaling does the reference use?

All models in our experiments are served in FP16, and we have made this explicit in Sec. 4.1 in the revised paper. The static FP8 reference uses E4M3 and vLLM’s default KV cache scaling configuration (i.e., calculate_kv_scales=False). We agree that the current high-byte BASE encoding is specific to FP16, and BF16 cannot be directly adopted with the same quality expectations. A potential BF16 extension is to use a scaled 8-bit BASE representation while preserving the original BF16 KV in CPU backing for exact restoration. This can reuse the proposed design of pair-structured elastic KV layout and pressure-aware fidelity management. We leave BF16 implementation and evaluation to future work and have explicitly discussed this limitation in Sec. 3.4.

Q4.2: §4.6 states 21.7%/21.8% lower TTFT at 32K, but the Fig. 8 bars read closer to 43%/38%. Which is correct?

We thank the reviewer for pointing this out. The 21.7%/21.8% reductions in Sec. 4.7 (original Sec. 4.6) are correct. The 32K Exact-offload bars in Fig. 9 (original Fig. 8) mistakenly used results from an earlier configuration with a fixed 48GB GPU KV budget, instead of the final configuration using vLLM’s gpu_memory_utilization=0.9. The correct mean TTFT values of Exact-offload are 54.19s and 77.79s on the two models. We have corrected this figure accordingly. We have also added this GPU memory utilization setting to the revised paper.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of rebuttal and discussion

We sincerely thank the reviewers for their constructive feedback and the Area Chair for their time and consideration. As the discussion period concludes, we briefly summarize the paper’s core contribution and the strengths recognized by reviewers, then clarify a major remaining misunderstanding regarding its practical value relative to static FP8.

To keep LLM requests executing under GPU memory pressure, our key idea is to execute directly with compact KV and restore the original KV values when resources permit. We introduce ElasticKV, a novel systems approach that proposes a mixed-fidelity execution contract and the required coordinated changes to the entire runtime, spanning attention kernels, physical KV organization, and online fidelity management. Reviewer NvLx explicitly appreciated the core insight that target fidelity need not block execution and how the components fit together. Reviewer MeZZ highlighted the capabilities and benefits of our key design components. Reviewer Zc31 recognized the complete implementation and its benefits across serving settings.

However, we believe the remaining concern about static FP8 partly reflects a misunderstanding of ElasticKV’s deployment objective. We would like to highlight that ElasticKV and static FP8 naturally offer different quality-efficiency trade-offs. Specifically, static FP8 keeps all KV at 8-bit throughout execution, providing greater memory capacity headroom by nature. ElasticKV retains FP16 where possible and uses reduced fidelity adaptively under memory pressure to preserve quality. Therefore, ElasticKV is particularly useful for quality-sensitive services that seek to preserve generation quality while meeting latency and throughput SLOs.

Our additional experiments and deployment discussion make this trade-off concrete. Reviewer NvLx acknowledged that the matched-backend comparison makes the trade-off clearer. Our latest follow-up presents serving results on MI355X with native FP8 support alongside LongBench quality results. These results show substantially better quality preservation on the question answering and code completion examples with near-FP8 throughput in the tested setting. When serving requirements are met, these quality benefits can favor ElasticKV even when static FP8 provides a faster first token.

We hope this summary helps to clarify ElasticKV’s core contribution and its quality-efficiency trade-off. Its practical value lies in giving quality-sensitive services the flexibility to reclaim capacity under memory pressure while limiting reduced-fidelity execution.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.