acceptodds
Under review as a conference paper at ICLR 2027

BOLT-KV: Rethinking KV Movement in Confidential LLM Serving

Abstract

Large language model (LLM) serving moves key-value (KV) caches between GPU and CPU memory to support long contexts and concurrent requests. While one request's cache moves in the background, other requests rely on small CPU–GPU transfers to continue decoding. Confidential computing (CC) protects data during execution and CPU–GPU communication, but can reduce transfer concurrency. On an H100 GPU with CC enabled, background KV movement reduces vLLM decoding throughput by roughly 30% relative to no movement, while the same background transfer workload leaves a non-CC comparison system near baseline. We identify two CC-associated interference modes: CUDA's legacy default stream can delay otherwise unrelated foreground operations, and bulk transfers can delay small transfers in the same direction even on separate streams. Foreground delay closely follows the time remaining until the active transfer completes, and stream isolation can shift the bottleneck to output readback. We present BOLT-KV, which removes avoidable legacy-stream coupling and divides selected KV movement into separately submitted, byte-bounded chunks. The resulting boundaries create opportunities for foreground work while preserving transfer order and completion dependencies. During GPU-to-CPU KV movement, BOLT-KV raises SGLang throughput from 78.5% to 99.7% of its no-movement baseline and reduces p99 inter-token latency from 101 to 8 ms. On vLLM's KV-store backend, throughput rises from 80.1% to 88.5% of its no-movement baseline, while p99 latency falls from 74 to 41 ms. BOLT-KV also reduces foreground disruption during the CPU-to-GPU KV restoration transfers, allowing protected KV movement to coexist more effectively with latency-sensitive confidential serving.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
LLM Serving, Confidential Computing, KV Cache, KV Cache Offloading, GPU Systems

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_bwMoOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies interference between background KV-cache movement and latency-sensitive foreground serving under GPU Confidential Computing (CC). Using framework-independent probes, the authors identify two CC-associated interference modes: (1) legacy-stream-dependent interference, where CUDA's legacy default stream implicitly synchronizes with blocking streams and delays otherwise unrelated foreground operations, and (2) same-direction occupation interference, where bulk transfers delay smaller transfers in the same direction even on separate streams. They show foreground delay approximately equals the residual time until the active transfer completes (slope ≈ 1.001, R² ≈ 1), and that stream isolation (PBSI) merely relocates the bottleneck to mandatory output readback rather than removing it. Based on this characterization, the paper proposes BOLT-KV, which (a) removes avoidable legacy-stream coupling via explicit non-blocking streams, and (b) divides KV movement into byte-bounded "bounded submissions," submitting the next chunk only after the previous one reaches a backend-specific issuance boundary — creating opportunities for foreground work without explicit prioritization or synchronization primitives. Implemented in both SGLang and vLLM, BOLT-KV raises SGLang throughput from 78.5% to 99.7% of the no-movement baseline (ITL p99: 101 → 8 ms) and vLLM KV-store throughput from 80.1% to 88.5% (ITL p99: 74 → 41 ms). During H2D restoration, BOLT-KV Restore more than doubles foreground progress versus coarse restore, exposing a trade-off between recovery completion (where Reprefill wins) and foreground responsiveness.

Critical strengths
  1. Novel problem discovery with precise characterization: The paper is the first to identify and quantify two distinct CC-associated interference modes. The measurement that CC drops serving throughput to R = 0.701 with foreground input-H2D p99 of 122.5 ms (vs. 0.29 ms on NonCC), while NonCC under 4.5× pressure remains at R = 0.981, convincingly demonstrates that the interference does not arise on non-CC systems.
  2. Strong empirical model : The residual-occupation model — foreground delay ≈ remaining duration of the active transfer (α = 1.001, R² ≈ 1) — is simple, predictive, and validated both in framework-independent probes (Tables 7–8) and at serving level (Table 2). The falsifiable predictions P1–P3 with pre-registered factorial validation (Section 5.1) reflect excellent experimental discipline.
  3. Elegant, low-overhead design: Bounded Submissions + issuance boundaries require no foreground-priority gate, no added synchronization primitives, and no deliberate idling, yet the factorial experiment (Table 3) cleanly demonstrates positive interaction (+0.08) between PBSI and bounded submissions — showing they address complementary interference modes.
  4. Honest limitation reporting and dual-backend generality: The paper retains rare ~200 ms excursions rather than trimming them (P3 explicitly acknowledges bounded submissions "reduce tails but do not bound them"), and reproduces headline effects on a second model (Llama-3.1-Nemotron-Nano-8B, Table 17) with pre-registered predictions.
  5. Addresses a real pain point on the serving critical path: By pinpointing where the interference enters the serving critical path (legacy-stream coupling vs. same-direction readback) and how to treat it, the paper fills a genuine gap — prior CC studies exposed serialized transfers but did not clarify the interaction on the serving path or the treatment.
Critical weaknesses
  1. Calibration audit contradicts the headline configuration: Appendix B.3 (Table 11) shows 2 MiB meets the latency criterion and is selected by the pre-defined fallback utility function U(q), while 4 MiB fails both primary criteria — yet the headline experiments retain 4 MiB justified only by "pre-registration." This leaves a "picked a sub-optimal point" suspicion that a 2 MiB end-to-end result is needed to resolve.
  2. CC vs. NonCC contrast is confounded, so the core motivation's causality is weak: The paper itself admits the two hosts differ in drivers (595.71.05 vs. 610.43.02) and VM configurations (Footnote 2; A.2, lines 700–701). Thus "CC causes ~30% throughput drop" can only be claimed as "CC-associated," not causal. Without a same-driver same-VM CC on/off control, the foundational motivation is on unstable ground, and the mechanism cannot be located inside NVIDIA's closed CC implementation.
  3. SGLang controlled replay bypasses the real scheduler: The replay mover directly accesses live KV buffers in the scheduler process and CUDA context (lines 1024–1025), sidestepping the real scheduler's queuing effects; and SGLang's end-to-end claims rest on this controlled replay rather than the runtime's native KV lifecycle.
  4. vLLM restoration experiment does not measure real recovery: Both restore arms write identical prepared KV bytes to a validated GPU scratch destination rather than resuming actual attention (lines 429–431, C.5). The experiment therefore measures only the transfer component of restoration, not the true restoration-into-serving path — limiting the external validity of the H2D restoration conclusions.
  5. Scope and robustness limits: Evaluation covers a single H100 NVL CC platform with Blackwell untested; the deterministic guarantee is explicitly absent (P3: rare long waits persist, lateness reaches p99 2.26 s at highest offered load); and vLLM's improvement is markedly smaller than SGLang's (88.5% vs. 99.7% of baseline), suggesting backend-dependent benefit magnitude that is underexplored.
Questions for authors
  1. Calibration consistency: Given that Table 11's own audit criteria select 2 MiB (and 4 MiB fails both primary criteria), can you provide the end-to-end serving results at q_max = 2 MiB? If 2 MiB performs equal or better, why retain 4 MiB as the headline operating point?
  2. Causality of the CC contrast: Can you run a CC-on/CC-off comparison on the same host with the same driver version and VM configuration (i.e., toggle CC mode only)? Without this, how do you justify the claim that the ~30% degradation is attributable to confidential computing itself rather than the confounding driver/VM differences?
  3. End-to-end validity of the replay/scratch methodology: (a) Could you integrate the bounded-submission mover into SGLang's actual KV lifecycle (not controlled replay) and report results? (b) For H2D restoration, can you evaluate the real path where restored KV is re-entered into attention, rather than writing to a validated scratch destination?
2 comments
AuthorsOpenReview ↗

Response to Reviewer bwMo

We thank the reviewer for recognizing the novel problem characterization, strong empirical validation, and practical design. We have conducted additional experiments and clarified the evaluation scope to address the concerns.


W1 & Q1. Calibration consistency and submission size

"If 2 MiB performs equal or better, why retain 4 MiB as the headline operating point?"

We added an end-to-end submission-size sweep (Table 13). At 2 and 4 MiB, throughput retention is nearly identical (R = 0.999 vs. 0.998), with ITL p99 of 7.58 vs. 8.00 ms. However, under higher movement load, 2 MiB increases background lateness p99 to 3.59 s, compared with 2.25 s at 4 MiB.

Thus, 4 MiB represents a reasonable foreground–background trade-off, not a universally optimal setting. We retain the originally fixed configuration while transparently reporting that the subsequent calibration audit selects 2 MiB.

Manuscript revisions: We added the end-to-end sensitivity analysis (§5.2; Appendix B.3, Table 13) to clarify the trade-off and resolve the apparent calibration inconsistency.


W2 & Q2. CC versus NonCC attribution

"CC vs. NonCC contrast is confounded, so the core motivation's causality is weak"

We repeated the comparison with matching NVIDIA drivers (595.71.05). Under the same D2H trace, throughput retention is 0.689 with CC versus 1.001 without CC, and input-H2D p99 is 123.4 versus 0.234 ms.

Importantly, our central findings do not depend on this cross-environment comparison: the interference modes, residual-occupation relationship, and BOLT-KV improvements are independently validated within the same CC environment. We retain the term CC-associated because VM differences remain.

Manuscript revisions: We replaced the original results with driver-matched measurements (§3.1; Appendix A.2, Table 7) and clarified the remaining confounders in Limitations.


W3–W4 & Q3. End-to-end validity of replay and restoration

"SGLang controlled replay bypasses the real scheduler"

"vLLM restoration experiment does not measure real recovery"

We strengthened the native vLLM KV-store evaluation (Table 4). BOLT-KV improves paired absolute client throughput by 14.5% over true Stock (Appendix C.3), with ITL p99 decreasing from 81.0 to 39.9 ms (Table 4).

We also measured the native store lifecycle with 16 decoding requests, confirming foreground latency improvements while quantifying delayed store completion and GPU-block reuse.

SGLang controlled replay remains a mechanism-isolation experiment. Likewise, H2D restoration uses validated scratch memory to isolate transfer interference; it does not evaluate resumed attention. We explicitly distinguish these results from full native lifecycle and recovery measurements.

Manuscript revisions: We clarified controlled replay versus native serving (§5.2; Appendices C.2–C.3), corrected the vLLM baseline (Table 4), and specified the scope of H2D restoration (§5.3; Appendix C.5).


W5. Scope and robustness

"Scope and robustness limits"

We acknowledge that Blackwell remains untested and deterministic latency guarantees are absent. Additional submission-size and movement-pressure experiments characterize the foreground–background trade-off. The different SGLang and vLLM workloads also preclude a direct comparison of their improvement magnitudes.

Manuscript revisions: We expanded the sensitivity analysis (Appendix B.3), clarified the backend-specific evaluation conditions (§5.2), and explicitly stated the remaining limitations.

AuthorsOpenReview ↗

Additional Evaluation on NVIDIA Blackwell

During the rebuttal period, we gained access to an NVIDIA B300 system with confidential computing enabled, which had not been available at the time of submission. We have incorporated the additional results into Appendix D (Tables 21–24). All revisions in the updated manuscript are highlighted in blue.


1. Calibration consistency (W1, Q1). The reviewer bwMo asked for "end-to-end serving results at q_max = 2 MiB." Our H100 SGLang results show comparable throughput at 2 and 4 MiB (Table 13). We additionally evaluated submission-size sensitivity on B300 CC vLLM (Table 24):

Metric2 MiB4 MiB8 MiB
Throughput (tok/s)491.9517.5497.0
ITL p99 (ms)16.3116.8621.20
Background lateness p99 (s)3.661.751.05

These results illustrate the foreground–background trade-off. We retain the previously fixed 4 MiB configuration without claiming universal optimality.


2. CC versus NonCC confounding (W2, Q2). We additionally compared B300 CC and NonCC under the same offered movement trace (Table 21). Throughput retention was 0.621 under CC versus 0.988 under NonCC. While VM differences still prevent strict causal attribution, this independently reproduces the CC-associated slowdown observed on H100.


3. Evaluation scope (W5). Both interference mechanisms persist on Blackwell (Table 22). BOLT-KV also improves vLLM throughput by 13.1% over Stock and substantially reduces extreme ITL stalls (Table 23).


These additional experiments strengthen the evidence for BOLT-KV across Hopper and Blackwell, while the limitations of controlled replay and scratch-based restoration (W3–W4) remain explicitly acknowledged.

Reviewer_chs4OpenReview ↗
Rating
1
Confidence
3(−1)
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies interference between background KV-cache movement and latency-sensitive decoding under NVIDIA H100 confidential computing (CC). It reports that a fixed background D2H eviction trace reduces vLLM decode throughput by roughly 30% on a CC host while leaving a non-CC host at baseline. Framework-independent probes attribute the slowdown to two mechanisms: (1) legacy-default stream synchronization, which stalls unrelated kernel launches and H2D copies behind an active bulk transfer, and (2) same-direction head-of-line blocking, where a small D2H copy on its own explicit stream still waits behind a bulk D2H transfer. In both cases the added delay closely tracks the remaining duration of the active transfer (regression slope ~1.0, R^2 ~1.0). The proposed system, BOLT-KV, combines (a) moving the inference path to a persistent non-blocking stream (PBSI) and (b) splitting selected KV transfers into byte-bounded submissions issued work-conservingly at backend-specific issuance boundaries. On SGLang throughput recovers from 78.5% to 99.7% of the no-movement baseline and p99 ITL falls from 101 to 8 ms; on vLLM's KV-store backend throughput rises from 80.1% to 88.5% and p99 ITL falls from 74 to 41 ms. A separate H2D restoration experiment shows that chunked restore reduces foreground disruption, but recomputation (Reprefill) completes faster and retains more aggregate foreground throughput. Results are reproduced on a second 8B model.

Critical strengths

S1. Rigorous experimental methodology. The independent process launch is the statistical unit; every movement arm is paired with a matched contemporaneous no-movement control; percentiles are computed within launch prior to aggregation; outliers (a ~200 ms excursion, a ~7 s Stock Restore stall) are retained rather than trimmed; and the second-model predictions were preregistered. This level of care exceeds the norm for serving-systems evaluations.

S2. Clean mechanism isolation. The framework-free probes vary victim stream, direction, and size, and reverse the mover direction. The near-unit residual-occupation slopes, the direction-specificity of Mode II, and the negative result for stream priority together provide strong evidence for the two interference modes. The factorial experiment in Table 3 (interaction approximately +0.08) convincingly establishes that stream isolation and bounded submissions address complementary interference paths.

S3. Large, consistent effect on SGLang with replication. Recovery to within 1% of the no-movement baseline, a 12.6x reduction in p99 ITL, and nearly identical completed movement bytes across arms support partitioning rather than reduced movement as the explanation. The Llama-3.1-Nemotron-Nano-8B replication, using a byte-identical trace with confidence intervals excluding the null, strengthens the result.

Critical weaknesses

W1. The central CC-vs-NonCC confound is unresolved. The two systems are different hosts with different drivers? and different VM configurations. The paper hedges with "CC-associated," but the motivation rests on CC being the cause. CC mode is toggleable on the same H100 (nvidia-smi conf-compute + reboot), so the clean experiment was available and was not run. This make the experiments result untrusting.

W2. The mover's own stream configuration is never specified, and the obvious alternative fix is untested. Tables 7-8 report the victim's stream but not the mover's flags. If the mover sits on a blocking stream, creating it with cudaStreamNonBlocking would remove Mode I without touching the inference path at all.

W3. Headline throughput hides a cost borne by the background work, and the evaluation cannot see it. The calibration audit shows 4 MiB retains ~90% of movement throughput (2 MiB ~85%); Table 15 shows BOLT-KV occupancy rising to 0.77 vs. 0.60 for bulk, lateness p99 of 2.26 s, and nonzero final backlog at the highest load. The foreground is protected in part because eviction is throttled. In a real offloading system, late eviction delays freeing memory and caps admission, a system-level throughput cost that the four-request controlled-replay setup does not measure. An experiment under genuine memory pressure, counting the offloaded requests, is required to establish net benefit. The workloads are also small (4 decoding requests, 96-word prompts); no sensitivity to batch size or context length is reported, though that is where KV tiering matters.

W4. Missing natural baselines and unexplained residual gap. The serving loop knows when readback occurs, so a gap-aware or gated mover (issue chunks only between decode steps; ChunkFlow-style yielding) is the obvious competitor. The paper frames "no foreground-priority gate" as a virtue but never shows the gated variant is worse. Nor is there comparison to PipeLLM-style overlap, mover rate-limiting, or CC-aware scheduling from prior work. On vLLM, BOLT-KV reaches only 88.5% with p99 ITL 41 ms (5x the 8.2 ms no-movement value) and only a 2.2% gain on Nemotron; the source of the remaining interference is not analyzed.

Questions for authors

Q1. Can the same host be evaluated with CC enabled and disabled, holding driver and VM configuration fixed? Q2. What stream flags does the mover use? Does placing only the mover on a cudaStreamNonBlocking stream remove Mode I without PBSI? Q3. Can end-to-end throughput, including offloaded and evicted requests, be reported under memory pressure where eviction is required for admission? Q4. How do the results vary at 32-128 concurrent requests and 16K-32K token contexts? Q5. How does BOLT-KV compare to a gap-aware or gated mover that issues chunks only between decode steps?

3 comments
AuthorsOpenReview ↗

Response to Reviewer chs4 (1/2)

We thank the reviewer for recognizing the rigorous methodology, clean mechanism isolation, and strong replicated results. To address the concerns, we conducted additional experiments that strengthen the CC comparison, rule out an alternative explanation for Mode I, quantify background movement costs, and evaluate foreground-aware gating.


W1 & Q1. CC attribution and environmental confounders

"The central CC-vs-NonCC confound is unresolved."

"Can the same host be evaluated with CC enabled and disabled, holding driver and VM configuration fixed?"

We repeated the CC/NonCC comparison with matching NVIDIA drivers (595.71.05). Under the same D2H trace, throughput retention remains substantially different (0.689 with CC versus 1.001 without CC), as does store-active input-H2D p99 (123.4 versus 0.234 ms). A higher-pressure NonCC control also remains near baseline (R = 0.983).

More importantly, our central findings do not depend on the CC/NonCC comparison. Both interference modes, their residual-occupation characteristics, and BOLT-KV's effectiveness are established through controlled experiments entirely within the same CC environment. The CC/NonCC comparison provides motivation rather than the sole evidence supporting our contributions.

Although a same-host CC on/off comparison would provide stronger causal attribution, the remaining environmental confounders do not affect the validity of our within-CC comparisons. We therefore use CC-associated without attributing the performance gap exclusively to CC.

Manuscript revisions: We updated §3.1 and Appendix A.2 (Table 7) with driver-matched measurements and clarified the remaining environmental confounders in Limitations.


W2 & Q2. Mover stream configuration

"If the mover sits on a blocking stream, creating it with cudaStreamNonBlocking would remove Mode I without touching the inference path at all."

Our additional experiments directly rule out this alternative explanation. We verified using cuStreamGetFlags that the original movers already use explicit non-blocking streams.

We further compared blocking and non-blocking mover streams. Legacy-stream H2D operations exhibit approximately 88 ms p99 latency under both configurations. In vLLM controlled replay, PBSI combined with bounded submissions improves throughput retention from 0.704 to 0.820 with a non-blocking mover, and from 0.702 to 0.822 with a blocking mover.

Thus, making the mover non-blocking does not eliminate Mode I, and the effectiveness of bounded submissions is not explained by the mover's stream flags.

Manuscript revisions: We clarified the original mover configuration and added probe and serving ablations in §3.2, §5.1, and Appendices A.3/C.1 to exclude this alternative explanation.


W3 & Q3–Q4. Background movement cost and system-level performance

"Headline throughput hides a cost borne by the background work, and the evaluation cannot see it."

"An experiment under genuine memory pressure, counting the offloaded requests, is required to establish net benefit."

We extended the evaluation to measure background store completion and GPU memory reuse in vLLM's native KV-store lifecycle, rather than relying solely on foreground throughput.

With 16 decoding requests, BOLT-KV reduces foreground ITL p99 from 285 to 107 ms. We also quantify its cost: mean store-submission-to-GPU-block-unpinning time increases from 69 ms to 2.48 s, and time-averaged pinned GPU KV blocks increase from 110 to 1,093.

We also piloted longer-context workloads with 32 requests at approximately 16K tokens and 16 requests at 32K tokens across three runtime configurations (36 valid launches). None of the unbounded-control launches met our predefined memory-pressure criterion, so we did not proceed to the planned system-wide comparison. Exploratory post-hoc analysis found memory pressure in 9 of 18 BOLT-KV pilot launches, highlighting a potential cost of delayed GPU-block release rather than evidence of net goodput gains.

These experiments establish the foreground–background trade-off and quantify previously unmeasured store-completion costs. We explicitly limit our claims to foreground interference reduction rather than unverified system-wide goodput improvements under memory pressure.

Manuscript revisions: We expanded §5.2 and Appendices C.2–C.3 to quantify movement progress, store completion, and GPU-block reuse costs, and clarified the limits of system-wide performance claims in Limitations.

AuthorsOpenReview ↗

Response to Reviewer chs4 (2/2)

W4 & Q5. Gated mover baseline and residual interference

"The serving loop knows when readback occurs, so a gap-aware or gated mover [...] is the obvious competitor."

"How does BOLT-KV compare to a gap-aware or gated mover that issues chunks only between decode steps?"

We added a foreground-aware gated mover baseline in vLLM controlled replay (Table 5), directly addressing this comparison.

Gating alone does not improve throughput with unbounded transfers (R = 0.702 versus 0.703). Adding gating to BOLT-KV further reduces ITL p99 from 12.31 to 10.95 ms, but yields no conclusive additional throughput improvement (paired ratio 1.006 [0.995, 1.012]). Meanwhile, background movement throughput decreases by 6.8%, and lateness p99 increases from 2.03 to 3.53 s.

These results establish a concrete trade-off: BOLT-KV achieves most of the foreground benefit without explicit gating, while allowing greater background progress. Gating provides a small additional tail-latency benefit, but cannot replace bounded submissions.

Regarding residual interference, the revised native vLLM evaluation shows ITL p99 decreasing from 81.0 ms (Stock) to 39.9 ms (BOLT-KV). We distinguish this native workload from controlled replay and acknowledge that some residual stalls remain unexplained. Other scheduling approaches, including PipeLLM-style overlap, have not been directly compared.

Manuscript revisions: We added the gated-mover comparison in §5.2 (Table 5) and Appendix C.1 to evaluate explicit foreground prioritization and its cost, and clarified residual latency and evaluation limitations in §5.2 and Limitations.

AuthorsOpenReview ↗

Additional Evaluation on NVIDIA Blackwell

During the rebuttal period, we gained access to an NVIDIA B300 system with confidential computing enabled, which had not been available at the time of submission. The additional results are now included in Appendix D (Tables 21–24).


1. CC-vs-NonCC confound (W1, Q1). The reviewer chs4 questioned whether the slowdown could be attributed to CC rather than environmental differences. Our H100 comparison now uses the same driver version. We additionally evaluated B300 CC and NonCC under the same offered movement trace (Table 21):

MetricB300 CCB300 NonCC
Throughput retention0.6210.988
D2H host-call occupancy0.6290.032

These results independently reproduce CC-associated interference, although VM differences still prevent strict causal attribution.


2. Mover stream configuration (W2, Q2). As clarified in our earlier response, the mover already uses a non-blocking stream. The additional ablation confirms that changing mover stream flags alone does not eliminate Mode I (Appendix A.3). All revisions in the updated manuscript are highlighted in blue.


3. Background cost and system-level benefit (W3, Q3). The reviewer chs4 noted that "headline throughput hides a cost borne by the background work." On B300, BOLT-KV improves throughput by 13.1% and reduces mean maximum ITL from 709.9 ms to 42.7 ms, but background lateness p99 increases from 704 ms to 1,747 ms (Table 23). We explicitly acknowledge this trade-off; net system-wide benefits under KV-memory pressure remain unestablished.


The additional experiments strengthen cross-generation validation while making the remaining limitations explicit.

Reviewer_PDgiOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates GPU transfer interference in cloud LLM serving, where background bulk KV-cache transfers and foreground token readbacks contend under CUDA stream synchronization. The paper identifies two interference modes and proposes stream isolation and bounded submission to mitigate the resulting latency, with implementations in vLLM and SGLang. The evaluation uses an H100 NVL platform and Qwen3-8B, including comparisons between confidential-computing (CC) and non-CC environments.

The main contribution is the characterization of transfer interference and the practical integration of existing CUDA stream-management techniques into LLM serving. However, the proposed techniques appear to be adaptations of established mechanisms, and the paper does not clearly establish a novel technical contribution beyond their application to KV transfers.

Critical strengths

Clear problem characterization: The paper identifies two interference modes in LLM KV transfers and explains how foreground token readbacks can be delayed by background bulk transfers under CUDA stream synchronization.

Practical implementation: The proposed techniques are simple to integrate and are implemented in both vLLM and SGLang, potentially making the findings useful to practitioners.

Critical weaknesses
  1. Limited technical novelty: The proposed approach primarily combines existing CUDA stream isolation and bounded submission techniques to mitigate interference in LLM KV transfers. In particular, bounded submission appears to simply submit transfer chunks sequentially, without a new scheduling policy tailored to LLM decode characteristics. While applying these techniques to LLM serving may be practically useful, the paper does not sufficiently establish a novel technical mechanism beyond this application and integration.

  2. Limited evaluation scope: The evaluation is limited to a single GPU platform (H100 NVL) and a single model (Qwen3-8B), leaving the generalizability of the results to other hardware platforms and serving workloads unclear.

  3. Limited differentiation from existing chunking techniques: The paper does not sufficiently establish what additional benefits its chunked submission strategy provides over existing transfer-chunking approaches, beyond applying them to the target KV-transfer scenario.

  4. Confounding factors in CC versus non-CC comparisons: The CC and non-CC environments use different drivers and virtual-machine configurations. These differences make it difficult to attribute the observed performance gap specifically to the overhead of confidential computing.

Questions for authors
  1. Technical novelty: What technical contribution does the proposed bounded submission mechanism provide beyond sequentially submitting transfer chunks using existing CUDA mechanisms? In particular, does it introduce any scheduling policy or guarantee tailored to LLM decode latency, beyond applying existing stream-management techniques to KV transfers?

  2. Attribution of CC overhead: Given that the CC and non-CC environments differ in both driver and VM configuration, what evidence supports attributing the observed performance differences specifically to CC-related overhead rather than these environmental differences?

5 comments
AuthorsOpenReview ↗

Response to Reviewer PDgi

We thank the reviewer for recognizing the clear problem characterization and practical implementation in vLLM and SGLang. We have conducted additional experiments and revised the manuscript to strengthen the technical evidence and clarify our contributions.


W1 & Q1. Technical novelty and bounded-submission mechanism

"The proposed approach primarily combines existing CUDA stream isolation and bounded submission techniques [...]"

"does it introduce any scheduling policy or guarantee tailored to LLM decode latency"

Our contribution is not a new decode-aware scheduling policy, but the characterization of two distinct interference modes and the design of their complementary mitigation.

Importantly, we establish that foreground delay closely tracks the remaining duration of active movement (slope ≈ 1, R² ≈ 1). Moreover, removing one bottleneck can expose another: stream isolation reduces input-side excess from 0.960 to 0.030 ms/iteration, while the observed excess at mandatory output readback increases from 0.559 to 2.567 ms/iteration.

These findings establish the need to control not only stream placement but also the duration of active movement units, motivating BOLT-KV's backend-specific issuance boundaries. Unlike merely partitioning bytes within a single submission, these boundaries create opportunities for foreground execution while preserving logical completion dependencies. We do not claim deterministic latency guarantees.

Manuscript revisions: To clarify the technical contribution, we strengthened the interference characterization and bottleneck-relocation analysis (§§3.2–3.3), specified issuance and completion semantics (§4; Appendix B.2), and clarified the limits of bounded submissions (§3.4).


W2. Limited evaluation scope

"The evaluation is limited to a single GPU platform (H100 NVL) and a single model (Qwen3-8B) [...]"

Our evaluation also includes Llama-3.1-Nemotron-Nano-8B (Appendix C.4, Table 19). Its different KV-block geometry allows two 2 MiB blocks per submission, rather than one 2.25 MiB block for Qwen3-8B. Both models show throughput and tail-latency improvements. Generalizability beyond H100 NVL remains unverified.


W3. Differentiation from existing chunking techniques

"The paper does not sufficiently establish what additional benefits its chunked submission strategy provides over existing transfer-chunking approaches [...]"

Our factorial experiment demonstrates that chunking and stream isolation address complementary interference paths, rather than providing independent, interchangeable benefits.

In Table 3, throughput retention increases from 0.695 to only 0.724 with bounded submissions alone, or to 0.714 with stream isolation alone. Their combination achieves 0.827, yielding a positive interaction of +0.083.

We also added a foreground-aware gating baseline (Table 5). Gating alone provides no throughput improvement with unbounded transfers (R = 0.702 vs. 0.703). Adding gating to BOLT-KV slightly reduces ITL p99 (12.31 to 10.95 ms), but yields no conclusive additional throughput gain (paired ratio 1.006 [0.995, 1.012]), while reducing background movement throughput by 6.8%.

These results establish the benefit of combining stream isolation with bounded submissions, without requiring foreground-aware gating. We do not claim superiority over existing chunking implementations that were not directly compared.

Manuscript revisions: To distinguish BOLT-KV from prior approaches, we revised Related Work (§2), strengthened the factorial evidence (§5.1), and added the gated-mover comparison (§5.2, Table 5; Appendix C.1).


W4 & Q2. CC attribution and environmental confounders

"The CC and non-CC environments use different drivers and virtual-machine configurations."

"what evidence supports attributing the observed performance differences specifically to CC-related overhead"

We repeated the comparison with matching NVIDIA drivers (595.71.05). Under the same D2H trace, throughput retention remains substantially different (0.689 with CC vs. 1.001 without CC), as does store-active input-H2D p99 (123.4 vs. 0.234 ms). A higher-pressure NonCC control also remains near baseline (R = 0.983).

Importantly, our central findings do not depend on the CC/NonCC comparison. The interference modes, residual-occupation relationship, and BOLT-KV improvements are independently validated through controlled experiments within the same CC environment.

Although VM configurations and kernel variants still differ, the driver-matched results strengthen the observed contrast. We therefore characterize the effects as CC-associated, without attributing the performance gap exclusively to CC.

Manuscript revisions: To reduce environmental confounding and clarify causal attribution, we updated §3.1 and Appendix A.2 (Table 7) with driver-matched measurements and documented the remaining differences in Limitations.

AuthorsOpenReview ↗

Additional Evaluation on NVIDIA Blackwell

During the rebuttal period, we gained access to an NVIDIA B300 system with confidential computing enabled, which had not been available at the time of submission. We have incorporated the additional results into Appendix D (Tables 21–24). All revisions in the updated manuscript are highlighted in blue.


1. Limited evaluation scope (W2). The reviewer PDgi noted that our evaluation was limited to "a single GPU platform." Our B300 experiments confirm that both interference modes persist on Blackwell. Bounded submissions reduce D2H readback p99 from 71.19 ms to 1.53 ms (Table 22). Together with our Nemotron evaluation (Appendix C.4), the revised manuscript now covers two GPU generations and two models.


2. CC versus NonCC confounding (W4, Q2). Our H100 comparison now uses the same driver version. On B300, throughput retention is 0.621 under CC versus 0.988 under NonCC with the same offered movement trace (Table 21). VM differences remain, so we claim CC-associated interference rather than strict causality.


3. Technical novelty and chunking differentiation (W1, W3, Q1). Our contribution is not a new priority scheduler or deterministic latency guarantee, but the characterization and complementary mitigation of two interference mechanisms. On B300, PBSI alone improves throughput by 7.8%, while bounded submissions add another 5.0%, yielding a 13.1% overall gain (Table 23).


These results reinforce the generality of the identified mechanisms and the motivation for BOLT-KV's design.

Thank you for conducting additional experiments and addressing concerns. However, my primary concern regarding the technical novelty remains unresolved, as the proposed techniques largely rely on existing CUDA mechanisms. Therefore, I maintain my original score of Weak Reject.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final Clarification

We sincerely thank the reviewers and Area Chair for their constructive feedback. We believe our additional experiments, technical clarifications, and manuscript revisions have substantially addressed the major actionable concerns.


1. Scientific Novelty and Key Evidence

BOLT-KV's novelty lies in discovering and quantitatively characterizing interference mechanisms in confidential LLM serving, and deriving an effective systems design—not in inventing new CUDA primitives.

Three findings support this contribution:

  • Mechanistic discovery: We identify two distinct interference modes—legacy-stream-dependent and same-direction occupation interference—and demonstrate that foreground delay tracks the remaining duration of active movement (slopes ≈ 1.0). Stream isolation can expose an existing downstream readback bottleneck rather than eliminate interference.

  • Mechanism-driven design: These findings motivate controlling submission granularity and backend-specific issuance boundaries beyond CUDA stream configuration. SGLang already uses explicit streams, yet BOLT-KV improves throughput retention from 78.5% to 99.7% without changing its stream configuration, reducing p99 ITL from 101.4 to 8.0 ms with essentially unchanged completed movement work (Table 4).

  • Experimental validation: Factorial experiments establish the complementarity of stream isolation and bounded submissions (interaction ≈ +0.083, Table 3). We also demonstrate a 14.5% paired throughput improvement in vLLM's native KV-store backend. The original submission includes a second language model, while rebuttal experiments extend the evidence to NVIDIA B300.

Together, these results establish a scientific contribution spanning problem discovery, quantitative mechanistic understanding, derived systems design, and experimental validation.


2. Reviewer Assessments and Rebuttal Outcomes

Reviewer chs4 — Overall: 1 → 1 | Confidence: 4 → 3

The initial review recognized our experimental rigor and convincing mechanism-level characterization. We addressed technical concerns through additional controls, ablations, and B300 experiments. Although the reviewer raised presentation concerns, no specific passages or examples were identified despite our request for clarification. Nevertheless, we revised the manuscript to clarify the logical progression from discovery to design and validation.

Reviewer PDgi — Overall: 2 → 2 | Soundness: 3/3 | Presentation: 3/3

The reviewer assigned the highest ratings for soundness and presentation. The remaining disagreement primarily concerns scientific novelty. We clarified why BOLT-KV's mechanistic discoveries and derived execution strategy go beyond existing CUDA stream configurations, supported by the SGLang results and factorial experiments.

Reviewer bwMo — Overall: 3 → 3 | No updated assessment posted

We appreciate the reviewer's positive assessment. Additional B300 experiments, submission-size analyses, and clarification of background movement costs further strengthen the evidence and address concerns about generality and evaluation scope.


3. Rebuttal Improvements and Scientific Significance

During rebuttal, we added driver-matched H100 comparisons, mover-stream ablations, foreground-aware gating baselines, background movement cost measurements, and NVIDIA B300 evaluations.

We also clarified the manuscript's scientific contributions and logical flow, with initial revisions highlighted in blue and subsequent revisions in red.

Importantly, the ICLR 2027 Reviewer Guide recognizes identifying new problems and producing impactful empirical insights as valuable scientific contributions. BOLT-KV advances both by revealing previously underexplored interference mechanisms and deriving experimentally validated design insights.

We believe the reviewers' major actionable concerns have been substantially addressed. Remaining limitations, including system-wide goodput under GPU-memory pressure and causal attribution of CC-associated overheads, are explicitly documented.


4. Closing Remarks

We believe that BOLT-KV makes a meaningful scientific contribution by uncovering and quantitatively explaining interference mechanisms in confidential LLM serving, and translating these findings into an experimentally validated systems design.

We hope the Area Chair will consider these contributions, together with the strengthened experimental evidence and our substantive responses to the reviewers' concerns, in the final assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.