- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies interference between background KV-cache movement and latency-sensitive foreground serving under GPU Confidential Computing (CC). Using framework-independent probes, the authors identify two CC-associated interference modes: (1) legacy-stream-dependent interference, where CUDA's legacy default stream implicitly synchronizes with blocking streams and delays otherwise unrelated foreground operations, and (2) same-direction occupation interference, where bulk transfers delay smaller transfers in the same direction even on separate streams. They show foreground delay approximately equals the residual time until the active transfer completes (slope ≈ 1.001, R² ≈ 1), and that stream isolation (PBSI) merely relocates the bottleneck to mandatory output readback rather than removing it. Based on this characterization, the paper proposes BOLT-KV, which (a) removes avoidable legacy-stream coupling via explicit non-blocking streams, and (b) divides KV movement into byte-bounded "bounded submissions," submitting the next chunk only after the previous one reaches a backend-specific issuance boundary — creating opportunities for foreground work without explicit prioritization or synchronization primitives. Implemented in both SGLang and vLLM, BOLT-KV raises SGLang throughput from 78.5% to 99.7% of the no-movement baseline (ITL p99: 101 → 8 ms) and vLLM KV-store throughput from 80.1% to 88.5% (ITL p99: 74 → 41 ms). During H2D restoration, BOLT-KV Restore more than doubles foreground progress versus coarse restore, exposing a trade-off between recovery completion (where Reprefill wins) and foreground responsiveness.
Critical strengths
- Novel problem discovery with precise characterization: The paper is the first to identify and quantify two distinct CC-associated interference modes. The measurement that CC drops serving throughput to R = 0.701 with foreground input-H2D p99 of 122.5 ms (vs. 0.29 ms on NonCC), while NonCC under 4.5× pressure remains at R = 0.981, convincingly demonstrates that the interference does not arise on non-CC systems.
- Strong empirical model : The residual-occupation model — foreground delay ≈ remaining duration of the active transfer (α = 1.001, R² ≈ 1) — is simple, predictive, and validated both in framework-independent probes (Tables 7–8) and at serving level (Table 2). The falsifiable predictions P1–P3 with pre-registered factorial validation (Section 5.1) reflect excellent experimental discipline.
- Elegant, low-overhead design: Bounded Submissions + issuance boundaries require no foreground-priority gate, no added synchronization primitives, and no deliberate idling, yet the factorial experiment (Table 3) cleanly demonstrates positive interaction (+0.08) between PBSI and bounded submissions — showing they address complementary interference modes.
- Honest limitation reporting and dual-backend generality: The paper retains rare ~200 ms excursions rather than trimming them (P3 explicitly acknowledges bounded submissions "reduce tails but do not bound them"), and reproduces headline effects on a second model (Llama-3.1-Nemotron-Nano-8B, Table 17) with pre-registered predictions.
- Addresses a real pain point on the serving critical path: By pinpointing where the interference enters the serving critical path (legacy-stream coupling vs. same-direction readback) and how to treat it, the paper fills a genuine gap — prior CC studies exposed serialized transfers but did not clarify the interaction on the serving path or the treatment.
Critical weaknesses
- Calibration audit contradicts the headline configuration: Appendix B.3 (Table 11) shows 2 MiB meets the latency criterion and is selected by the pre-defined fallback utility function U(q), while 4 MiB fails both primary criteria — yet the headline experiments retain 4 MiB justified only by "pre-registration." This leaves a "picked a sub-optimal point" suspicion that a 2 MiB end-to-end result is needed to resolve.
- CC vs. NonCC contrast is confounded, so the core motivation's causality is weak: The paper itself admits the two hosts differ in drivers (595.71.05 vs. 610.43.02) and VM configurations (Footnote 2; A.2, lines 700–701). Thus "CC causes ~30% throughput drop" can only be claimed as "CC-associated," not causal. Without a same-driver same-VM CC on/off control, the foundational motivation is on unstable ground, and the mechanism cannot be located inside NVIDIA's closed CC implementation.
- SGLang controlled replay bypasses the real scheduler: The replay mover directly accesses live KV buffers in the scheduler process and CUDA context (lines 1024–1025), sidestepping the real scheduler's queuing effects; and SGLang's end-to-end claims rest on this controlled replay rather than the runtime's native KV lifecycle.
- vLLM restoration experiment does not measure real recovery: Both restore arms write identical prepared KV bytes to a validated GPU scratch destination rather than resuming actual attention (lines 429–431, C.5). The experiment therefore measures only the transfer component of restoration, not the true restoration-into-serving path — limiting the external validity of the H2D restoration conclusions.
- Scope and robustness limits: Evaluation covers a single H100 NVL CC platform with Blackwell untested; the deterministic guarantee is explicitly absent (P3: rare long waits persist, lateness reaches p99 2.26 s at highest offered load); and vLLM's improvement is markedly smaller than SGLang's (88.5% vs. 99.7% of baseline), suggesting backend-dependent benefit magnitude that is underexplored.
Questions for authors
- Calibration consistency: Given that Table 11's own audit criteria select 2 MiB (and 4 MiB fails both primary criteria), can you provide the end-to-end serving results at q_max = 2 MiB? If 2 MiB performs equal or better, why retain 4 MiB as the headline operating point?
- Causality of the CC contrast: Can you run a CC-on/CC-off comparison on the same host with the same driver version and VM configuration (i.e., toggle CC mode only)? Without this, how do you justify the claim that the ~30% degradation is attributable to confidential computing itself rather than the confounding driver/VM differences?
- End-to-end validity of the replay/scratch methodology: (a) Could you integrate the bounded-submission mover into SGLang's actual KV lifecycle (not controlled replay) and report results? (b) For H2D restoration, can you evaluate the real path where restored KV is re-entered into attention, rather than writing to a validated scratch destination?
2 comments
Additional Evaluation on NVIDIA Blackwell
During the rebuttal period, we gained access to an NVIDIA B300 system with confidential computing enabled, which had not been available at the time of submission. We have incorporated the additional results into Appendix D (Tables 21–24). All revisions in the updated manuscript are highlighted in blue.
1. Calibration consistency (W1, Q1). The reviewer bwMo asked for "end-to-end serving results at q_max = 2 MiB." Our H100 SGLang results show comparable throughput at 2 and 4 MiB (Table 13). We additionally evaluated submission-size sensitivity on B300 CC vLLM (Table 24):
| Metric | 2 MiB | 4 MiB | 8 MiB |
|---|---|---|---|
| Throughput (tok/s) | 491.9 | 517.5 | 497.0 |
| ITL p99 (ms) | 16.31 | 16.86 | 21.20 |
| Background lateness p99 (s) | 3.66 | 1.75 | 1.05 |
These results illustrate the foreground–background trade-off. We retain the previously fixed 4 MiB configuration without claiming universal optimality.
2. CC versus NonCC confounding (W2, Q2). We additionally compared B300 CC and NonCC under the same offered movement trace (Table 21). Throughput retention was 0.621 under CC versus 0.988 under NonCC. While VM differences still prevent strict causal attribution, this independently reproduces the CC-associated slowdown observed on H100.
3. Evaluation scope (W5). Both interference mechanisms persist on Blackwell (Table 22). BOLT-KV also improves vLLM throughput by 13.1% over Stock and substantially reduces extreme ITL stalls (Table 23).
These additional experiments strengthen the evidence for BOLT-KV across Hopper and Blackwell, while the limitations of controlled replay and scratch-based restoration (W3–W4) remain explicitly acknowledged.
Response to Reviewer bwMo
We thank the reviewer for recognizing the novel problem characterization, strong empirical validation, and practical design. We have conducted additional experiments and clarified the evaluation scope to address the concerns.
W1 & Q1. Calibration consistency and submission size
We added an end-to-end submission-size sweep (Table 13). At 2 and 4 MiB, throughput retention is nearly identical (R = 0.999 vs. 0.998), with ITL p99 of 7.58 vs. 8.00 ms. However, under higher movement load, 2 MiB increases background lateness p99 to 3.59 s, compared with 2.25 s at 4 MiB.
Thus, 4 MiB represents a reasonable foreground–background trade-off, not a universally optimal setting. We retain the originally fixed configuration while transparently reporting that the subsequent calibration audit selects 2 MiB.
Manuscript revisions: We added the end-to-end sensitivity analysis (§5.2; Appendix B.3, Table 13) to clarify the trade-off and resolve the apparent calibration inconsistency.
W2 & Q2. CC versus NonCC attribution
We repeated the comparison with matching NVIDIA drivers (595.71.05). Under the same D2H trace, throughput retention is 0.689 with CC versus 1.001 without CC, and input-H2D p99 is 123.4 versus 0.234 ms.
Importantly, our central findings do not depend on this cross-environment comparison: the interference modes, residual-occupation relationship, and BOLT-KV improvements are independently validated within the same CC environment. We retain the term CC-associated because VM differences remain.
Manuscript revisions: We replaced the original results with driver-matched measurements (§3.1; Appendix A.2, Table 7) and clarified the remaining confounders in Limitations.
W3–W4 & Q3. End-to-end validity of replay and restoration
We strengthened the native vLLM KV-store evaluation (Table 4). BOLT-KV improves paired absolute client throughput by 14.5% over true Stock (Appendix C.3), with ITL p99 decreasing from 81.0 to 39.9 ms (Table 4).
We also measured the native store lifecycle with 16 decoding requests, confirming foreground latency improvements while quantifying delayed store completion and GPU-block reuse.
SGLang controlled replay remains a mechanism-isolation experiment. Likewise, H2D restoration uses validated scratch memory to isolate transfer interference; it does not evaluate resumed attention. We explicitly distinguish these results from full native lifecycle and recovery measurements.
Manuscript revisions: We clarified controlled replay versus native serving (§5.2; Appendices C.2–C.3), corrected the vLLM baseline (Table 4), and specified the scope of H2D restoration (§5.3; Appendix C.5).
W5. Scope and robustness
We acknowledge that Blackwell remains untested and deterministic latency guarantees are absent. Additional submission-size and movement-pressure experiments characterize the foreground–background trade-off. The different SGLang and vLLM workloads also preclude a direct comparison of their improvement magnitudes.
Manuscript revisions: We expanded the sensitivity analysis (Appendix B.3), clarified the backend-specific evaluation conditions (§5.2), and explicitly stated the remaining limitations.