acceptodds
Under review as a conference paper at ICLR 2027

LSV: EFFICIENT REVERSIBLE GPU MULTIPLEXING FOR LLM SERVING

Abstract

Intra-GPU multiplexing lets Preffll and Decode share one LLM replica while using different GPU resources. The useful split changes with the request mix. Existing controllers can select a new split quickly. But work already submitted under the old split can delay the ffrst useful work under the new split by milliseconds. We call this delay reconffguration debt. We present Logical Stream Virtualization (LSV), an execution layer with stable logical lanes for Preffll and Decode. LSV tracks issued and completed work with resource epochs. It limits future old-epoch commitments and changes both lanes at one safe boundary. We implement LSV in SGLang and evaluate it on an eight-H100 NVLink server across three model scales and two model families. A controlled test shows that host handoff stays near 9.5 µs, while physical activation reaches 16.3 ms after sixteen old-epoch replays. Across the three models, LSV preserves more latency headroom, compliant goodput, and joint SLO attainment as load rises. The results show that adaptive multiplexing needs a runtime contract for when a resource decision becomes physical, not only a policy for which split to choose.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
GPU Multiplexing, LLM serve, LLM infra

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_93wMOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents a change in how resources are allocated between prefill and decode on a single GPU box for severing systems. The work identifies an additional physical switching cost when switching between prefill/decode profiles that results from already scheduled kernels being on the wrong configuration. They propose a logical lease system that seems to shorten the physical profile transition time and the results show small but consistent gains in the approach when switching is critical.

Critical strengths
  1. The paper shows promising results on SG-Lang for workloads in which this switching cost is likely to appear.

  2. The authors clearly put significant effort in their implementation with "over 3,000 lines of runtime code" in SG-Lang.

Critical weaknesses
  1. The paper was exceptionally difficult to follow. It has the writing style of Opus 5 with short punchy sentences that don't provide great clarity. Example: "LSV does not require a particular predictor. It accepts a target resource request from the policy." I work in this space and still don't fully understand the technique.

  2. This is very much a systems paper and would be much better reviewed at a systems venue (e.g., OSDI or SOSP) where there is more room in the paper to properly explain the system design.

Questions for authors
  1. Maybe try to re-explain the key-insight that underlies the technique.

  2. This is built on the relatively new cuda green-context and I would like to know if that does most of the "heavy lifting".

Reviewer_wqhKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies resource reconfiguration delays when Prefill and Decode share a GPU in LLM serving. After a controller selects a new SM allocation, previously submitted work can still delay its physical activation, which the authors call reconfiguration debt. Logical Stream Virtualization binds resource configurations, CUDA Graphs, and progress state to stable logical lanes. It limits the estimated outstanding Prefill work and transitions both phases at a consistent resource-epoch boundary to reduce the delay between allocation selection and execution. The prototype is integrated into SGLang and evaluated on an 8×H100 NVLink server with three models spanning the Qwen and Llama families and dense/MoE architectures. The evaluation includes steady load, bursts, component ablations, and controller sensitivity.

Critical strengths

S1. The distinction between allocation selection and physical activation is useful.

The paper distinguishes completing a host-side configuration change from actually executing work under that configuration, and the queued-replay experiment shows that these times can differ by orders of magnitude. Appendix A also reports measured transition-latency distributions, extending the observation beyond a qualitative discussion of asynchronous submission. Accounting for outstanding work helps explain why an appropriate allocation decision may still fail to improve latency promptly. This is the paper's most useful systems perspective.

S2. The design connects safe transitions with service-level behavior.

LSV combines profile-dependent state, consistent epoch transitions, and submission limits while explicitly protecting Prefill progress. The evaluation covers multiple model scales and dense/MoE architectures. Its ablations examine the debt budget, fast revocation, and starvation protection, as well as replacing the policy on the same executor. These design choices and experiments provide initial support for practical usefulness and make the work more convincing than a demonstration of host-side switching speed alone.

Critical weaknesses

W1. The evaluation protocol does not yet make the main performance claims verifiable.

Although the experiments report gains across models, Section 5.1 does not specify the TTFT/TBT SLO thresholds, compliance rules, or measurement duration. In Figure 6, the Azure request-rate axis spans approximately 0.4–0.9 req/s, while request throughput reaches approximately 2–5 req/s, without an explanation of scaling or different measurement windows. This may reflect a difference in definitions, but the curves cannot currently be interpreted using the usual end-to-end arrival and completion rates. Since the main conclusions depend on these metrics, clarifying their definitions and providing the corresponding request counts and timestamps is necessary to reassess the results. This is my main reason for leaning toward weak rejection.

W2. The benefit beyond simpler submission control remains unclear.

Figure 7 provides meaningful component ablations, but does not establish how much estimated-time debt budgeting adds over a simple limit on outstanding Prefill tickets. PDMux already splits Prefill into blocks and controls how many to launch, while Bullet uses layer-wise progress monitoring and dynamic resource allocation. PDMux, Bullet. Comparing the two submission controls under the same policy, work quantum, and safe transition boundary would better establish the contribution of the added execution mechanism. Appendix B reports parameter changes relative to defaults, but omits the absolute budgets and estimator update rules, further limiting the ability to reproduce this comparison.

Questions for authors

Answers to the following questions would be most likely to affect my score.

  1. How are request rate and request throughput computed in Figure 6? Why does the Azure curve reach approximately 5 req/s when the horizontal axis remains below 1 req/s? Are different measurement windows, an initial backlog, or trace scaling involved?
  2. What TTFT/TBT thresholds and joint-attainment rules are used for each model and workload? Could the authors provide the request counts, measurement windows, number of repetitions, and definitions of the error bars in Figures 7 and 9?
  3. Under the same policy, Prefill quantum, and safe epoch boundary, how do a fixed outstanding-ticket limit and an estimated-time budget compare in activation tail latency and goodput?
  4. Which models and workloads supply the transition samples in Appendix A? Could the authors break down the Prefill tail, Decode tail, and first-dispatch delay, and report the overhead of maintaining multiple graphs/profiles and tracking per-ticket progress?
1 comment

I thank the authors for the response. While the rebuttal provides useful additional evidence, the proposed mechanisms remain relatively straightforward extensions of existing execution-control techniques, leaving my concern about the significance of the techniques unresolved. I therefore keep my score.

Reviewer_Gz2LOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the delay between selecting a new intra-GPU prefill/decode resource allocation and executing useful work under that allocation. LSV manages this reconfiguration debt through logical execution lanes, profile-specific executor state, submission budgeting, and coordinated resource-epoch transitions. An online lease controller adjusts the allocation while protecting decode latency and prefill progress. The implementation extends SGLang and is evaluated on an eight-H100 server with three models. The paper reports improvements in latency, compliant goodput, and joint SLO attainment under steady and bursty workloads.

Critical strengths
  1. The paper identifies and measures a meaningful runtime problem. The distinction between host-side handoff and physical activation is important for adaptive GPU multiplexing. Figure 2 shows that a roughly 9.5 μs host handoff can coexist with a 16.3 ms activation delay after sixteen queued replays. Appendix A provides further transition measurements. These results give concrete evidence that selecting a resource split quickly is insufficient without controlling outstanding work.

  2. The design is coherent and supported by useful ablations. Consistently publishing profile-dependent state and limiting old-profile submissions address complementary aspects of reconfiguration. Figure 7 separates several mechanism and policy contributions, supporting the benefit of debt budgeting and the controller protections. The evaluation across three models provides evidence beyond the controlled transition test. The paper also appropriately distinguishes its empirical responsiveness from a hard real-time guarantee.

Critical weaknesses
  1. The evaluation protocol and controller details need clearer specification. Figure 6 reports Azure arrival rates of approximately 0.4–0.9 req/s alongside request throughput of approximately 2–5 req/s, without explaining whether their aggregation scopes or measurement windows differ. Section 5.1 also leaves the absolute SLO thresholds and compliance accounting unclear. The budget condition is defined, but the duration-estimation update, quantum adaptation, and absolute controller defaults are insufficiently specified. These details matter for interpreting and reproducing the main serving results.

  2. The incremental benefit and cost of the execution abstraction are not fully isolated. The ablations establish that budgeting helps, but do not directly compare it with a simpler submission cap and safe transition mechanism under the same allocation policy. The cost of preparing and retaining profile-specific contexts and CUDA Graph states is also unclear. In addition, the conditional bound in Appendix C retains the Decode-tail term, whose contribution is not separately characterized. These omissions limit the evidence for the abstraction's advantage and practical operating range, although they do not invalidate the demonstrated improvements.

Questions for authors
  1. Clarification: Please reconcile the arrival-rate and throughput units, aggregation scopes, and windows in Figure 6. What are the absolute SLO thresholds, how is token compliance defined, and how are unfinished requests counted?

  2. Implementation clarification: Please specify the estimator update, quantum-adaptation rule, and default budget and lease parameters. Are profile-specific contexts and graph states prepared in advance or on demand, and what preparation time and additional memory do they require?

  3. Evidence affecting contribution and scope: Under the same allocation policy and safe transition semantics, how much benefit remains over a tuned fixed submission cap? A focused comparison would help. From the existing transition measurements, how often does the Decode tail dominate activation delay? These results would clarify which part of the improvement depends on the proposed execution design.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.