acceptodds
Under review as a conference paper at ICLR 2027

Queueing-Theoretic GPU Provisioning for Disaggregated Multimodal LLM Serving

Abstract

Disaggregated multimodal LLM serving separates modality encoding from prefill and decode, but its throughput depends on how GPUs are divided between the two stages. We develop a queueing-theoretic framework for choosing this allocation under a fixed GPU budget. The main modeling challenge is estimating encoder service demand: CPU and GPU work overlap across requests, so neither GPU execution time nor isolated request latency determines sustainable throughput. We decompose the encoder path into resource demands, fit their dependence on visual token count, and identify the bottleneck that limits request completions. A theoretical two-stage tandem queueing model then yields a closed-form allocation rule and explicit rate intervals within which each integer allocation remains optimal. The rule supports initial provisioning and can provide allocation targets for runtime adaptation. We study the encoder pipeline on an RTX 4090 testbed and evaluate provisioning in vLLM on eight H800 GPUs. Using a single measured pair of stage rates, the model predicts capacity within 6% across five tested allocations and selects the best measured split. Compared with an equal split, the computed allocation raises saturation throughput from 59.9 to 86.0 requests per second, a 43.6% improvement without additional GPUs.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
GPU Provisioning, Disaggregated LLM Serving, Multimodal LLM Serving, Queueing Theory

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_yp59OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how to divide a fixed GPU budget between modality encoding and combined prefill/decode. It models serving capacity as the minimum of the two aggregate stage rates and derives an integer allocation rule and intervals over which a split remains optimal. A component-level profiling method estimates encoder resource demand while accounting for overlapping CPU and GPU work. Experiments evaluate the calibration-to-allocation procedure on four RTX 4090 D GPUs and, separately, use directly measured stage rates on eight H800 GPUs to select a split with 43.6% higher saturation throughput than an equal split.

Critical strengths
  • The paper measures the complete encoder path rather than only GPU forward time. Accounting for overlapping CPU and GPU work is relevant to provisioning, and the real-GPU measurements provide useful evidence about this distinction.
Critical weaknesses
  • The contribution is limited, and the positioning against prior serving work is incomplete. Once the stage rates are known, the allocation rule is basic capacity balancing; the TTFT extension assumes suitable tail bounds rather than deriving them for the deployed system. The paper also omits relevant multimodal serving work, including SpaceServe [1], HydraInfer [2], ElasticMM [3], Cornfigurator [4] and Resonator [5]. These systems address encoder/decoder resource sharing, stage allocation, elastic parallelism, or deployment planning. Even though their settings differ, they are directly relevant to deciding how multimodal stages should use a GPU budget. The paper needs to explain what its profiling and allocation procedure adds to this literature and when its fixed E:PD pool model is an appropriate choice.

  • The experiments do not establish the value of the proposed profiling procedure. The 43.6% headline improvement on H800 uses directly measured stage rates and compares against an equal split. A simple capacity-balancing baseline would choose the same allocation. The complete calibration procedure is evaluated only on a fixed-token synthetic workload on the smaller platform, where GPU-only timing also selects the same 2:2 split. Moreover, the ModServe comparison measures GPU-forward predictions against full encoder wall time, so much of the error gap comes from predicting different quantities. These results do not show that the proposed calibration makes better provisioning decisions than simpler, properly calibrated alternatives on realistic changing workloads.

[1] SpaceServe: Spatial Multiplexing of Complementary Encoders and Decoders for Multimodal LLMs. NeurIPS 2025. https://proceedings.neurips.cc/paper_files/paper/2025/hash/720e7ebc49c84252abc0754dddf80976-Abstract-Conference.html

[2] HydraInfer: Hybrid Disaggregated Scheduling for Multimodal Large Language Model Serving, 2025. https://arxiv.org/abs/2505.12658

[3] ElasticMM: Efficient Multimodal LLMs Serving with Elastic Multimodal Parallelism. NeurIPS 2025. https://arxiv.org/abs/2507.10069

[4] Cornfigurator: Automated Planning for Any-to-Any Multimodal Model Serving, v3, April 2026. https://arxiv.org/abs/2512.14098v3

[5] Symbiotic MLLM Serving: Dynamically Balancing Parallelism Across GPUs and Resources Within GPUs. ISCA 2026. https://doi.org/10.1109/ISCA66397.2026.00173

Questions for authors
  • What does the proposed provisioning procedure add beyond existing multimodal resource-sharing and deployment-planning approaches [1–5]? With the same profiling budget, how does it compare with direct sustainable-throughput measurements followed by capacity balancing, or a predictor calibrated to the same resource-demand target?
  • Can the full calibration procedure select allocations on held-out heterogeneous workloads without directly measuring their stage rates? What additional usable latency prediction does the TTFT analysis provide for the evaluated system?
Reviewer_qzePOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a queueing-theoretic framework for GPU allocation in disaggregated multimodal LLM serving. By modeling encoder and prefill/decode service demands, it derives an optimal GPU allocation strategy under a fixed budget. Experiments demonstrate accurate throughput predictions and a 43.6% throughput improvement over equal GPU allocation.

Critical strengths
  1. Principled resource allocation: The queueing-theoretic formulation provides an interpretable approach to GPU provisioning, accounting for CPU–GPU overlap and stage-specific service demands.
  2. Practical efficiency improvements: The framework demonstrates accurate throughput predictions and substantial gains without additional GPUs, highlighting its potential for practical deployment.
Critical weaknesses
  1. Limited baseline comparisons: The reported 43.6% throughput improvement is primarily measured against equal GPU allocation, which is a relatively weak baseline. Comparisons with stronger profiling-based or adaptive provisioning strategies would better establish the practical advantages of the proposed approach.
  2. Limited scalability and workload generalization: Experiments are restricted to single-node configurations with at most 8 GPUs and predominantly image-based workloads. The constant per-GPU service rate assumption may not hold under larger-scale deployments or heterogeneous, dynamically changing workloads, limiting the generality of the conclusions.
Questions for authors
  1. Baseline Comparison: How does the proposed method compare with stronger allocation baselines, such as profiling-based optimization or existing adaptive provisioning strategies, beyond equal GPU splitting?
  2. Scalability: How well does the framework generalize to larger multi-node deployments? In particular, how would communication overhead, resource contention, and nonlinear GPU scaling affect the allocation decisions?
  3. Mixed Workloads: How robust is the allocation strategy under changing image-size distributions, multi-image requests, and mixed text-only/multimodal traffic? Specifically, how does the model account for text-only requests that bypass the encoding stage?
Reviewer_PVgQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies how to allocate a fixed number of GPUs between the encoder and prefill/decode stages in disaggregated multimodal LLM serving. The authors argue that encoder GPU execution time or single-request latency alone cannot accurately capture the encoder's throughput, since CPU preprocessing, GPU execution, and other operations can overlap across requests. To address this, they break down the encoder pipeline into different resource components and estimate its service demand based on the bottleneck resource. They then use a two-stage queueing model to derive an allocation rule that balances the capacity of the encoder and prefill/decode stages. The paper also analyzes the robustness of the allocation decision and discusses sufficient conditions for meeting TTFT requirements. The authors evaluate their approach on four RTX 4090D GPUs and eight H800 GPUs. On the H800 setup, using measured stage rates, their model selects a 2:6 encoder-to-PD split, achieving a 43.6% improvement in saturation throughput and a 50% increase in the highest tested SLO-compliant offered rate compared with an equal 4:4 allocation.

Critical strengths
  1. The paper addresses a practical and important problem in multimodal LLM serving. I think one useful insight is that GPU execution time or single-request latency does not necessarily reflect the actual serving capacity of the encoder. The authors carefully break down the encoder pipeline and account for overlapping CPU and GPU operations. This makes the resource demand estimation more meaningful for GPU provisioning, rather than simply relying on isolated latency measurements.

  2. The experimental results are fairly convincing. The authors evaluate their approach on both RTX 4090D and H800 GPUs, covering different hardware and model configurations. On eight H800 GPUs, the model predicts throughput within 6% across five allocations and correctly identifies the best allocation, achieving a 43.6% throughput improvement over equal allocation. Although the improvement is measured against a relatively simple baseline, I think the results still demonstrate the practical value of the approach. The experiments also provide useful evidence that a relatively simple capacity model can guide GPU provisioning without exhaustively testing every configuration.

Critical weaknesses

My biggest concern is the novelty of the queueing-theoretic part (§3). Once the encoder and PD service rates are known, the allocation problem is basically just balancing the throughput of two pipeline stages. The optimal split comes from maximizing the minimum of their capacities, which is a pretty straightforward result. The integer rounding and decision stability intervals also follow from this. I don't see much new queueing theory here. The math looks fine, but I feel the paper spends a lot of effort formalizing something that is already quite intuitive.

Another major issue is that the strongest experiment doesn't actually evaluate the complete proposed method. The paper spends most of §4 developing a component-wise model to estimate encoder service demand. But in the main H800 experiment (§5.3–5.4), the authors directly measure the encoder and PD rates, 57 and 14.3 requests/s/GPU, and then plug them into the allocation formula. So the headline 43.6% throughput improvement mainly shows that balancing two measured stage capacities works. It doesn't really validate the profiling method developed earlier. The actual end-to-end profiling-to-allocation experiment is only done on four RTX 4090D GPUs with a fixed visual token count, and even there the predicted capacity is 11.6% higher than what they measure. I find this gap between the proposed method and the main experiment pretty important.

I'm also not fully convinced that the detailed encoder profiling is necessary. In Appendix D, using GPU execution time alone actually gives the same correct 2:2 allocation on the RTX 4090D setup. The capacity prediction error is 14.0%, versus 11.6% for the proposed model. So after all the decomposition and regression, the allocation decision is still the same. The authors do show that using full serial wall time gives a wrong allocation, which is a good observation. But showing that wall time is a bad proxy is not the same as showing that their complicated profiling method is needed. I would really like to see a case where simpler GPU-based profiling gives a wrong allocation and their method fixes it.

The baseline comparisons are also quite weak. The 43.6% improvement is over an equal 4:4 split, but the encoder is roughly four times faster than PD on the H800 setup. Why would we expect an equal split to perform well in this case? Even a very simple rate-based allocator should pick something close to 2:6. I think the paper needs to compare against that, or against existing workload-aware provisioning methods, rather than mainly showing that equal allocation is inefficient.

The ModServe comparison in §5.1 is also a bit strange. ModServe's profiler measures GPU encoder forward time, while the authors compare its prediction against full CPU-inclusive encoder wall time. These are not the same quantities. Of course a GPU-only measurement will underestimate the full request latency. The large MAPE difference (67.54% vs. 11.76%) therefore doesn't tell us much about whether the proposed method is better at estimating serving capacity. A comparison where both methods target the same quantity would be much more useful.

I also expected more from the resource bottleneck analysis in §4. The authors divide the encoder pipeline into six components, but then find that the worker–GPU cycle is the bottleneck across the entire tested range. So in the end, the encoder demand just becomes one quadratic function of visual token count. It would be much more interesting if they showed how bottlenecks change under different workloads, for example when CPU preprocessing, GPU computation, or data transfer becomes dominant. Right now I'm not sure what we gain from modeling all these components separately. And using only visual token count may be too simple, since image format, aspect ratio, number of images, and preprocessing costs can differ even for requests with similar token counts.

There is also a gap between the queueing model and how vLLM actually works. The model assumes fixed per-instance service rates, uniform routing, FIFO servers, and no additional transfer bottlenecks. But real serving has batching, scheduling, CPU contention, cache behavior, and other interactions. The paper itself shows that per-PD throughput changes with the number of PD instances on H800. On the RTX 4090D platform, the PD rate estimate is also off by 21%. The authors acknowledge that the resource-load bound is not necessarily achievable capacity, but that makes me wonder how reliable the allocation formula would be when the workload or resource contention changes. The theory is correct under its assumptions, but these assumptions are doing quite a lot of work.

I'm less convinced by the SLO analysis (§3.4 and Appendix A.3). The TTFT guarantee assumes uniform tail bounds across the different allocations and arrival processes. But these bounds are exactly the difficult part in a real serving system, and the paper doesn't actually establish them for the H800 experiments. So the theorem feels quite conditional. The experimental SLO results are based on 120-second load waves rather than longer sustained runs. At the reported 85.8 requests/s operating point, the system completes 83.8 requests/s and 1.9% of requests are incomplete or failed at the end of the wave. The authors count these as violations, which is fair, but I would still like to know whether this allocation can sustain the same SLO over a longer period. Also, the 50% SLO improvement is based on the highest tested compliant rates, not the actual SLO capacity boundaries.

The workloads are somewhat limited too. The 4090D experiment uses a synthetic fixed-token workload, while the H800 experiment uses a repeated MMMU trace where around 95% of requests only have one image. Both use fixed 128-token outputs and Poisson arrivals, and encoder-cache misses are forced. I understand why the authors make these choices for controlled experiments, but this is a multimodal serving paper. What happens with video inputs, multiple images, variable output lengths, cache hits, or bursty traffic? These could change the encoder-to-PD demand ratio quite a lot. The paper claims the allocation can be recomputed when workload distributions change, but doesn't really test that claim. The H800 evaluation also only covers five of the seven possible nontrivial allocations, and the 1:7 configuration is not pushed into deep overload, so its saturation capacity is not fully measured. More repeats and error bars for the main throughput results would help as well. The second encoder experiment in Appendix G only tests serial latency prediction, rather than whether the provisioning decision is still correct.

Finally, the practical deployment story is not fully demonstrated. Algorithm 1 can recompute an allocation target from request logs, but there is no real runtime adaptation experiment. We don't know how often the allocation should change, what the switching overhead is, or whether the model remains accurate when traffic shifts. Appendix F argues that calibration is much cheaper than configuration search, but this is mostly an estimated cost comparison against a full allocation sweep. A smarter search method might need much less work. So I think the paper has a reasonable argument for reducing profiling cost, but it hasn't really shown the advantage in a changing serving environment.

Overall, I think the paper has some useful systems measurements, and distinguishing encoder wall time from actual resource demand is a good point. The experiments also show that a bad GPU allocation can waste a lot of throughput. But I keep coming back to the same question: what does this method do that a much simpler stage-throughput profiler and capacity-balancing rule cannot do? The paper gives quite a lot of theory and modeling, but the experiments don't clearly answer that question. This is my main reservation about the contribution.

Questions for authors

Why is the proposed profiling method not evaluated in the main H800 experiment? In §5.3–5.4, the encoder and PD rates are directly measured and then plugged into the allocation formula. So the 43.6% throughput improvement seems to mainly validate the capacity balancing rule, rather than the profiling method in §4. Is there any result showing that the complete profiling-to-allocation pipeline can predict the correct split on H800 without directly measuring the stage rates?

What is the actual advantage over a simpler throughput-based allocator? Once we know the encoder and PD rates, balancing their capacities is pretty straightforward. Why not just measure the saturated throughput of each stage and allocate GPUs accordingly? How much additional accuracy or lower profiling cost does the proposed method provide compared with this simple baseline? I think this comparison is quite important for understanding the novelty.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.