- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies PD-disaggregation LLM serving from queueing theory perspective. It can be seen as some extension to the Nie et al (ICML 2026), trying to extend from PD mixing LLM serving to PD-disaggregated LLM serving. It models this serving process as a tandem queue, where prefill is the first stage and decode is the second stage. The prefill is modeled as linear, and the decode is described carefully: the model contains continuous batching, concurrency limit, KV cache size limit, etc. The major contribution of this paper is to provide theoretical guarantees on given some P,D servers, what is the waiting time, resource redundancy, etc. Finally, the authors run a numerical simulation to verify the theory.
Critical strengths
I like the decode-step modeling. It differentiates the concurrency limit and the KV size limit. It also contains the linear increasing KV usage pattern. Therefore, I think the model is really trying to provide some insights specific on LLM serving, not something can be replaced by general cloud computing.
Critical weaknesses
However, there are two big weaknesses from my mind:
-
the model and theoretical insights still have huge gap to real LLM serving. I have two comments on the model: (i) the author models the prefill process as linear to the prompt length. However, based on my understanding and my engineering experience, the prefill process is usually quadratic due to attention computation. (ii) the authors completely ignore the communication cost between prefill and decode, which is a big cost from my own experience. Then, on theory, I think there are many rigorous theorems, but I wonder what are the insights behind the theory. For example, I read the cited Nie. et al ICML 2026 paper, I think the authors try to extends Nie et al's PD mixing to PD disaggregation. However, in Nie. et al 2026, they provide exact theorems on stable condition and service rate, then engineers can compute service rate according to the math formula, and decide the provisioning plan (how many GPUs are required for a given workload distribution). In this paper, although there are many theorems, from my own understanding on math, I cannot get the most important decision: what is the P/D ratio on provisioning. The theorems on how congested the prefill/decode given a provisioning plan is not that insightful since one can easily observe from the realization.
-
the numerical experiments do not have any hardware configuration, and therefore, I guess it is some model-based simulation. If yes, I think this completely cannot provide any insight. If the authors do not have GPU resources access, at least they should use Vidur simulator (which can run on your own CPU). A model-based simulation just verifies the theory on your specific model, but what if the model has huge gap with real? (I believe there are huge gaps between the model and real) Therefore, I strongly encourage the authors to run some experiments on either real GPUs or Vidur, and then, the theory can show its value. Specifically, for provisioning and queueing, once the formula can be verified on real GPU provisioning (e.g. Nie et al 2026 used 8*A100 to verify their theory), formulas can bring strong value and impact.
Questions for authors
- What is the experiment's setting? It is model-based, or Vidur simulator, or real experiments?
- Given on all of these theorems, for practitioners, if I care about how much P&D instances are needed given a workload distribution, how to derive this term?
8 comments
Response(2/8)
2.Part of W1: the communication cost between prefill and decode
Thank you for highlighting the impact of the communication cost between prefill and decode omitted from our original model. We first (a) clarify the rationale for this modeling choice and then (b) extend the analysis following your suggestion. We hope that these clarifications and additional results will strengthen your confidence in our work.
(a) Rationale for the original model. As stated in Section 3, the original model targets high-bandwidth deployments in which communication is not a primary bottleneck. This abstraction is supported by prior systems work. DistServe [1] explicitly treats cross-node KV-cache transfer overhead as negligible in its high-bandwidth placement analysis. Its experiments further show that, for OPT-175B on ShareGPT, KV-cache transfer accounts for less than 0.1% of aggregate request latency. Splitwise [2] similarly reports that its optimized transfer overhead is approximately 0.8% of end-to-end latency in the coding-trace experiment. Accordingly, our original model focuses on high-bandwidth settings to concentrate on the more complex inference mechanisms of the prefill and decode stages.
(b)Explicit modeling of KV-cache transfer. Following your suggestion, we agree that explicitly incorporating a KV-cache transfer queue is important under limited bandwidth. We have therefore added modeling and analysis of the transfer stage, together with experimental validation, as detailed below.
KV-cache transfer. Requests completing prefill enter a nonpreemptive FCFS single-server transfer queue. For a fixed model, KV volume is proportional to prompt length . We model transfer time as an affine function with a fixed overhead, rounded to base slots:
\qquad a_{\mathrm T}>0,\quad c_{\mathrm T}\ge0.$$ Here $\delta$ is the base-slot duration, and $S$ denotes the prefill service requirement in slots. The transfer population $K_{\mathrm T}$ includes waiting and in-service requests. With $\lambda=\mathbb E[A_t]/\delta$, we assume $\lambda\delta\mathbb E[S]<1$ and $\lambda\delta\mathbb E[S_{\mathrm T}]<1$ under the paper's infinite-buffer, no-blocking assumptions. **Transfer analysis.** We retain the temporal dependence of prefill departures and the dependence between each request's two service requirements through $\chi$. Immediately after a nonempty external arrival batch, let $x$ be the residual time to its last scheduled prefill completion and $z$ the corresponding transfer-completion lag. Elapsing time $g$ and appending a request with joint physical service times $(p,t)=(\delta S,\delta S_{\mathrm T})$ give $$\mathcal D_g(x,z)=((x-g)^+,(x+z-g)^+-(x-g)^+),$$ $$\mathcal J_{p,t}(x,z)=(x+p,(z-p)^++t).$$ These maps follow directly from FCFS completion-time recursions. Under the paper's i.i.d. external batch-arrival model, composing them and averaging over batch sizes, inter-batch gaps and joint service marks defines a two-dimensional transition kernel $\mathcal P_{\mathrm{PT}}$. We compute the stationary law from $\nu_{\mathrm T}=\nu_{\mathrm T}\mathcal P_{\mathrm{PT}}$. Weighting batches by their request counts and selecting a request uniformly within each chosen batch yields the typical-request Palm law. Let $d_i^{\mathrm P}$ and $d_i^{\mathrm T}$ denote completion times of the tagged request ($i=0$) and its successors. The occupation identity $$C_k=\Pr\{K_{\mathrm T}\ge k\} =\lambda\mathbb E^\star\!\left[ \left(\min\{d_k^{\mathrm P},d_0^{\mathrm T}\} -d_{k-1}^{\mathrm P}\right)^+ \right],\qquad k\ge1,$$ gives the stationary time-sampled population, where $\mathbb E^\star$ is expectation under that Palm law. The PMF follows as $\pi_0^{\mathrm T}=1-C_1$ and $\pi_k^{\mathrm T}=C_k-C_{k+1}$ for $k\ge1$. Numerical evaluation uses the stationary equation and occupation rewards. **Interface to decode.** Let $A_n^{\mathrm D}$ count transfer completions in a fixed analysis slot of duration $\Delta_{\mathrm D}$, which need not coincide with a dynamic decode iteration. Departure-window rewards determine $$q_a^{\mathrm D}=\Pr\{A_n^{\mathrm D}=a\},\qquad G_{\mathrm D}(z)=\sum_{a\ge0}q_a^{\mathrm D}z^a,\qquad \lambda_{\mathrm D}=\lambda\Delta_{\mathrm D}.$$ This marginal replaces the original prefill-departure input law in the decode stage. We have incorporated these additions into Sections 3 and 4.2 and Appendix C of the revised manuscript, including detailed derivations and proofs, and updated the experimental validation in Section 7. [1] Zhong et al. *DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving*. OSDI 2024. [2] Patel et al. *Splitwise: Efficient Generative LLM Inference Using Phase Splitting*. ISCA 2024.Response(3/8)
3.W2&Q1-(1)
Thank you for pointing out the limitations of our numerical experiments. We acknowledge that the experiments in the original manuscript were based on model-driven Monte Carlo simulations, which were intended to verify the correctness and accuracy of our derivations and approximation methods. We also agree that, as you suggested, our formulas could offer significant value and impact if validated through more realistic experiments. Consequently, we have conducted further, more realistic experiments using the Vidur simulator and have refined our model and theory based on these results to better reflect real-world conditions.
(1) Dynamic batch processing time. In the original version, following Nie et al., we approximated the duration of each batch iteration as a constant. Nie et al. primarily study system stability and service capacity. Our paper further investigates detailed stationary queue-length distributions and their tail behavior. Particularly under heavy load, the spare service capacity available to drain accumulated requests is small, so even slight deviations in processing time can be amplified through queueing accumulation, leading to substantial errors in distribution predictions. Therefore, a timing approximation that adequately captures long-run service capacity may not be sufficient to ensure accurate queue-length distribution predictions. Our Vidur experiments also indicate that this modeling assumption requires further refinement under the configurations and workloads considered.
Accordingly, we extend the duration of a single decode batch iteration to a linear function of the active batch size :
This model is directly supported by existing studies. Zhang et al. [1] explicitly model the duration of a single decode iteration using a linear relationship and provide execution-time measurements for Qwen3-4B on NVIDIA H200 and RTX PRO 6000 GPUs. Yang et al. [2] also observe, through experiments on an NVIDIA A100, that the time required to generate a single token increases approximately linearly with batch size. We additionally use Vidur to validate the suitability of this approximation under our experimental configuration (We have provided detailed justification in Appendix K.2).
Based on this refined modeling assumption, we have extended all analyses of the decode stage in the original paper and validated them experimentally. In model-based simulations, the extended theory largely maintains the predictive accuracy of the original approach, while achieving clear improvements in the Vidur-based experiments. The key ideas are as follows.We retain the original fixed-time-window framework and decomposition into resource-dominated regimes, incorporating the effects of dynamic batch processing times into the service description within each fixed window.Under specific parameter settings, the extended theory naturally reduces to the original constant-duration model.
We have incorporated these theoretical extensions into Sections 5.2 (Memory-Dominated Decode) and 5.3 (Concurrency-Dominated Decode), with detailed derivations and proofs in Appendix D (joint reference distribution and concurrency threshold), Appendix E (memory-dominated full distribution), Appendix F (large-deviation tails for both regimes), and Appendix G (concurrency-dominated QED approximation).
① Large-deviation analysis: update the equivalent work requirements while retaining the original tail-exponent framework. Normalizing the service budget in each fixed window to one, the equivalent per-request work requirements in the memory-dominated and concurrency-dominated regimes become
\qquad S_B=\frac{1}{\Delta_D}\left(\frac{c_D}{c}+a_D\right)\omega.$$ The original large-deviation arguments based on reflected random walks and inverse renewal-counting processes therefore remain applicable; the service-demand distribution changes. Under the stated reference-system and tail-regularity assumptions, the two tail exponents satisfy $$\gamma_M=\log\mathbb E\exp\\!\left[ \frac{\Lambda_D(\gamma_M)}{\Delta_D} \left(\frac{c_DW}{M}+a_D\omega\right) \right],$$ $$\gamma_B=\log\mathbb E\exp\\!\left[ \frac{\Lambda_D(\gamma_B)}{\Delta_D} \left(\frac{c_D}{c}+a_D\right)\omega \right].$$ [1] Zhang et al. Threshold-Based Exclusive Batching for LLM Inference. ICML 2026. [2] Yang et al. A Queueing Theoretic Perspective on Low-Latency LLM Inference with Variable Token Length. WiOpt 2024.
Response(1/8)
We thank the reviewer for recognizing the direction of our work and for raising these valuable questions. We acknowledge that your concerns have highlighted opportunities to further improve our work. Over the past period, we have carefully revisited and refined the manuscript in response to your comments. We hope that the following clarifications and additional results will address your concerns.We will add the remaining content soon.
1.Part of W1: prefill process Modeling
We agree with your observation that "due to the computation involved in the attention mechanism, the prefill process typically involves quadratic computational complexity." We would like to clarify two points:
Our analytical method for the prefill stage employs transform-domain analysis of a structured Markov model and does not rely on the assumption of linear service time. It is generally applicable to arbitrary known service-time distributions.
In this paper, we model the prefill service time as a linear function of the input length, which is supported by both theoretical analysis and empirical evidence under certain configurations.
SARA [1] derives the computational complexity of the prefill stage for Transformers employing GQA as follows:
\left[ L^2+ \left(2(1+g)N_{\mathrm{mod}} +2N_{\mathrm{hid}}+1\right)L \right],$$ where $L$ denotes the input length, $N_{\mathrm{mod}}$ is the hidden dimension, $N_{\mathrm{hid}}$ is the FFN intermediate dimension, and $N_{\mathrm{layer}}$ is the number of Transformer layers. Here, $g=N_{\mathrm{KV}}/N_{\mathrm{head}}$. When the model dimensions are sufficiently large relative to the input length, the linear terms dominate the overall computational cost, and the contribution of the quadratic attention term is relatively limited. For example, SARA reports that, under the NVIDIA A100 and LLaMA 3.1 8B configuration ($N_{\mathrm{layer}}=32$, $N_{\mathrm{mod}}=4096$, $N_{\mathrm{hid}}=14336$, and $g=0.25$), the quadratic term accounts for only approximately 2.56% of the total computation when $L=1024$: $$\frac{N_{\mathrm{layer}}N_{\mathrm{mod}}L^2} {N_{\mathrm{mul}}(L)} \approx 2.56\\%.$$ Preble [2] conducts GPU profiling of Mistral-7B on an NVIDIA A6000 and explicitly observes that linear layers dominate prefill latency, with the overall prefill time increasing approximately linearly with prompt length. Experiments in Sarathi-Serve [3] on NVIDIA A100 further demonstrate that, within the evaluated range where linear operators dominate, they contribute more than 80% of the prefill execution time, even when accounting for the quadratic complexity of attention. Moreover, once the system enters the compute-bound regime, the execution time grows approximately linearly with the number of tokens. In addition, following your suggestion, we conducted further experiments using the Vidur simulator. Specifically, under the configuration of 4 $\times$ NVIDIA A100 (TP=4) and LLaMA-3-8B, we fitted the prefill service times across different input lengths using a linear model (Figure K.1 in revised Appendix K.1). We have added a statement regarding this modeling in the revised version (Appendix K.1). [1] Shicong Liu, Xianghao Yu, Zhen Gao, and Jun Zhang. SARA: SLO-Aware Resource Allocation for Disaggregated Agentic LLM Services. arXiv preprint arXiv:2609.26763, 2026. [2] Vikranth Srivatsa, Zijian He, Reyna Abhyankar, Dongming Li, and Yiying Zhang. Preble: Efficient Distributed Prompt Scheduling for LLM Serving. In International Conference on Learning Representations (ICLR), 2025. [3] Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, and Ramachandran Ramjee. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI), pp. 117–134, 2024.