- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a framework for predicting the tail probability of steady-state queue lengths under different batching strategies. The framework takes into account the randomness of request sizes and service times and supports the comparison of performance under different arrival rates, batch sizes, and service configurations. Experiments on real servers show that, under near-full load conditions, the model can accurately estimate the tail probability of queue lengths at various arrival rates.
Critical strengths
-
From modeling reasoning time and deriving from queuing probability theory to long-term experiments on real servers, the accuracy and effectiveness of the model are verified layer by layer.
-
Focus on testing high-load scenarios that approach the system capacity limit, making the validation of the model's accuracy more convincing.
Critical weaknesses
-
This paper studies fixed batching and dynamic elastic batching, but does not cover the more complex continuous batching behavior in modern serving systems.
-
This paper's system model is a single LLM inference node. Actual deployments typically have multiple GPUs. How can it be extended to multiple GPUs?
Questions for authors
- This paper's linear service time model diminishes the impact of input length and prefill length. Therefore, in scenarios with long contexts or high prefill overhead, such as RAG, can this model still accurately predict service time?
2 comments
Thanks for your timely and detailed response. It partially solves my concerns. I am still concerned about continuous batching in modern serving systems.I keep my score. Thank you for your time and effort.
Response on Multi-GPU Settings and Long-Context Applicability
We thank the reviewer for recognizing our validation approach, from modeling to measurements, and our focus on high-load scenarios. We respond below to the questions about extending the model to multi-GPU settings and its applicability to long contexts.
We have added multi-GPU validation in Section 5.2.2, “Reliability under GPU Scaling,” on pages 9–10 of the manuscript. Using vLLM, we deploy the same LLM with two-way and four-way tensor parallelism and compare empirical and theoretical results under fixed and dynamic elastic batching at a load of ρ = 0.90.
The experimental results show that our theoretical tail probabilities remain very close to the empirical tail probabilities. This further demonstrates that our mathematical formulas can accommodate the additional communication and synchronization overhead in multi-GPU environments without introducing extra parameters: only the existing parameters of the service-time model H need to be refitted, followed by recomputation of the arrival rate and stationary distribution.
We agree that the effect of input length cannot generally be ignored. For long-context or prefill-dominated RAG workloads, our mathematical model may need an additional input-token-length parameter to estimate the non-negligible additional time incurred during prefill.
In this paper, we restrict input lengths to 10–200 tokens and omit the prefill-stage timing parameter because our primary focus is short question-answering scenarios. We believe that users in long question-answering scenarios are less sensitive to waiting time and can tolerate some queueing delay, whereas users in short question-answering scenarios expect rapid responses and do not want service to be affected by excessive queueing. Since inputs in short question-answering scenarios are not excessively long, we omit the contribution of the input token count to prefill processing time.