- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper use results in queuing theory to optimize the admisson policy of inference engine. They provide a solid formulation for known output length of request. They derive an integer optimization formulation for selecting the budgets for a periodic flow-controlled policy. For unknown output length of request, they propose an easy-to-implement periodic flow-controlled scheduling algorithm with LIFO eviction. The numerical simulation results show the effectiveness of the method.
Critical strengths
1.The target question for optimizing LLM serving engines is good. 2.The theoretical results are elegant and offer novel structural insights. 3.The scheduling policy is simple, interpretable, and straightforward to implement.
Critical weaknesses
1.The theory is solid, but how can this theory be translated into practical scheduling benefits under specific workloads? 2.Except for evaluation section, there is no picture.
Questions for authors
1.How can your theory direct me in real system deployment? Could you present some experiments of real vllm (not simulation by Vidur)? 2.Why only short query (input/output length < 500)? Long querys need more KV cache which seems to be more useful to demonstrate effectiveness of your method.
Thank you for reviewing our paper and for the helpful comments.
1. Deployment and real GPU experiments
Thank you very much for the positive assessment of the theory. To test the performance of our algorithm in practice, we evaluated our policy on Mistral-7B-Instruct-v0.3 using an RTX 4090 and 4 GiB of KV cache and compared it against vLLM and Sarathi-Serve. We selected the budgets and tested each configuration three times on ten request flows. Each request flow consists of a sequence of different requests. The arrival rate was set to be 48 requests/s. Results are as follows:
Across the three budgets, our method reduces average latency by 16.2%–17.7% compared with vLLM and 7.7%–9.3% compared with Sarathi-Serve. To check repeatability, we ran 15 repetitions on two request flows. Across these two flows, the mean latency remained 14.4%-17.1% lower than vLLM and 8.9%-10.3% lower than Sarathi.
2. Figures
Thank you for highlighting the question on pictures. We will add more illustrations to the main text to make the algorithm and theoretical results easier to follow. We will also add a figure of the real GPU results.
3. Longer requests
Thank you for pointing out this question. Following your suggestions, we considered all requests such that the input and output length at most 4,096 tokens and set the arrival rate as 10 requests/s over a time period of T=1000 seconds. Because the average workload increased, we set the budget b=0.6 (i.e., at each slot, we admit one request with probability 0.6 and no request otherwise). We report the mean and 95% latency:
Our mean latency is 18.4% and 5.9% lower than vLLM and Sarathi, respectively. However, Sarathi achieves lower 95% latency. The results suggest that our advantages over the benchmarks may not become more pronounced. This is because longer requests make evictions more expensive. Since our policy restarts evicted requests without storing the prefill and decoded tokens, these costs may offset the effectiveness of our method. By contrast, Sarathi uses chunked prefill to reduce interference with ongoing decoding, which can make it less sensitive to longer prompts.