- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper provides a simulator for LLM deployment with multiple deployment parameters, including scheduling policies, parallelism, KV cache management, MoE routing, and speculative decoding. The major contribution is that the authors use a graph-level simulation framework to represent the LLM serving process, which is of vital importance and helpful for inference researchers. Preliminary experiments show that the framework predicts simulation time well in TTFT and TBT with different LLMs and hardware.
Critical strengths
- The simulation covers plenty of details in the inference serving system, including graph-level kernel calculation, queueing, and scheduling.
- The experiments provide an end-to-end comparison with the real inference system.
Critical weaknesses
- The framework is useful for researchers to analyze the impact of scheduling and specific operators; however, it is not helpful for thorough what-if analysis.
Questions for authors
What is the major difference between your work and a combination of existing simulators? Do you provide more precise predictions, or do you provide a function that has never been included in any existing simulator?