acceptodds
Withdrawn from ICLR 2027

DistLMSim: Graph-Level Simulation of Dynamic LLM Inference Serving

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Deploying large language model (LLM) inference serving requires navigating a vast design space: scheduling policies, parallelism strategies (tensor, pipeline, expert), KV cache transfer mechanisms, Mixture-of-Experts (MoE) routing, and speculative decoding parameters—each combination leading to vastly different performance outcomes. Evaluating these configurations through real cluster profiling is prohibitively expensive. Existing simulators each address a subset of this space: fine-grained simulators provide operator-level fidelity but cannot model continuous batching or online serving dynamics; another line of work faithfully models request-level scheduling but sacrifices operator granularity. Critically, neither simulator captures the interactions between these dimensions—for example, how MoE load imbalance affects decode iteration latency under continuous batching, or how KV cache transfer strategy interacts with chunked prefill scheduling—leaving important design questions unanswered. We present DistLMSim, a graph-level simulator that accurately predicts LLM request timing under realistic serving dynamics by combining speculative decoding, continuous batching simulation, and disaggregated serving within a single framework. Across profiled prefill and decode configurations, DistLMSim predicts end-to-end request timing within 0.6%/0.2% (TTFT/TBT) of vLLM on A800 (Qwen3-30B-A3B, TP=4), 2.3%/3.3%/3.0% on H100 (Llama-2-13B), and 2.4%/1.4% on Blackwell-class hardware (DeepSeek-V4-Pro, 8275 GB); when graph-level profiling data is available, its Graph-Level Predictor reproduces every profiled configuration exactly, requiring no interpolation and no fusion correction. Through nine case studies spanning scheduling, MoE routing, speculative decoding, PD disaggregation, chunked prefill, and KV cache transfer, DistLMSim's design-space exploration identifies optimal configurations—TTFT reduced by up to 17 (48-config exploration at QPS=20), a 205 TBT degradation averted by PD disaggregation at QPS=30, and 4.17 TBT speedup at the speculative decoding block size that minimises tail latency () —validated against real deployment measurements. DistLMSim's open-source implementation includes reproducible experiment scripts covering all reported results.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
LLM Simulation, LLM Inference Simulation, Graph-Level Simulation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_eHanOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper provides a simulator for LLM deployment with multiple deployment parameters, including scheduling policies, parallelism, KV cache management, MoE routing, and speculative decoding. The major contribution is that the authors use a graph-level simulation framework to represent the LLM serving process, which is of vital importance and helpful for inference researchers. Preliminary experiments show that the framework predicts simulation time well in TTFT and TBT with different LLMs and hardware.

Critical strengths
  1. The simulation covers plenty of details in the inference serving system, including graph-level kernel calculation, queueing, and scheduling.
  2. The experiments provide an end-to-end comparison with the real inference system.
Critical weaknesses
  1. The framework is useful for researchers to analyze the impact of scheduling and specific operators; however, it is not helpful for thorough what-if analysis.
Questions for authors

What is the major difference between your work and a combination of existing simulators? Do you provide more precise predictions, or do you provide a function that has never been included in any existing simulator?

Reviewer_RXAQOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Deploying LLM inference serving requires managing complex, multi-dimensional design space such as scheduling, parallelism scheduling, MoE routing, speculative decoding, disaggregation. The paper claims that existing simulators are heavily fragmented offering only portions of the search space. The paper proposes DistLMSim to be a unified graph-level simulation framework designed to predict LLM request under realistic serving conditions.

Core insight seems to be that the simulator's minimum execution unit CUDA graph kernel launch helps improve the accuracy. It also integrates continuous batching, disaggregated serving, MoE expert routing, and speculative decoding.

Critical strengths
  • Effort to unify the multi-dimensional design-space exploration seems interesting.
  • Proposing potential actions for production systems.
Critical weaknesses
  • Dependence on target hardware profiling data.
  • Incomplete modeling of low-level framework overheads.
  • Narrow range of evaluation.
  • Seems like "unify" is an overclaim considering that the proposed approach does not seem that extendable.
Questions for authors
  • How does DistLMSim maintain its high accuracy when deployed on new hardware platforms or novel model architectures where pre-existing fused-kernel profiling data is unavailable?
  • Given that continuous batching is abstracted at the iteration level, to what extent do unmodeled factors like PagedAttention dynamic page allocation, kernel preemption, and CUDA-graph capture overheads impact the simulator's fidelity under extreme traffic?
  • Similarly, how much do simplified protocol-level bandwidth formulas—which omit kernel-bypass and CPU offloading effects—skew the RDMA KV-cache transfer predictions?
  • How well would DistLMSim scale its accuracy and simulation overhead to ultra-large dense models exceeding 100B parameters or entirely different execution paradigms?
  • How can this be extended to Non-GPU platforms or CPU-GPU heterogeneous systems that may require more complicated modeling of the system?
  • How can this be extended to various scheduling policies, parallelism, ... especially given the context of heterogeneous and disaggregated systems?
Reviewer_QHw1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes DistLMSim, a graph-level simulator for dynamic LLM inference serving. Its premise is that a simulator should time the same unit the serving engine executes, namely the fused CUDA-graph kernel launch.

  1. Architecture: a scheduling loop that admits requests, runs prefill, transfers the KV cache and simulates decode per iteration under a choice of scheduling policies; several KV transfer strategies; a backend that looks up profiled fused-kernel times and falls back to Charon's analytical, profiled and learned estimators; and a communication model for tensor, expert and pipeline parallelism.

  2. Claimed contributions: continuous batching simulation, disaggregated serving with KV cache transfer, fused-kernel profiling, and support for MoE routing and speculative decoding, all in one framework.

  3. Validation: the paper compares predicted request latency with vLLM measurements on three GPU generations and three models, and with Vidur and Frontier; it compares per-layer accuracy with Charon and KernelSight-LM. It then reports case studies on scheduling, MoE balancing, speculative decoding, chunked prefill, KV transfer and disaggregation.

Critical strengths

The paper identifies a sound timing unit for a serving simulator: the fused CUDA-graph kernel launch that the serving engine actually executes. On a Llama-2-13B prefill layer, the Roofline model underestimates the measured time by 64% and summing separately measured sub-operators still misses by 46%, which the paper attributes to launch overhead, optimized kernels and kernel fusion. The paper presents its serving capabilities, including continuous batching and prefill-decode disaggregation, as following from this alignment. This matters because the premise is a useful lesson for anyone who models the performance of production inference engines.

Critical weaknesses
  1. It is unclear whether the end-to-end accuracy numbers are predictions or calibrated fits, and the measurements are too uniform to separate a good simulator from a trivial one. The paper itself presents the sub-0.1% per-layer error as in-distribution lookup, so the predictive claim rests on the A800 end-to-end errors of 0.6% TTFT and 0.2% TBT in Table 1, which are reached only after phase-specific correction factors whose fitting data the paper does not state. Across the five A800 configurations, the measured TTFT stays between 116.1 and 117.8 ms and the TBT between 40.4 and 40.7 ms, even as the prefill length grows from 256 to 1024 tokens. By my calculation, a constant predictor set to the mean of these measurements would average 0.4% TTFT and 0.2% TBT error, and on the five Blackwell-class points it would average about 2.7% TTFT and 0% TBT error, against the reported 2.4% and 1.4%. This matters because I cannot tell from these comparisons how much predictive skill the simulator has.

  2. The case-study conclusions are drawn at loads the paper does not validate, and two of them depend on modelling choices that the paper does not justify. No scored end-to-end comparison with vLLM goes above 10 queries per second, and the paper notes that its single-GPU run at 20 queries per second is not scored because the decode batch exceeds the profiled range, yet two of the abstract's headline findings sit at 20 and 30 queries per second and are described as validated against real deployment measurements. The case studies also report a baseline decode time of about 2.5 ms per token, while vLLM measures about 40.6 ms on the stated default setup, and their tables and figures are deferred out of the main text. The chunked-prefill result relies on the claim that chunking reduces attention cost from O(n²) to O(c·n), although the paper states that each chunk attends to the accumulated KV cache of its predecessors, so by my calculation the chunks together still cover about n²/2 token pairs; the PIPELINED transfer result uses chunk size 1024, and the paper does not state the chunk size of the DIRECT baseline it is compared with. This matters because the case studies are the paper's main use of the simulator, and I cannot tell which of their conclusions reflect the serving system rather than the model.

  3. The method differs from prior serving simulators less than Figure 1 suggests. The paper describes Vidur as a discrete-event serving simulator that relies on re-profiling for each GPU, and it states that per-operator time comes from the hybrid backend introduced by Charon. Figure 1 places DistLMSim alone in the full-stack online-serving quadrant without a rule for how a simulator's position on its axes is set, and the paper describes LLMServingSim through its 2024 version as cycle-accurate only for specific NPU architectures. This matters because I see the increment over Charon mainly in the fused-kernel timing unit and the serving loop, and the increment over current serving simulators mainly in combining continuous batching, disaggregation, MoE routing and speculative decoding in one tool.

Questions for authors

Clarifications (each answer could change my assessment):

  1. How were the decode and prefill correction factors for the four-GPU A800 results obtained, and were they fitted on the same five configurations reported in Table 1? How was the Blackwell-class result calibrated? If the factors were fitted on the evaluated configurations, I read the reported errors as in-sample and keep my soundness assessment; if they were set on separate configurations, I would raise it.

  2. The single-GPU eager continuous-batching check reports vLLM values identical to the four-GPU A800 measurements: 40.4 and 40.7 ms per token, and 5235 and 5274 ms end to end. The script-to-artifact table also maps both vLLM columns to one benchmark file. Are these the same runs, and if not, how do the two deployments produce identical latencies? If the single-GPU ground truth repeats the four-GPU data, I would no longer count that check as validation.

  3. Which backend and profile produce the case-study decode time of about 2.5 ms per token, given that vLLM measures about 40.6 ms on the stated default setup? If the case studies do not use the validated configuration, I would treat their conclusions as unvalidated.

  4. The single-GPU check notes that a run at 20 queries per second is not scored because its decode batch exceeds the profiled range. Is the vLLM measurement for that run available, and are there other hardware measurements at 20 queries per second or more? A match there would show the simulator's accuracy for that configuration, though not for the scheduling, disaggregation or multi-GPU conclusions.

  5. How does the simulator compute the total attention cost of a chunked prefill, given that each chunk attends to the accumulated KV cache of its predecessors? If the whole prefill is charged O(c·n) rather than about n²/2 token pairs, I would read the chunked-prefill savings as an artifact of the cost model.

  6. Did the DIRECT baseline in the KV-transfer comparison use the same chunk size of 1024 as PIPELINED? If it did not, I would attribute part of the 14–16% TTFT gain to chunking rather than to transfer overlap.

  7. How does DistLMSim differ in capability from LLMServingSim 2.0 (arXiv 2602.23036), which reports prefill-decode disaggregation and MoE support, and from Frontier (arXiv 2605.21312), which reports prefill-decode disaggregation and speculative decoding? The answer would change how much weight I give the claim that the design point is unoccupied.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.