acceptodds
Under review as a conference paper at ICLR 2027

When Does Scheduling Matter? An Empirical Study of Distributed LLM Inference

Abstract

Distributed large language model (LLM) inference requires coordinating parallel execution with the scheduling of requests that vary in arrival time, prompt length, and generation length. These decisions jointly shape throughput, time to first token, time per output token, and completion latency, yet their interaction remains insufficiently understood. We present a systematic study of parallelism and scheduling in distributed LLM inference using SGLang, spanning tensor, pipeline, and expert parallelism, five scheduling policies, diverse workloads, and full- and hybrid-attention models across model sizes and GPU counts. We find that scheduling sensitivity depends on the execution regime established by parallelism, model architecture, and workload. Median performance variation across policies ranges from 1.7% to 35.8% across model–scale groups, while variation across parallelism configurations is frequently larger. Policy gains can reverse across layouts, and configurations that maximize throughput can differ from those that minimize latency. Quantitative and qualitative analyses connect these differences to communication costs, state memory, effective concurrency, and queueing. Separating admission waiting from post-admission execution explains how scheduling affects time to first token, while greater queue pressure accompanies stronger policy effects across deployments. Across the 32 evaluated contexts, our staged selection strategy matches the recorded exhaustive optimum for all five latency objectives, with a maximum throughput loss of 3.84%, while reducing configuration evaluations by 26.7–48%, including screening.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
distributed LLM inference, parallelism, scheduling policy

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_ofNtOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

This paper provides an empirical study on 4 Large Language Models, while varying parallelism layout, batch scheduling policy, workload characteristics, and load. The authors show that parallelism affects inference KPIs more so than the scheduling policy.

Critical strengths

I would first like to thank the authors for their submission. The results section of this paper are experimentally vast and detailed.

Critical weaknesses

Unfortunately, my main issue with this paper is that I cannot find a cohesive story or learning in the narrative. While I appreciate your work, high volume of experimentation, analyses and interpretation does not result in a good empirical study paper. A powerful empirical study often focuses on a key few points, analyzes seldomly studied aspects of a system or problem, or disproves a common assumption about that problem. Some empirical studies are successful because they carry insights from production systems in the industry. Batch scheduling and parallelism are some of the most well-studied topics in LLM inference. The traces, systems and models in this paper are all well-known and open-source. None of the learnings in the paper are at odds with common knowledge in the community.

I believe the authors can substantially improve this paper, if they focus on any of the above. As it stands, the paper is not ready for publication.

Questions for authors

Could the authors describe the key learnings from this work, ideally 1 or 2 insights?

Reviewer_M9wCOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper performs a comprehensive empirical study pairing different model parallelism configurations against various scheduling policies. They used a mix of full-attention and hybrid-attention MoE models from the Qwen family, and replayed the Qwen-Bailian usage traces to measure both throughput and latency. The work is exhaustive and comprehensive, but the conclusions mostly confirm intuition. They found that parallelism choice generally matters more than scheduling policy, that scheduling sensitivity depends on the execution regime and that a policy that helps under one layout can hurt under another. They also proposed using FCFS to screen layouts as the first stage of a staged selection procedure, which cuts down search cost.

Critical strengths

I believe that the work is comprehensive and can act as a useful guide for those looking to optimize model inference settings; specifically for Qwen family of models. The paper itself is easy to read and follow, with reproducibility details.

Critical weaknesses

First, while the mechanical aspect of the work is very comprehensive, I find the novel contribution shallow. The work can be described as a thorough grid search over parallelism settings and request scheduling policies, on a specific family of models (Qwen) and a subset of one traffic dataset. I also find the conclusions largely intuitive and unsurprising.

Second, the claim that FCFS can serve as the screening policy for staged selection is not well substantiated. The choice of FCFS and of keeping two layouts appears to have been made after observing the full grid, and the method is then evaluated on that same grid. There is no held-out model, workload, arrival scale, or hardware to test whether it generalizes, and the baseline (random two layouts) is weak.

Third, from my understanding the Qwen-Bailian traces provide metadata about each request (lengths, timing, prefix structure), but the prompt content is hashed and no response content is recorded. The model is therefore fed synthetic tokens with no semantic content, which can strongly influence how tokens are routed to experts. Under this setting, EP may benefit from a more balanced expert load than real traffic would produce, or real traffic may be more skewed. The paper acknowledges this limitation (App. A.4) but does not measure it, so the EP-related conclusions are unclear.

Questions for authors
  1. The paper uses FCFS as the screening policy for staged selection. Table 3 checks FCFS's shortlist against the other policies, but no other policy is tried as the screen. Does screening with SJF, LOF, or RANDOM work as well? Can staged selection also be evaluated on a held-out model, workload, arrival scale, or hardware setup?

  2. The prompts are synthetic tokens reconstructed from hashed blocks, and App. A.4 notes that expert routing may differ from production traffic. Can the authors report per-expert load or all-to-all imbalance for synthetic and real prompts? How sensitive are the EP-versus-TP conclusions to routing balance?

  3. Each configuration appears to be a single run. Several median policy spreads are below 4% (Fig. 1), and staged selection's mean throughput loss is 0.26%. How much do results vary across repeated runs?

  4. Can the authors rerun the selected layouts and policies on real conversations, and report whether the layout rankings and policy gains match the synthetic-token results?

  5. SJF and LOF receive exact output lengths. How do they perform with predicted or noisy lengths, and with natural stopping instead of ignore_eos=True?

Reviewer_7eugOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the interaction between scheduling policies and parallelism configurations in distributed LLM inference. The authors evaluate different scheduling policies and parallelism layouts across four Qwen models, several workloads, and arrival rates, using SGLang as the serving framework. The main claimed finding is that scheduling performance is quite dependent on the parallelism layout, workload characteristics, and attention architecture. The paper also looks at queueing and KV-cache related factors to explain these differences. Based on the findings, the authors propose a FCFS-based monitoring/screening approach for lowering the size of the configuration search space.

I overall find the novelty of the work pretty limited, and several parts of the paper make it hard to know what the actual takeaways should be for practical model serving systems at scale. The proposed configuration selection method is also quite simple and does not seem to advance much beyond existing work on LLM serving configuration search.

Critical strengths
  • The problem the paper studies is relatively interesting, although I would argue the dynamics the authors show are already well known.

  • There's a decent amount of experimental work which includes multiple models, parallelism layouts, scheduling methods/polices, and at least 4 different workload traces. And the GPU types the authors use are H100 datacenter GPUs which make the evaluations more representative of practical deployments.

  • The authors go beyond just reporting throughput and TTFT numbers and report statistics of KV cache utilizations and admission/queueing delays, esp. in Figs 6-7.

Critical weaknesses
  • I struggle to see what we actually learn beyond something like "configuration matters." How does this change how we design serving systems?

  • I overall see the novelty as very limited. Most observations seem fairly expected to me at least. The FCFS screening is interesting, but the method itself is rather straightforward. Other than that, much of the behavior seems expected from existing understanding of LLM serving (and its deployments). The interaction of workload, scheduling, and parallelism has also been explored before. I don't see a particularly surprising finding here.

  • I don't see much systematic analysis of the much more complicated requirements for systems in deployment such as batching, admission control, SLOs, and preemption here. The authors generally only change the queue ordering. It is not clear to me if the findings would transfer to real-world serving systems rather the single homogeneous H100 cluster and serving stack considered here.

  • Do SJF and LOF assume the output lengths are known beforehand here? If so, that is not something that is known in advance in practice. Which further raises the question of practical applications for me.

Questions for authors
  • What is the main actionable takeaway? (see my first weakness item)

  • Why are only these relatively simple queue-ordering policies considered? Any specific reason why you did't even, for instance, consider different batching strategies or even simple SLO requirements? (as far as I see the authors only use SGLang's default batching configurations)

  • What happens if the system moves from the homogeneous H100 cluster you considered to a heterogeneous setting (which is, btw, very common is real-world deployment settings)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.