- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposed a two-stage scheduler for inter-instance prefill-completed request routing and intra-instance batch scheduling. The core contribution is a predictor that is sampling-aware and fuses intermediate LLM activations with sampling parameters into a five-class output-length distribution. Evaluations across models and traces show promising improvements in goodput.
Critical strengths
- Simple solution, easy to follow
- Substantial prediction accuracy improvement over existing output length predictors
- Evaluation results look comprehensive and promising
Critical weaknesses
- Did not evaluate on reasoning models, whose reasoning tokens might be challenging to predict
- Lack of ablation study on the impact of the proposed predictor, vs. the predictor used in TRAIL++ or µ-Serve
Questions for authors
- Why the difference among RR, Dynamo, and Presage looks minimal in Figure 9 for BurstGPT traces?
- How's the goodput compared to simply applying the predictor from TRAIL++ or µ-Serve to the PD disagg setup? i.e., an ablation study on the impact of the predictor in Presage.
- How would the performance improvement be on the reasoning models? reasoning tokens introduce another layer of uncertainty which is hard to predict. Reference: ICML 2026, Li, Yueying, et al. "Beyond Prediction: Tail-Aware Scheduling for LLM Inference." arXiv preprint arXiv:2606.18431 (2026).
- How would you compare with this ICML 2026 paper on tail-aware scheduling?
4 comments
Ack to rebuttal
Thank you for addressing my questions and additional experiments. I'll remain my overall positive assessment of the paper's scope and contribution.
One follow up question on the predictor/classifier: how the decision of five output length categories is decided? Is it robust to workload distribution shift? Some lessons learned from adopting prediction based approach in production LLM inference is that we find it hard to generalize across customer workloads; the number of categories is something tricky to decide. If the categorization is coarse-grained, the accuracy is high but we miss capturing the variance within a bucket; if we have fine-grained categorization, it's hard to train the classifier. So there's a tradeoff. I wonder have authors been observing similar issue and if so, what's their mitigation?
We thank the reviewer for raising this. It is something we ran into ourselves while designing the predictor. We observed the same trade-off: finer buckets lower prediction accuracy, while coarser buckets carry less length information. We therefore predict a distribution rather than a length or a single bucket class.
The distribution is what the scheduler consumes: its continuous expected length
plus tail mass, so the decision is not restricted to five discrete values. The training target is a distribution too: we generate each request 50 times(Section F Fig 16) and use those lengths as the distribution label, so the predictor learns the conditional length distribution rather than a one-hot bucket. We use five buckets, which we tuned across our workloads: it suits the current workloads, keeping prediction error low while still providing enough length information for the scheduling decisions.
We agree that prediction generalization to customer workloads is indeed not strong. Prediction error is unavoidable. What matters is the loop that corrects it. Therefore, in each decode instance, Presage refines the predicted remaining length online with updates
The decode instance reports its real-time load back to the global scheduler, which revises later routing decisions, and its measured per-token latency feeds back to admission, which then adjusts the window. So we have a predict, feedback, adjust loop (adjust covers routing and eviction): residual error is corrected downstream rather than accumulating.
In short, we predict a distribution rather than commit to a bucket: the scheduler consumes a continuous length estimate from a coarse classifier, and whatever prediction error remains is corrected downstream by the predict–feedback–adjust loop.
Thank reviewer for the Constructive Discussion
As the discussion period comes to a close, we would like to sincerely thank you for your thoughtful review, constructive questions, and helpful follow-up.
Your suggestions prompted us to add a reasoning-model evaluation, predictor ablations against TRAIL++ and µ-Serve, and a comparison with tail-aware scheduling. We especially appreciate your practical observation about the trade-off in choosing output-length categories and the difficulty of generalizing prediction models across customer workloads. This encouraged us to examine the limitations of our predictor more carefully and to clarify that Presage predicts a distribution rather than committing to a single output-length category, while using online feedback to mitigate the remaining prediction errors.
We agree that workload distribution shift remains an important and challenging practical issue, and our original response did not explain this aspect sufficiently clearly. We will revise the manuscript to present the design trade-offs, limitations, and mitigation mechanisms more precisely. Thank you again for the careful and constructive discussion, which has helped us improve the manuscript.
We thank the reviewer for the careful and constructive review. All goodput tables below are measured under a 1000 ms TTFT and 30 ms TPOT SLO.
Q1 (BurstGPT light load):
On BurstGPT the three methods' goodput is lowest because the workload itself is too light. Its average rate is about avg 2 req/s, so the throughput upper bound is 1334 tok/s, goodput cannot exceed throughput, so even if every request meets the SLO, goodput is capped at about 1334 tok/s. Qwen and Mooncake traces have far higher throughput (9905 and 6539 tok/s), so their goodput is higher despite lower SLO attainment. BurstGPT's low goodput thus reflects the low throughput bound, not dropped requests. We have added Appendix F Fig. 13, which confirms this: the QPS panel is low and bursty, goodput overlaps at 1334 tok/s with P99 TPOT 18.5~20 ms, and the three TPOT CDF curves lie entirely left of the 30 ms line.
W2&Q2&Q3 (predictor ablation, tail-aware):
Thank you for pointing out our shortcomings, we have added the following experiments to demonstrate the effectiveness of our method:
Prediction MAE is the length-prediction error in tokens (lower is better); bucket top-1 is the top-1 accuracy over the five length buckets (higher is better).
Fixing PD disaggregation, routing, and GWF admission and swapping only the predictor, better prediction yields higher goodput.
Tail-aware targets intra-instance scheduling in the PD-fused setting, we swap only GWF for tail-aware and keep predictor and routing fixed. Comparing Tail-aware with GWF demonstrates GWF's effectiveness. Although tail-aware modestly reduces tail latency (p99 TPOT: tail-aware: 172 ms vs. Presage: 180 ms), its model-free estimation incurs large errors that amplify the impact of preemption (recompute: tail-aware: 3M tokens vs. Presage: 32k tokens), degrading performance. We add this comparison to Appendix G(Table 3).
W1&Q4 (reasoning models):
We agree reasoning tokens add output-length uncertainty, but it stays predictable because the sampling configuration (temperature, top-p, top-k, repetition penalty) still governs the autoregressive output, and that configuration is exactly what Presage conditions on.
We train TRAIL++, STAR, and Presage on 10k Qwen3-14B (thinking) generations, only Presage is sampling-aware. The sampling-embedding ablation is:
The sampling-parameter embedding contributes +9.09 percentage points accuracy and +10 percentage points cosine, and regression MAE 6575 is below STAR's 7558. The remaining randomness is absorbed by predicting a distribution rather than a point length or a single bucket. This comparison is added to the appendix G (Table 4).