acceptodds
Under review as a conference paper at ICLR 2027

Presage: Bottleneck-Aware Two-Stage Scheduling for Disaggregated LLM Serving

Abstract

PD-disaggregated LLM serving splits request scheduling into two decisions: inter-instance routing of prefill-completed requests, and intra-instance admission into the running batch. Both commit resources for a duration set by the request's remaining output length, which is unknown at decision time, so existing systems fall back on myopic heuristics over observable state, which produces multidimensional load imbalance across decode instances and preemption cascades within them. We present Presage, a two-stage scheduler that couples both decisions through a shared distributional predictor. A sampling-aware predictor fuses intermediate LLM activations with sampling parameters into a five-class output-length distribution, improving cosine similarity by 16.5%–47.9% over the matched TRAIL++ and μ-Serve baselines; it runs concurrently with prefill at a median latency of 12.7 ms and, for 99.8% of requests, completes within the prefill duration, adding no end-to-end latency. Guided by this distribution, a dual-path router spends decision effort in proportion to load: below saturation it takes a sub-microsecond token-reservation fastpath, while near saturation, where routing errors are costly, it projects load onto compute, memory-bandwidth, and capacity pressure. Within each decode instance, a TPOT-driven admission controller tracks the measured per-token latency and adapts its admission window to it, halving the window when latency exceeds its target and growing it otherwise. On 8 A800 GPUs across three production traces, two model families, and two PD ratios, Presage achieves 1.02× to 2.17× the goodput of round-robin and NVIDIA Dynamo, with the largest gains where prefix-cache affinity creates hotspots.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
LLM Serving, Request Scheduling, Inference System

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_rQ14OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposed a two-stage scheduler for inter-instance prefill-completed request routing and intra-instance batch scheduling. The core contribution is a predictor that is sampling-aware and fuses intermediate LLM activations with sampling parameters into a five-class output-length distribution. Evaluations across models and traces show promising improvements in goodput.

Critical strengths
  • Simple solution, easy to follow
  • Substantial prediction accuracy improvement over existing output length predictors
  • Evaluation results look comprehensive and promising
Critical weaknesses
  • Did not evaluate on reasoning models, whose reasoning tokens might be challenging to predict
  • Lack of ablation study on the impact of the proposed predictor, vs. the predictor used in TRAIL++ or µ-Serve
Questions for authors
  • Why the difference among RR, Dynamo, and Presage looks minimal in Figure 9 for BurstGPT traces?
  • How's the goodput compared to simply applying the predictor from TRAIL++ or µ-Serve to the PD disagg setup? i.e., an ablation study on the impact of the predictor in Presage.
  • How would the performance improvement be on the reasoning models? reasoning tokens introduce another layer of uncertainty which is hard to predict. Reference: ICML 2026, Li, Yueying, et al. "Beyond Prediction: Tail-Aware Scheduling for LLM Inference." arXiv preprint arXiv:2606.18431 (2026).
  • How would you compare with this ICML 2026 paper on tail-aware scheduling?
4 comments

We thank the reviewer for the careful and constructive review. All goodput tables below are measured under a 1000 ms TTFT and 30 ms TPOT SLO.

Q1 (BurstGPT light load):

On BurstGPT the three methods' goodput is lowest because the workload itself is too light. Its average rate is about avg 2 req/s, so the throughput upper bound is 1334 tok/s, goodput cannot exceed throughput, so even if every request meets the SLO, goodput is capped at about 1334 tok/s. Qwen and Mooncake traces have far higher throughput (9905 and 6539 tok/s), so their goodput is higher despite lower SLO attainment. BurstGPT's low goodput thus reflects the low throughput bound, not dropped requests. We have added Appendix F Fig. 13, which confirms this: the QPS panel is low and bursty, goodput overlaps at 1334 tok/s with P99 TPOT 18.5~20 ms, and the three TPOT CDF curves lie entirely left of the 30 ms line.


W2&Q2&Q3 (predictor ablation, tail-aware):

Thank you for pointing out our shortcomings, we have added the following experiments to demonstrate the effectiveness of our method:

MethodConfigurationGoodput (tok/s)Prediction MAEbucket top-1
Presagesampling-aware + dual-path+ GWF581.35056.146.0%
tail-awaresampling-aware + dual-path+ tail-aware510.6——
TRAIL++TRAIL++ + dual-path+ GWF500.346301.838.2%
µ-Serveµ-Serve+ dual-path + GWF458.527067.730.1%

Prediction MAE is the length-prediction error in tokens (lower is better); bucket top-1 is the top-1 accuracy over the five length buckets (higher is better).

Fixing PD disaggregation, routing, and GWF admission and swapping only the predictor, better prediction yields higher goodput.

Tail-aware targets intra-instance scheduling in the PD-fused setting, we swap only GWF for tail-aware and keep predictor and routing fixed. Comparing Tail-aware with GWF demonstrates GWF's effectiveness. Although tail-aware modestly reduces tail latency (p99 TPOT: tail-aware: 172 ms vs. Presage: 180 ms), its model-free estimation incurs large errors that amplify the impact of preemption (recompute: tail-aware: 3M tokens vs. Presage: 32k tokens), degrading performance. We add this comparison to Appendix G(Table 3).


W1&Q4 (reasoning models):

We agree reasoning tokens add output-length uncertainty, but it stays predictable because the sampling configuration (temperature, top-p, top-k, repetition penalty) still governs the autoregressive output, and that configuration is exactly what Presage conditions on.

We train TRAIL++, STAR, and Presage on 10k Qwen3-14B (thinking) generations, only Presage is sampling-aware. The sampling-embedding ablation is:

MethodAccuracyF1CosineMAE
TRAIL++0.39760.38040.5532–
Presage0.48850.45920.6532–
STAR–––7558
Presage–––6575

The sampling-parameter embedding contributes +9.09 percentage points accuracy and +10 percentage points cosine, and regression MAE 6575 is below STAR's 7558. The remaining randomness is absorbed by predicting a distribution rather than a point length or a single bucket. This comparison is added to the appendix G (Table 4).


Reviewer_rQ14OpenReview ↗

Ack to rebuttal

Thank you for addressing my questions and additional experiments. I'll remain my overall positive assessment of the paper's scope and contribution.

One follow up question on the predictor/classifier: how the decision of five output length categories is decided? Is it robust to workload distribution shift? Some lessons learned from adopting prediction based approach in production LLM inference is that we find it hard to generalize across customer workloads; the number of categories is something tricky to decide. If the categorization is coarse-grained, the accuracy is high but we miss capturing the variance within a bucket; if we have fine-grained categorization, it's hard to train the classifier. So there's a tradeoff. I wonder have authors been observing similar issue and if so, what's their mitigation?

We thank the reviewer for raising this. It is something we ran into ourselves while designing the predictor. We observed the same trade-off: finer buckets lower prediction accuracy, while coarser buckets carry less length information. We therefore predict a distribution rather than a length or a single bucket class.

The distribution is what the scheduler consumes: its continuous expected length

E[L]=∑k=15qkbk(Eq. 6)E[L] = \sum_{k=1}^{5} q_k b_k \quad \text{(Eq. 6)}

plus tail mass, so the decision is not restricted to five discrete values. The training target is a distribution too: we generate each request 50 times(Section F Fig 16) and use those lengths as the distribution label, so the predictor learns the conditional length distribution rather than a one-hot bucket. We use five buckets, which we tuned across our workloads: it suits the current workloads, keeping prediction error low while still providing enough length information for the scheduling decisions.

We agree that prediction generalization to customer workloads is indeed not strong. Prediction error is unavoidable. What matters is the loop that corrects it. Therefore, in each decode instance, Presage refines the predicted remaining length online with O(1)O(1) updates

pr(g)=∑k=15qkmax⁡(0,bk−g)(Eq. 9)p_r(g) = \sum_{k=1}^{5} q_k \max(0, b_k - g) \quad \text{(Eq. 9)}

The decode instance reports its real-time load back to the global scheduler, which revises later routing decisions, and its measured per-token latency feeds back to admission, which then adjusts the window. So we have a predict, feedback, adjust loop (adjust covers routing and eviction): residual error is corrected downstream rather than accumulating.

In short, we predict a distribution rather than commit to a bucket: the scheduler consumes a continuous length estimate from a coarse classifier, and whatever prediction error remains is corrected downstream by the predict–feedback–adjust loop.

AuthorsOpenReview ↗

Thank reviewer for the Constructive Discussion

As the discussion period comes to a close, we would like to sincerely thank you for your thoughtful review, constructive questions, and helpful follow-up.

Your suggestions prompted us to add a reasoning-model evaluation, predictor ablations against TRAIL++ and µ-Serve, and a comparison with tail-aware scheduling. We especially appreciate your practical observation about the trade-off in choosing output-length categories and the difficulty of generalizing prediction models across customer workloads. This encouraged us to examine the limitations of our predictor more carefully and to clarify that Presage predicts a distribution rather than committing to a single output-length category, while using online feedback to mitigate the remaining prediction errors.

We agree that workload distribution shift remains an important and challenging practical issue, and our original response did not explain this aspect sufficiently clearly. We will revise the manuscript to present the design trade-offs, limitations, and mitigation mechanisms more precisely. Thank you again for the careful and constructive discussion, which has helped us improve the manuscript.

Reviewer_xgXyOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper uses output-length prediction to guide both routing and admission in PD-disaggregated serving. The predictor takes intermediate activations and sampling settings, and produces probabilities over five length buckets. The router estimates future resource pressure, while the admission controller adjusts concurrency using TPOT feedback. The evaluation uses eight A800s, two model families, three arrival traces, and two PD ratios, and shows improvements in SLO goodput.

Critical strengths
  • Including sampling parameters in the length predictor makes sense. Table 1 gives a reasonably direct comparison showing what this input adds. I also like that the prediction is used for a real scheduling decision, rather than just reporting prediction accuracy.

  • The routing ablation is helpful and show a fairly substantial gain: adding predicted demand raises goodput from 2,155 to 2,861 tok/s with the admission controller unchanged. Together with the different traces and models, this is enough to make me positive about the basic direction.

Critical weaknesses

I am a bit confused about the use of the distribution. Algorithm 1 calls EXPECTEDLENGTH, so it is not clear where uncertainty beyond the mean enters the routing decision. Equation 7 also seems to use the original probabilities after we already know a request has survived to g tokens. For example, with bucket centers (100,300,500,700,900), probabilities (0.9,0,0,0,0.1), and 200 tokens generated, the equation gives 70 remaining tokens; conditioning on the request still running gives 700 under this model. Is there another correction in the implementation? The heuristic may still work well, but the interpretation needs to be clear.

The admission part is harder to follow than the routing part. I would like more detail on eviction value, and how the 120 ms feedback target works with the 30 ms evaluation SLO. A small admission comparison would help, though I do not think a full study of every component combination is needed. The predictor and routing results are already useful.

Questions for authors
  1. Is Equation 7 meant as a heuristic score, or as expected remaining length for an unfinished request? How does the implementation avoid underestimating the long requests that are still running?

  2. Could you walk through one example of how predicted length, eviction value, and the 120/30 ms targets affect admission? An existing trace or a small admission-only result would help explain what the second stage adds.

3 comments

We thank the reviewer for the careful and constructive review. All goodput tables below are measured under a 1000 ms TTFT and 30 ms TPOT SLO.

W1&Q1 (distribution use, Eq 7):

Thank you for pointing out the issue. On ExpectedLength, it uses the expected length over the whole distribution, not the argmax: your example gives 0.9×100 + 0.1×900 = 180, not 100. The long-bucket mass pulls the routing signal to 180, so uncertainty enters routing; a point estimate gives 100 and discards the tail.

On Equation 9 (Equation 7 in the submitted version), you are right that it estimates 70 rather than 700, a deliberate trade-off: it discounts the tail (0.1×700) by the original probability. Conditioning would turn a wrong tail guess into a full 700 tokens, which would hurt preemption and load feedback, while re-running the predictor mid-decode would keep full activations, interrupt decoding, and add extra latency and storage overhead.

Equation 9 is the compromise: O(1) update, no re-prediction, but it is a heuristic score rather than the strict conditional remaining length, at the cost of underestimating still-running requests. We have added the ExpectedLength formula to the main text (Equation 6).


W2&Q2 (eviction value, 120 vs 30 ms):

Thank you for pointing out that the admission part is unclear, we clarify the eviction value and the 120/30 ms targets, and add an admission-only comparison. On a 1P1D setup with routing and prediction fixed, comparing only FCFS and GWF admission on Qwen-2.5-7B-Instruct with 4k LMSYS-Chat-1M requests (0.85 req/s), FCFS does not limit concurrency and, when KV is full, repeatedly preempts, causing a recomputation storm (10912 preemptions, 1.55M recomputed tokens, P99 TPOT 204.4 ms).

AdmissionGoodput (tok/s)
FCFS133.51
GWF150.17

GWF has two improvements:

  • Admission window: this controls admitting waiting-queue requests. It takes the 30 ms TPOT SLO as baseline and sets θ = 120 ms (4×30 ms). The window shrinks or grows with τ, the generation interval of the last smin(=4) tokens of running requests (smin = 4 avoids jitter; requests with fewer than 4 tokens use their actual generated count): τ > θ halves the window, and τ < θ grows it by an eighth.
  • Eviction value: eV(r) = (1 − γr) + σr + ρr, where:
TermDefinitionWhy it raises eV
1 − γrγr = gr/ℓr, fraction of length already generatedgenerated little → cheap to recompute
σr1 − TPOTr/BTPOT, TPOT headroomhas slack → absorbs a preemption
ρr(ℓr − gr)/Lmax, remaining-KV fractionowes much output → frees the most KV

gr, ℓr: generated and predicted output lengths; TPOTr: observed per-token latency; BTPOT: TPOT SLO; Lmax: maximum context length. We have corrected Algorithm 2 to preempt in descending eV order (hopeless first, largest eV first).

Compared with FCFS, GWF has 7.4× fewer preemptions (GWF: 1474 vs FCFS: 10912) and 31.5% less recomputation (GWF: 1.06M vs FCFS: 1.55M), the window decides how many to admit and the eviction value decides whom to evict, together giving +12.5% goodput, P99 TPOT drops from FCFS's 204.4 ms to GWF's 171.8 ms. We have added the eV formula to main text (Equation 8) and this comparison to the Appendix H (Table 5).


Reviewer_xgXyOpenReview ↗

Thanks for the admission comparison

Thanks for adding the admission-only comparison. Holding routing and prediction fixed makes it easier to see what GWF adds: goodput rose from 133.5 to 150.2 tok/s, with substantially fewer preemptions in that setup. The explanation of the 120 ms control threshold and the 30 ms evaluation SLO is helpful, as is the clarification that Eq. 9 is a heuristic for remaining work. My rate of weak accept remains unchanged

We sincerely thank you for your thoughtful comments and support of our work. We are grateful for the time and effort that you devoted to reviewing our manuscript.

Reviewer_cgTvOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Presage, a system with a two-stage scheduler. In the first stage, a sampling-aware predictor uses sampling parameters to build a prediction model for the output length distribution. In the second stage, a router uses this distribution to evict or demote certain requests in favor of others. The evaluation shows that Presage performs better than round-robin.

Critical strengths

The authors propose using sampling parameters as part of the prediction model, which I believe has not been explored before.

Critical weaknesses
  • The paper should compare against more realistic baselines. Round-robin is known to perform poorly. A much stronger baseline would be least-loaded routing (with respect to both requests and tokens), and a slightly more advanced baseline could combine different optimization functions, as real tools like llm-d [1] do. Additionally, the paper claims that prediction-based solutions perform poorly owing to their stochastic nature, yet it does not include any such baselines in the evaluation.
  • Section 4 is difficult to follow, and it is hard to identify any new observation. Previous works have shown that throughput is not a good measure, which is why most works measure goodput. In the goodput discussion, I also do not understand what the budget refers to.
  • Observation 1 needs to be tightened; the existing literature has already explored almost everything it discusses.
  • For Observation 2 in particular, I wonder whether simply tracking KV cache utilization would solve the problem. The authors claim that admitting more requests improves utilization but can also cause preemption. This is correct, and it is the key reason most systems track KV cache usage and keep a buffer rather than admitting requests without regard to cache utilization.
  • The evaluation is lacking. The paper needs to show goodput as a function of varying workload. Figure 9 does not make clear whether the system is at peak load or simply underloaded. It is also unclear what numbers Figure 9 reports: the mean or some point on the tail?

[1] https://github.com/llm-d/llm-d-router

Questions for authors
  1. In Figure 1(b), what if the scheduler used KV cache usage rather than the number of requests in the queue?
  2. How does your load model differ from the ones used in DistServe [1] or Ascendra [2]?
  3. In the bottom row of Figure 9, why do all baselines perform the same?
  4. How does Presage compare to a realistic setup [3] or a prediction-based system [4, 5]? Or even to a simple cache-capacity-aware, least-loaded routing strategy?
  5. What limits this solution to a PD-disaggregated setup? Why can’t the same approach work in a non-disaggregated setup?
  6. Is the scheduler proposed in Section 5.4 fair? Couldn’t some requests starve indefinitely because of the demotion mechanism?

[1] Zhong, Yinmin, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. "{DistServe}: Disaggregating prefill and decoding for goodput-optimized large language model serving." In 18th USENIX symposium on operating systems design and implementation (OSDI 24), pp. 193-210. 2024.
[2] Ikram, Azam, Xiang Li, Sameh Elnikety, and Saurabh Bagchi. "Ascendra: Dynamic request prioritization for efficient llm serving." arXiv preprint arXiv:2504.20828 (2025).
[3] https://github.com/llm-d/llm-d-router
[4] Li, Yueying, Yuanfan Chen, Jiayang Chen, Esha Choukse, Haoran Qiu, G. Edward Suh, Rodrigo Fonseca, Ziv Scully, and Udit Gupta. "Beyond Prediction: Tail-Aware Scheduling for LLM Inference." arXiv preprint arXiv:2606.18431 (2026).
[5] Zheng, Haoyu, Yongqiang Zhang, Fangcheng Fu, Xiaokai Zhou, Hao Luo, Hongchao Zhu, Yuanyuan Zhu, Hao Wang, Xiao Yan, and Jiawei Jiang. "Scheduling LLM Inference with Uncertainty-Aware Output Length Predictions." arXiv preprint arXiv:2604.00499 (2026).

4 comments

We thank the reviewer for the careful and constructive review. Goodput tables below: 1000 ms TTFT / 30 ms TPOT SLO

Q1 (KV cache occupancy routing):

If routing changes from queue length to KV cache occupancy, it does not overload this instance, but it shifts the imbalance elsewhere: reading only current occupancy, it ignores the additional load that later requests will still generate. We add a concrete example as Fig 12 in Appendix E.


W3&Q2 (load model vs DistServe/Ascendra):

All three compute load and memory access the same way, they differ only in the output. DistServe and Ascendra turn these into latency. Both ask when a request completes, not which resource is exhausted. We output the pressure each request puts on three resources relative to the TPOT SLO budget, the maximum is the bottleneck, and the gap to the budget ceiling is the remaining headroom.


W5&Q3 (Fig 9 load range and reported metric):

Under the BurstGPT workload, the performance of various methods tends to converge due to the relatively light load. The average request rate is approximately 2 req/s, with an effective throughput (goodput) of about 1334 tok/s. P99 TPOT ranges from 18.5 ms to 20 ms, all of which meet the 30 ms TPOT SLO. Presage and baselines show almost identical goodput. We have included additional charts in Appendix F. Fig 9 reports mean output-token goodput, not per-request percentile.


W1&Q4 (stronger baselines, prediction-based systems):

We add llm-d, tail-aware [4], cache-capacity (P2D-Capacity) and least-loaded (P2D-3DLoad) baselines. Tail-aware targets intra-instance scheduling in the PD-fused setting, we swap only GWF for tail-aware and keep predictor and routing fixed.

MethodConfigurationGoodput (tok/s)Prediction MAEbucket top-1
Presagesampling-aware + dual-path+ GWF581.35056.146.0%
llm-dlatency predictor + FCFS512.4——
llm-d+GWFlatency predictor + GWF564.2——
tail-awaresampling-aware + dual-path+ tail-aware510.6——
TRAIL++TRAIL++ + dual-path+ GWF500.346301.838.2%
µ-Serveµ-Serve+ dual-path + GWF458.527067.730.1%

llm-d can predict that a request may time out, but it has no intra-instance admission policy to optimize for that, GWF admission alone adds +10.1%, the distributional predictor plus routing a further +3.0%.

Although tail-aware modestly reduces tail latency (p99 TPOT: tail-aware: 172 ms vs. Presage: 180 ms), its model-free estimation incurs large errors that amplify the impact of preemption (recompute: tail-aware: 3M tokens vs. Presage: 32k tokens), degrading performance. We add this comparison to Appendix G.

Regarding [5], its distributional length prediction for SJF is the same class as a model-based predictor plus tail-aware scheduling, so we only test the latter.

TRAIL++ and μ-Serve are bucket classifiers that output a bucket interval, we treat them as point predictors. Presage predicts the full distribution over the five bucket intervals and routes on its expectation, retaining the tail risk that a point estimate discards.

The routing ablation, already in Appendix D, Fig 11(a), is:

RouterGoodput (tok/s)
P2D-Capacity2147
P2D-3DLoad2155
Presage-3DLoad2861

The results demonstrate the effectiveness of the Presage inter-instance routing strategy.


Q5 (PD-disaggregation scope):

Presage targets PD-disaggregated serving: predictor reads intermediate prefill activations, which exist during prefill and are more accurate than prompt-only signals (Table 1). On a co-located setup, after prediction a request either stays local (the prediction cannot inform inter-instance scheduling) or migrates (incurring KV-migration overhead). Disaggregation is also mainstream: DistServe, Splitwise, Mooncake, and Dynamo all separate prefill from decode, so this assumption is well grounded.


Q6 (fairness / starvation):

Algorithm 2 walks the queue in arrival order and never reorders it: a request is demoted only when it exceeds its TTFT/TPOT budget, then force-promoted after at most Dmax=2000 ms.


W2&W4 (budget, goodpur, KV-utilization tracking):

Here budget refers to how many requests/tokens an instance can still serve within the TPOT SLO headroom.

Fig 6 shows throughput stays high after goodput collapses, so we use goodput to measure system performance. We also agree static KV buffers are common, but their size is a dilemma: too small preempts, too large idles memory (Fig 2) A conservative strategy that sets the admission threshold at 0.8 times total capacity may result in requests being blocked because they fail to pass the admission check. The result shows:

AdmissionGoodput (tok/s)
Conservative90.56
GWF118.80

The 20% buffer fails: long outputs consume it, admission freezes and requests starve. This comparison is added to Appendix I (Table 6).


I thank the authors for adding the llm-d baseline and for including GWF with it. Both additions strengthen the manuscript.

I agree that any policy that does not predict output length will run into issues and will therefore need an admission control policy. Since Presage estimates the output length, I would expect it to perform better. This leads me to a clarifying question: is it fair to say that the key contribution of this work is using the sampling parameters to learn the output length? If so, can Presage be understood as llm-d + SP?

On Q2, I agree with the authors that there is essentially no difference between the models proposed by existing methods and Observation 1. The authors state that existing methods use the model only to estimate latency and not to identify the bottleneck. However, to estimate the latency of a request, a model must first identify the bottleneck and then compute the time along that path. In that sense, Observation 1 appears to follow from existing work, and I would appreciate it if the authors could clarify what it adds beyond those works.

On the BurstGPT load test, the authors explain that the load is not high enough to saturate the system, so all baselines perform similarly. I appreciate the explanation, but I am unsure what the reader should take away from this result. It would be more informative to see how the system behaves near its capacity. I suggest using another dataset or bootstrapping BurstGPT to increase the load.

Q1: Is Presage llm-d plus a sampling-aware predictor?

We thank the reviewer for the question. The essential difference is the decision horizon. llm-d uses the state available at arrival time to select a suitable instance, targeting instantaneous load balance. In autoregressive inference, however, output length remains uncertain after dispatch; a decision that is initially safe can later create decode pressure, KV-capacity pressure, and preemption. Presage instead targets sustained goodput over the lifetime of requests.

This requires more than a better predictor. Presage predicts an output-length distribution conditioned on sampling parameters, updates the expected remaining length in O(1) as tokens are generated, and uses measured load and per-token latency to inform later routing and admission decisions. These mechanisms mitigate the downstream impact of imperfect predictions.

The ablation isolates the TPOT-driven admission controller by adding it to llm-d while retaining llm-d's predictor. This change alone raises goodput by 10.1% (Appendix G, Table 3), without using our sampling-aware predictor. Thus, Presage is not llm-d plus SP: it coordinates inter-instance routing and intra-instance control to maintain long-term goodput.


Q2: Observation 1 and the load model

Our contribution is the systematic derivation of the bottleneck model, rather than another empirical routing heuristic. We formulate compute, memory-bandwidth, and KV-capacity pressure as three dimensionless indicators, their maximum identifies the active bottleneck without assuming it in advance. The scheduler design follows from this analysis.

The cited works take different approaches. DistServe assumes compute-bound prefill and memory-bound decode at design time. Ascendra, as cited by the reviewer, estimates latency from FLOPs and memory accesses through a fitted model. These approaches predict performance within their assumptions but do not identify whether compute, bandwidth, or capacity is currently binding. In particular, a scalar latency model cannot represent KV capacity, which is a stock rather than a flow and is not bounded by the TPOT SLO.

Fig 4(a) shows that the binding resource changes across configurations, while Fig 14 in Appendix J shows that it also changes over time within one decode instance. A request can satisfy the execution-time SLO while the KV cache is nearly full, so a latency-only model may still route it to an instance that later overflows and preempts requests. The novelty is therefore the analysis and formulation showing that the bottleneck is three-dimensional and dynamic, followed by a scheduler derived from that result.


Q3: BurstGPT in Fig 9

BurstGPT is an intentionally light-load case rather than a scheduling stress test. Appendix F, Fig. 13 confirms that its native request rate is only approximately avg 2 requests/s and that all methods remain comfortably below the 30 ms TPOT SLO. Their similar goodput is therefore expected under limited contention. Crucially, Presage matches the baselines in both throughput and TPOT, showing that its prediction, routing, and admission mechanisms introduce no observable latency overhead, throughput loss, or unnecessary scheduling interference when the system is underloaded.

We additionally evaluate a saturated BurstGPT workload in Appendix K, Fig. 15, using a longer trace segment containing 3,089 requests from LMSYS-Chat-1M on 2P6D Llama-3-8B-Instruct. Under saturation, requests compete for decode capacity and scheduling decisions become consequential. Presage then improves goodput by 1.70× over Round-Robin and 1.16× over Dynamo. Together, the two experiments show that Presage preserves baseline performance under light load while providing substantial gains once the workload creates meaningful resource contention and SLO pressure.


AuthorsOpenReview ↗

Thank reviewer for the Constructive Discussion

As the discussion period comes to a close, we would like to sincerely thank you for the time and care you devoted to reviewing our work, as well as for your detailed follow-up questions.

Your comments encouraged us to examine several aspects of the work more carefully. In response, we added more realistic routing and prediction-based baselines, clarified the distinction between Presage and llm-d, refined our explanation of the dynamic three-dimensional bottleneck model, and expanded the BurstGPT evaluation to cover both light-load and saturated settings. Your questions also helped us better explain the roles of KV-capacity tracking, admission control, fairness, and the assumptions behind the PD-disaggregated setting.

We recognize that several of these points were not presented clearly enough in the original submission, and we are grateful for the opportunity to improve both the analysis and its presentation. Thank you again for the rigorous and constructive discussion. We will carefully incorporate these clarifications and additional results into the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.