acceptodds
Under review as a conference paper at ICLR 2027

SLOPE: Decision-Aligned Marginal Resource Allocation for Co-Located LLM Serving

Abstract

Joint TTFT/E2E violations are terminal outcomes, but an online LLM scheduler must choose variable-duration actions before future arrivals and generation demand are known. Starting from the same latency allowances as the terminal objective, SLOPE jointly designs service valuation and execution-cost learning for online resource allocation. Its central idea is to translate terminal SLO risk into a computable local service comparison: an SLO-normalized potential unifies input progress, phase completion, and decode service, while analysis of the variable-duration service process yields potential removal per unit time as the allocation criterion. SLOPE then identifies marginal resource effects under a common request plan, making candidate comparisons determine both how service value is assessed and which execution-cost differences must be learned. Theoretical analysis establishes a pathwise upper bound on joint violations and a selection-quality guarantee under relative-cost error. Across three model/workload pairs and a 10×–30× sweep of system-normalized TTFT/E2E thresholds, SLOPE improves joint-SLO performance across the evaluated SLO regimes. At the common 20× reference point, it reduces load-averaged joint violations by **15.5%–30.1% relatively** against the best evaluated baseline at each load, with scheduling-call p95 below **2 ms** on Llama 70B. Real-GPU trace analysis and controlled replay further show that planned service can be realized during execution and that cost-aware local choices can yield more jointly successful requests under the same continuation policy, providing empirical mechanism evidence for the allocation criterion.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
LLM serving, prefill-decode scheduling, online resource allocation, decision-aligned cost learning, SLO-aware inference

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_JtBiOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SLOPE, a dynamic prefill-budget allocation framework for co-located LLM serving, aiming to minimize joint SLO violation rates. Starting from a continuous SLO-risk upper bound, the authors derive a service value numerator that unifies decode coverage, continuous prefill progress, and discrete phase-completion jumps. Efficient scheduling is then achieved through a two-stage decoupling design and a lightweight execution latency predictor (ELP).

Critical strengths

Strengths: 1.Exemplary Mechanism Audits: The authors not only close the loop on the theoretical decomposition via real trace residual audits, but also validate the effectiveness of single-step cost-aware selection through controlled single-step counterfactual replay. 2.High Practical Value: By decoupling request ordering from budget sizing and employing a micro-ELP, SLOPE keeps the p95 scheduling overhead under 2 ms on Llama-70B, demonstrating strong practical utility.

Critical weaknesses

none

Questions for authors

Major Concerns & Questions: 1.Extremely Low Proportion of Core ELP Decision Frames: According to the appendix data, out of 4,747 execution frames generated under 1.5$\times$ load, only 329 (less than 7%) correspond to "normal mixed ELP decisions," while the vast majority are handled by pure phases, single candidates, or memory-pressure fallback rules. This raises a critical question: if the sophisticated ELP budget selector is rarely triggered in practice, how much of the end-to-end performance gain stems from the algorithm itself rather than the underlying base queuing rules (SRPT + predicted SJF)? 2.Paradoxical Hyperparameter Insensitivity vs. Theoretical Calibration: The main text emphasizes the precise physical meaning of mapping weights to SLO-normalized parameters (λ\lambda and d0d_0). However, the sensitivity experiments show that varying d0d_0 and λ\lambda drastically by a factor of 9 results in a mean joint violation rate change of no more than 0.63% and 1.34% for the two models. If the system is overwhelmingly insensitive to these relative weights, the complex theoretical calibration in the paper appears superficial, and operators lose a reliable tuning handle when manual intervention is needed. 3.Severe TTFT Degradation in Long-Context Workloads: In the Hunyuan-A13B (SCROLLS) experiments, while SLOPE improves E2E latency, its TTFT violation rate degrades by up to 23.35 percentage points compared to the best baseline. This exposes that the system aggressively starves the prefill of newly arriving requests to protect the E2E deadlines of ongoing long requests. Combined with the second concern (where increasing weights fails to reverse the trend), it is highly questionable whether SLOPE sacrifices an unacceptable level of user first-response experience in long-text interactive scenarios.

6 comments
AuthorsOpenReview ↗

Response to Reviewer JtBi (1/4): ELP Activation and Independent Budget-Selection Gains (Q1)

We sincerely thank you for your careful evaluation of SLOPE, especially for your positive recognition of the real-GPU mechanism audits, controlled single-step replay, and low scheduling overhead. In response to your further concerns about the frequency and performance contribution of ELP activation, the practical role of the theoretical parameters, and the TTFT/E2E trade-off in long-context scenarios, we provide a focused response based on controlled ablations, theoretical analysis, and request-level serving results. We hope to further clarify SLOPE's core mechanism and practical value.

1. ELP Activation Frequency and Contribution of the Core Mechanism (Q1)

We thank you for your attention to ELP activation frequency and its performance contribution. We believe the central question is: When cost-aware budget selection occurs in only a small fraction of execution frames, how can we confirm that the end-to-end performance gains genuinely come from this mechanism rather than primarily from the underlying request ordering and admission rules? We address this from the perspectives of controlled performance attribution and the actual impact of resource decisions.

1.1 Controlled Ablations: Budget Selection Makes an Independent and Substantial Contribution to Terminal Outcomes

On Hunyuan-A13B/SCROLLS under the 20× SLO, we keep the request ordering, admission, and safety fallback mechanisms unchanged and vary only the resource-budget selection method:

Table 1: Joint-SLO violation under budget-selection ablations (%, lower is better).

Budget selection1.0× capacity1.5× capacity
SLOPE50.2560.71
Best tested fixed budget67.5391.13
Progress-only selection68.3289.87

Compared with the best tested fixed budget, SLOPE reduces joint-SLO violation by 17.28 and 30.42 percentage points, respectively; compared with progress-only selection, which maximizes service progress alone, it reduces violations by 18.07 and 29.16 percentage points, respectively. The former demonstrates the value of dynamically adjusting resource investment with the system state. The latter retains the underlying planning and candidate-construction rules and removes only the execution-time-aware selection criterion, yet still leads to substantial degradation. Therefore, SLOPE's gains cannot be attributed solely to underlying queuing rules such as SRPT and predicted SJF; cost-aware budget selection itself makes a substantive contribution.

1.2 Selective Activation: A Few Critical Decisions Can Affect the Entire Serving Trajectory

The key to evaluating ELP's actual contribution is whether cost-aware selection delivers independent terminal gains, rather than how frequently it is triggered. As shown in Table 1, keeping the underlying ordering, admission, and candidate-construction rules unchanged, replacing the ELP-supported (N/\hat{T}) selection with progress-only (N) increases joint-SLO violation on Hunyuan by 18.07 and 29.16 percentage points at 1.0× and 1.5× load, respectively. In addition, ELP's relative-cost learning reduces candidate-selection loss from 18.06% to 13.28%. These results demonstrate that SLOPE's gains cannot be explained solely by the underlying queuing rules; ELP-supported cost-aware decisions make a substantial contribution.

A single mixed-budget decision can change the execution time of the current frame and affect concurrent decode requests. It may also complete prefill, thereby changing the request's subsequent serving phase and resource competition. Its impact can extend beyond the current frame and propagate to the terminal deadlines of multiple requests. This effect is also directly supported by controlled single-step replay: with the subsequent scheduling policy held fixed and only the current budget choice changed, SLOPE achieves more terminal joint-SLO successes in five of the six controlled states and ties in the remaining state.

In summary, selective ELP activation does not mean that its contribution is limited to 6.93% of the execution process. Instead, SLOPE concentrates cost-aware resource optimization on critical states where genuine budget choices exist, avoiding unnecessary online computation while influencing subsequent serving trajectories through these decisions. The controlled ablations and single-step replay further validate the substantial terminal benefits of this mechanism.

AuthorsOpenReview ↗

Response to Reviewer JtBi (2/4): Theoretical Calibration and Parameter Robustness (Q2)

2. Practical Value of Theoretical Calibration and Parameter Robustness (Q2)

We thank you for your thoughtful consideration of the relationship between theoretical calibration and parameter sensitivity. We understand your question: if joint-SLO performance remains stable even after changes to λ and d₀, does theoretical calibration have practical significance? We believe an effective resource-allocation method should have two important properties: first, it should provide an evaluation basis with clear SLO meaning for resource benefits across different serving phases; second, this evaluation principle should remain effective in practice without requiring fine-grained parameter tuning. SLOPE's theory addresses the former, while the experiments further show that its effectiveness does not depend on fine-grained tuning.

2.1 Value of Theoretical Calibration: Defining the Basis for Evaluating Resource Benefits Rather Than Seeking Sensitive Optimal Weights

Starting from terminal joint-SLO risk, we construct an SLO-normalized potential that unifies decode coverage, continuous prefill progress, and discrete phase completion into a comparable local service value. Here, λ controls continuous prefill shaping while preserving the validity of the risk upper bound, whereas d₀ defines the reference-concurrency normalization of decode coverage. In particular, for any λ ≥ 0, the constructed potential remains an upper bound on the original SLO risk. Thus, theoretical calibration provides a family of resource-evaluation criteria with clear SLO interpretations, rather than claiming that one precise parameter setting is the only effective choice.

This evaluation basis is also supported by evidence from actual execution. In real-GPU audits, the backlog-weighted discrepancy between planned and realized service is only 3.34% ± 2.64%, and the positive service discrepancy is zero for all 258 decisions with decode concurrency no lower than d₀. This further demonstrates that SLO-normalized service evaluation not only has a theoretical definition but can also be linked to actual resource execution.

2.2 Parameter Robustness: An Effective Decision Mechanism Without Fine-Grained Weight Tuning

SLOPE selects a candidate budget based on the ratio of its service benefit to execution time. In this discrete candidate optimization, parameter changes affect the actual resource allocation only if they change the relative ranking of candidates. Thus, having a clear theoretical interpretation does not require terminal performance to be highly sensitive to these parameters.

The experiments validate this design from two complementary perspectives:

ExperimentModel and loadComparison conditionChange in joint-SLO violation
Parameter sensitivityHunyuan, 1.0×/1.5×Tested λ and d₀ values span a factor of 9Maximum absolute change 0.63 pp
Parameter sensitivityQwen, 1.0×/1.5×Tested λ and d₀ values span a factor of 9Maximum absolute change 1.34 pp
Core mechanism ablationHunyuan, 1.0×Progress-only vs. SLOPE+18.07 pp
Core mechanism ablationHunyuan, 1.5×Progress-only vs. SLOPE+29.16 pp

Parameter-sensitivity results report the maximum absolute change from the default across five configurations; mechanism-ablation results report the increase in joint-SLO violation relative to SLOPE after removing cost-aware selection.

These two experiments reveal an important advantage of SLOPE: terminal gains depend on sound resource evaluation and cost-aware decisions, but not on fine-tuning internal weights. The parameter sweep shows that a shared default configuration maintains stable performance across the tested models and loads, while the mechanism ablation confirms that this stability does not mean the core budget-selection mechanism is unimportant. For practical deployment, parameter robustness reduces the need for repeated tuning under different operating conditions. Meanwhile, λ and d₀ calibrate internal service values rather than serving as high-sensitivity knobs for directly controlling user-experience preferences; requirements of different applications should be expressed through explicit TTFT/E2E SLO objectives.

AuthorsOpenReview ↗

Response to Reviewer JtBi (3/4): Long-Context TTFT/E2E Trade-offs and Joint-SLO Outcomes (Q3)

3. TTFT/E2E Trade-off in Long-Context Scenarios (Q3)

We fully understand your concern about first-response experience in long-context scenarios. Prefill and decode share GPU resources and iteration time. Increasing prefill investment for new requests may delay the decode of ongoing requests, while excessive protection of decode may postpone TTFT for new requests. This cross-request opportunity cost is precisely the central design problem addressed by SLOPE: whether dynamic coordination of resource competition on a shared GPU can allow more requests to meet both TTFT and E2E deadlines, rather than improving only one metric.

3.1 Joint SLO Objective: Meeting Both TTFT and E2E Rather Than Sacrificing the Former for the Latter

SLOPE's joint-SLO violation counts any request that misses either its TTFT or E2E deadline as a violation. Therefore, even if a request finishes on time, it is still counted as a joint violation if its first response is late. SLOPE's objective does not encourage sacrificing the TTFT of a request in exchange for meeting its E2E deadline; instead, it seeks to increase the fraction of requests that meet both deadlines.

At the resource-decision level, SLOPE jointly considers decode coverage, continuous prefill progress, and the discrete benefit of prefill completion, and incorporates execution-time cost when selecting a budget. Increasing prefill investment may accelerate the first response of a new request but may also lengthen decode waiting time for other requests. Therefore, SLOPE does not assign prefill a fixed low priority; instead, it dynamically evaluates whether additional prefill investment justifies its shared execution-time cost given the current resource competition and service progress.

3.2 Experimental Results: The TTFT/E2E Trade-off Changes with Resource Competition

To clarify the request-level outcomes produced by different policies, we further partition joint violations into three mutually exclusive categories: TTFT-only (late first response only), E2E-only (late full completion only), and Both (both deadlines missed).

Table 2: Request-level violation decomposition on Hunyuan-A13B/SCROLLS under the 20× SLO (%, mean ± standard deviation).

LoadSchedulerTTFT-onlyE2E-onlyBothJoint violation
1.0×SLOPE7.45 ± 1.577.98 ± 12.9934.83 ± 7.7950.25 ± 5.40
FCFS7.11 ± 0.9342.30 ± 9.3929.98 ± 21.7479.39 ± 13.61
1.2×SLOPE5.98 ± 0.960.00 ± 0.0048.42 ± 3.4954.40 ± 2.64
FCFS3.36 ± 0.7830.98 ± 10.2959.74 ± 14.6994.08 ± 4.57
1.5×SLOPE4.57 ± 1.660.00 ± 0.0056.14 ± 3.5560.71 ± 1.89
FCFS2.73 ± 0.8314.92 ± 2.7281.44 ± 4.0799.09 ± 1.11

These results reveal two important observations.

First, SLOPE does not always improve E2E at the expense of TTFT. At 1.2×–1.5× load, SLOPE improves both TTFT and E2E: TTFT violation decreases by 8.70 and 23.46 percentage points, respectively, while E2E violation decreases by 42.30 and 40.22 percentage points. This shows that SLOPE's resource selection does not follow a fixed pattern of suppressing prefill for new requests and prioritizing ongoing decode across loads. Instead, its serving outcomes change with resource competition, reflecting dynamic resource coordination.

Second, SLOPE substantially improves the consistency between timely first responses and timely full completion. At 1.2× and 1.5× load, FCFS has E2E-only violation rates of 30.98% and 14.92%, respectively, meaning that a considerable fraction of requests receive their first response on time but fail to complete generation by their E2E deadline.

Overall, the results indicate that SLOPE's E2E gains do not arise from a fixed suppression of prefill for new requests, but from dynamically balancing the service benefits of different phases against shared execution-time costs. This is SLOPE's core value: under resource competition in co-located serving, it does not focus solely on optimizing an individual latency metric, but instead enables more requests to receive both timely first responses and timely full completion.

Reviewer_jje3OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SLOPE, a principled online scheduler for co-located LLM serving that allocates prefill resources under joint TTFT/E2E deadlines. Starting from a weight-free, pathwise upper bound on joint SLO violations, the authors derive an SLO-normalized local potential-removal rate for variable-duration frames and design a plan-conditioned comparison that isolates true marginal resource effects. The resulting selection rule compares a hybrid service decrement (decode coverage, continuous prefill progress, and discrete prefill completion) against a learned execution-time predictor trained with decision-aligned relative-cost supervision. Experiments across three model/workload pairs and multiple SLO regimes indicate consistent improvements over strong baselines, supported by audits that connect planned service to realized execution and by controlled replays demonstrating terminal benefits from single local choices.

Critical strengths

Significance:

  • Addresses a central and practically important problem: allocating prefill in the presence of ongoing decode under joint SLOs.
  • Offers a general framework that can inform future schedulers by unifying terminal objectives, local decision rules, and prediction objectives.

Technical novelty and innovation

  • Derives the local allocation criterion directly from terminal joint-SLO risk via an SLO-normalized potential and variable-duration frame analysis, rather than postulating a utility/time heuristic.
  • Introduces plan-conditioned marginal identification to disentangle true resource effects from plan-substitution confounds; this is conceptually clean and practically actionable.

Experimental rigor and validation

  • Evaluates on three distinct model/workload pairs, sweeps 10×/20×/30× normalized SLO thresholds and multiple loads, and compares against a broad set of baselines.
  • Provides ablations (fixed budgets, progress-only vs. cost-aware selection, demand prediction) and sensitivity studies for λ and d0.
  • Includes real-GPU trace audits showing planned-to-realized service alignment and controlled replays that isolate the terminal impact of a single allocation decision.
Critical weaknesses

Experimental gaps or methodological issues

  • The output-length predictor is central to planning but its accuracy and error distributions are not reported in the main text; sensitivity to prediction error is not fully explored.
  • The relative-cost training uses a relatively small number of same-state pairs; generalization across workloads/hardware shifts and the need for online adaptation are not deeply evaluated.
  • Controlled replays cover only six states; while positioned as mechanism evidence, broader coverage would strengthen causal claims.

Clarity or presentation issues

  • The paper is dense; some key intuitions (e.g., the impact of λ and d0 across regimes) could be distilled with more concrete, end-to-end examples.
  • Occasional formatting artifacts (likely from PDF extraction) slightly hinder readability of equations/figures.

Missing related work or comparisons

  • While baselines are broad, comparisons to recent learning-based ranking schedulers (beyond SJF/SRPT) could be expanded, for example: https://arxiv.org/abs/2408.15792 https://arxiv.org/abs/2606.18431.
  • The paper focuses on co-located serving; a short empirical or discussion-only comparison to disaggregated strategies (e.g., DistServe-style) would help contextualize scope.
Questions for authors
  1. What specific conditions are required for the pathwise upper bound (Theorem B.10) to hold, and how sensitive is SLOPE in practice to violations of those assumptions (e.g., nonstationarity, bursty arrivals, engine-to-client delays beyond h̄n)?
  2. Could you report output-length predictor accuracy and analyze sensitivity of joint SLO attainment to prediction errors? How does misprediction affect plan quality and the N/T comparison?
  3. How large is the candidate set Ωt\Omega_t in practice, how are budget discretizations chosen, and what is the runtime/quality trade-off as you vary the number and granularity of candidates?
  4. How frequently must ELP be retrained to remain accurate across workload/hardware shifts? Does the contrastive supervision transfer across model versions or GPU types without re-collection?
6 comments
AuthorsOpenReview ↗

Response to Reviewer jje3 (1/5): Theoretical Applicability and Dynamic-Workload Robustness

We sincerely thank you for your careful evaluation of SLOPE, especially for your positive recognition of the importance of the research problem, the theoretical derivations, and the overall design rationale. Below, we provide consolidated responses to your concerns regarding theoretical applicability and robustness under dynamic workloads, prediction reliability and cost-model adaptability, the candidate space and mechanism validation, as well as related-work comparisons and presentation. We hope to further clarify SLOPE's applicability conditions, the effectiveness of its key designs, and its practical research value.

1. Theoretical Applicability and Robustness under Dynamic Workloads (Q1)

We thank you for your attention to the conditions under which Theorem B.10 holds and to SLOPE's robustness in dynamic environments. We recognize the importance of this issue: Online scheduling faces continuously changing request arrivals, resource contention, and actual service times. Thus, it is necessary not only to clarify the applicability conditions of the theoretical results but also to validate the effectiveness of the corresponding resource-decision mechanism under dynamic workloads. We respond from both theoretical and experimental perspectives.

1.1 Theoretical Foundation: A Pathwise Upper Bound on SLO Risk and Its Applicability Conditions

SLOPE adopts a pathwise analysis to establish an evaluation basis related to terminal joint-SLO objectives for dynamic serving trajectories without relying on a particular request-arrival distribution. Specifically, Theorem B.10 considers a finite set of requests starting from an empty system, requires all requests to complete normally before the terminal time (H), and assumes positive TTFT/E2E allowances. Under these conditions, the theorem uses the occupancy identity for request incompletion time and an upper bound on the joint-violation event to relate terminal joint-SLO violation to the area under the SLO-normalized potential along the actual serving trajectory.

This pathwise upper bound requires neither stationary request arrivals nor independence among output lengths. Therefore, although nonstationary arrivals and bursty traffic change the resource-contention states and serving trajectories experienced by the system, they do not invalidate the upper bound merely because the arrival pattern changes. This also enables SLOPE to evaluate resource allocation based on the actual serving state without relying on accurate modeling of the future arrival distribution.

1.2 Experimental Validation: Online Resource Allocation under Different Bursty Workloads

We further evaluate five burst profiles on Llama-3.3-70B/RAGBench, covering stationary arrivals, periodic bursts, and short periods of concentrated arrivals. The experiments maintain the same set of requests and tenant-specific SLOs, primarily varying the arrival-time distribution.

Table 1: Joint-SLO violation under different burst profiles (%, mean ± standard deviation; lower is better).

Burst profileSLOPEBest baselineImprovement
b028.50 ± 2.5232.06 ± 3.473.56 pp
b128.94 ± 3.5432.94 ± 0.774.00 pp
b236.00 ± 3.1836.72 ± 3.760.72 pp
b343.11 ± 5.0348.06 ± 6.684.95 pp
b447.555 ± 2.6753.611 ± 3.046.06 pp

Best baseline denotes the best-performing tested baseline for each arrival pattern.

SLOPE achieves the lowest mean joint-SLO violation across all five profiles, improving upon the best baseline in each profile by 0.72–6.06 percentage points. The pathwise analysis provides a unified basis for risk evaluation, while these results show that online resource selection built on this basis remains effective across different states of dynamic resource contention.

AuthorsOpenReview ↗

Response to Reviewer jje3 (2/5): Prediction Reliability, Decision Quality, and ELP Adaptability

2. Prediction Error, Decision Quality, and ELP Adaptability (W1, W2, Q2, Q4)

We thank you for your attention to output-length prediction reliability and ELP adaptability. We respond from the following two perspectives by combining theoretical analysis, additional experiments, and an assessment of deployment costs.

2.1 Decision Value and Error Robustness of Output-Length Prediction (W1, Q2)

We agree that evaluating a prediction module requires more than examining its average accuracy; it is also important to assess how errors propagate into scheduling decisions and how sensitive terminal joint-SLO performance is to these errors. The measured output-length prediction MAE is 94.58 tokens. We additionally conduct controlled perturbation experiments with three random seeds: Native uses the original predictions; Constant assigns the same predicted length to every request (the setting uses the median output length across validation requests); and Noise-mild and Noise-strong introduce multiplicative random perturbations with standard-deviation parameters σ=0.25\sigma=0.25 and 0.50.5, respectively. All experiments use Hunyuan-A13B/SCROLLS with a 20× SLO.

Table 2: Joint-SLO violation under different output-length prediction settings (%, mean ± standard deviation; lower is better).

Prediction setting1.0× capacity1.2× capacity
Native50.25 ± 5.4054.40 ± 2.64
Constant62.31 ± 4.9065.52 ± 2.80
Noise-mild (σ=0.25\sigma=0.25)53.65 ± 5.4462.90 ± 0.26
Noise-strong (σ=0.5\sigma=0.5)58.45 ± 4.8863.59 ± 2.40
FCFS79.39 ± 13.6194.08 ± 4.57

These results show that output-length prediction has clear decision value, while SLOPE also exhibits some robustness to prediction errors. Compared with Constant, Native reduces joint-SLO violation by 12.06/11.12 pp at the two load levels. Even under strong noise, SLOPE still reduces violation by 20.94/30.49 pp relative to FCFS, although violation increases by 8.20/9.19 pp compared with Native, demonstrating that prediction quality indeed matters.

From a theoretical perspective, output-length errors primarily affect subsequent budget comparisons through request planning. Under a fixed plan, SLOPE evaluates the service benefit NN from the current state, while ELP estimates the execution time T^\hat T. Thus, ELP's relative cost errors directly affect the N/T^N/\hat T ranking among candidates. We will present the prediction-error distribution and its propagation through decisions more clearly in the revised manuscript.

2.2 ELP's Relative-Cost Learning and Cross-Deployment Adaptability (W2, Q4)

Regarding the number of ELP training samples, we would like to clarify that its core design is to learn relative execution costs that directly affect resource selection, rather than merely pursuing accurate absolute execution-time prediction. Consistent with the decision-relevant error analysis in Theorem B.3, a matched ablation shows that Full ELP reduces candidate-selection loss from 18.06% to 13.28% (a relative reduction of 26.5%) compared with the absolute-only counterpart, demonstrating that targeted relative supervision can substantially improve budget-selection quality.

For ELP's deployment adaptability, we distinguish changes in real-time system state from changes in the execution environment. ELP's parameters remain fixed after offline training, but it can update candidate-cost estimates using real-time states such as decode concurrency and prefill backlog, without periodic retraining for normal workload fluctuations. If an environmental change only scales all candidate costs by a common multiplicative factor, their relative ranking remains unchanged; if a new model or GPU changes the relative costs among candidates, recalibration is required. For the Qwen deployment, dual-GPU profiling takes approximately 39.80 minutes, followed by only 10.383 seconds of total CPU training for three seeds each of ELP and the absolute-only counterpart. Therefore, data collection dominates cross-deployment calibration costs, while the lightweight model and relative-supervision mechanism can be directly reused. We do not claim calibration-free transfer across GPUs.

AuthorsOpenReview ↗

Response to Reviewer jje3 (3/5): Candidate Efficiency and Controlled Mechanism Evidence

3. Candidate Space and Local Mechanism Validation (Q3, W3)

We thank you for your suggestions regarding candidate-space size, budget discretization, and the coverage of controlled replays. These issues concern two important aspects of SLOPE: whether it can effectively select resource allocations with limited online overhead, and whether the value of local resource decisions can be validated through actual execution and terminal serving outcomes. We address candidate-selection efficiency and mechanism evidence below.

3.1 Effective Candidate Coverage and Online Overhead (Q3)

SLOPE's candidate design balances executable resource actions and online decision efficiency. As described in Appendix C.2, SLOPE uses 20 nominal budget ratios, ranging from 0.05 to 1.00 in increments of 0.05. In each iteration, these budgets are mapped to actual token quotas under current resource constraints, and candidates producing identical execution plans are merged. We further measure the number of effective candidates actually compared.

Table 3: Number of effective candidates under the default budget grid.

Nominal candidatesMeanP50P95
209.06918

On average, only 9.06 effective candidates need to be compared per iteration. Under the current configuration, the scheduling-call p95 on Llama 70B is 1.06–1.86 ms (1.0×–1.5× capacity), with a corresponding scheduling-time fraction of 0.53%–0.59%, indicating that this budget construction and selection process meets online overhead requirements.

These results show that increasing the number of nominal budget ratios does not necessarily expand the actual decision space. Since budgets are ultimately mapped to discrete token quotas, a finer grid may produce identical executable plans, which do not increase the number of effective candidates after deduplication. Additional choices become available only when new quota combinations arise (e.g., crossing a prefill completion boundary). Therefore, through moderate discretization and execution-plan deduplication, SLOPE covers distinct executable resource actions with a limited candidate set, avoids unnecessary cost-prediction and evaluation overhead, and balances resource-selection capability with online efficiency.

3.2 Controlled Replay: Mechanism Evidence and Causal Interpretation (W3)

The central purpose of controlled replay is to test a specific mechanism question: under the same system state and subsequent scheduling policy, can changing only the current resource decision affect terminal joint-SLO satisfaction? In our experiments, we restore identical system states and future request traces on real GPUs, change only the first-round budget action, and then follow the same SLOPE scheduling policy. The experiments cover the arrival and drain phases across three independent request streams, with two technical repetitions for each action. Across the six controlled states, SLOPE achieves a higher mean number of terminal joint-SLO successes in five states and ties in the remaining one, with an average gain of 0.83 successful requests per state. This shows that a single cost-aware resource decision can change terminal serving outcomes even when the subsequent scheduling policy is identical.

This mechanism-level conclusion is further supported by broader experimental evidence: an audit of 329 real-GPU mixed decisions reports a backlog-weighted discrepancy of 3.34% ± 2.64% between planned and actual service, while end-to-end experiments across three model/workload pairs, multiple SLO thresholds, and different loads further validate the overall benefit of the complete online scheduling policy.

Thus, controlled replay provides direct controlled evidence for the local mechanism studied in the paper: resource selection affects not only immediate service progress but also terminal joint-SLO outcomes. Broader state sampling could further characterize the frequency and magnitude of this effect, while its overall value in the complete online system has already been evaluated through end-to-end experiments spanning different models, workloads, and SLO conditions.

Reviewer_mQTUOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper is about co-located LLM serving with chunked prefill, where each request has to hit both TTFT and E2E deadlines. The SLOPE tunes the prefill chunk size in each iteration. The idea goes like this:

  1. Freeze the request plan from vLLM’s native admission and ordering.
  2. Try a few candidate prefill budgets on that same plan.
  3. Score each one as “SLO progress per unit time.”
  4. Pick the best one.

The time per iteration comes from a learned predictor, trained to get the time ratios between candidates right. On three exps, the paper reports 15–30% fewer joint violations than the best baseline at the 20× SLO setting, with scheduling overhead under 2 ms.

Critical strengths
  1. The core intuition is clean. Small chunks protect decode, but a bigger chunk is worth it when it can finish a request’s prefill.
  2. Training the cost model on same-state time ratios is a sensible design choice, and the ablation shows it helps.
  3. I appreciat author's effort to back the scheduling algorithm with theory instead of only proposing a heuristic.
Critical weaknesses
  1. The contribution is narrow. At the end of the day, the method is adaptive prefill chunk sizing per iteration with a greedy one-step rule. I’m fine with a focused paper, but then the gain on this one knob has to be clearly shown against the right competitors, and right now it isn’t (see 3).

  2. The core idea of the paper could fit in a paragraph, but the paper is still hard to follow. It took me a lot of time to understand the idea behind the jargon everywhere, like “potential removal per unit time,” “plan-conditioned marginal identification,” “hybrid service numerator,” and “contrast-aligned execution-time learning.” Also, Fig. 1 drops some magic numbers (102 vs. 307-token budgets, +53.7% / +45.2% / +5.8%) with little explanation. I’d strongly suggest adding a minimal worked example. For instance:

  • 3–4 requests (one early in a long prefill, one close to finishing prefill, a couple decoding).
  • a simple iteration-time model.
  • the score of each candidate budget. That would show why SLOPE picks a small chunk in one state and a large one in another much better than the current figure does.
  1. The baselines don’t match what the method changes. Most baselines (FCFS, SJF, SRPT, FastServe, EDF, JITServe, QoServe) change which requests run. SLOPE changes chunk size and keeps the native ordering. The most natural competitors are missing: 1) static chunked prefill (Sarathi-style) with a well-tuned chunk size, 2) other adaptive chunking approaches, 3) SLO-aware optimization methods that also target joint TTFT+TPOT, e.g., SLOs-Serve, Scorpio, and similar.

  2. The evaluation regime is overloaded. Load goes from 1.0× to 1.5× capacity, and absolute violation rates are very high. From the ablation table, SLOPE itself seems to violate around 50–60% of requests on Hunyuan. The relative reductions look good, but a system failing about half its requests isn’t a realistic operating point. Results below capacity (e.g., 0.6–0.9×) and absolute numbers in the main text would help a lot. The SLOs are also loose (10–30× of a normalized latency), so it’s worth explaining how that maps to realistic deadlines.

  3. The identification result basically says “if the budget and plan change together, you can’t separate their effects.” True, but it’s just standard confounding.

  4. The characterization of the score assumes additive / separable / symmetric structure, which pretty much hands you the three-term form. Theorem B.3 is a three-line argmax-under-perturbation argument.

  5. The decision space is small. SLOPE only picks among a few fixed budget ratios on a frozen plan. I’m curious whether a finer or larger candidate set, or jointly choosing the plan and the budget, would do better, especially on bursty or shifting workloads. If the cost is low (the score is just a ratio), why keep it so small?

  6. Adaptivity of the cost model is unclear. ELP is trained offline per model/hardware pair, then it frozen. I don’t see device information in the model, so it seems a new GPU needs new profiling, and it’s unclear how robust it is when the deployment/workload drifts.

  7. In summary, I think this paper is an interesting experimental study, but doesn't reach ICLR bar right now. The presentation (#2), novelty (#1), and experiments (#3, #4, #8...) need to be improved.

Questions for authors

See above.

5 comments
AuthorsOpenReview ↗

Response to Reviewer mQTU (1/4): Core Contribution and Presentation Clarity (W1–W2)

We thank the reviewer for the careful and thorough evaluation of our work. We especially thank your recognition of SLOPE’s core design intuition, its cost-learning approach based on same-state execution-time ratios, and its research approach of supporting scheduling design with theoretical analysis. Below, we provide consolidated responses to your concerns, aiming to further clarify SLOPE’s core ideas, design rationale, and research value.

1. SLOPE’s Core Contribution and Presentation Clarity (W1, W2)

1.1 Core Research Problem: From Terminal SLO Objectives to Online Resource Decisions

In co-located LLM serving, there is a structural tension between terminal joint-SLO objectives and online resource decisions: whether a request satisfies both TTFT and E2E depends on its complete service trajectory, while the scheduler must make resource decisions iteratively without knowing future demand. A deeper challenge is that terminal joint-SLO objectives cannot directly assign service value to each local resource investment: the same amount of prefill computation may merely advance unfinished input processing or cross a phase-completion boundary, while also affecting the decode service of other requests through execution time. Consequently, maximizing local service progress and maximizing terminal joint-SLO attainment may lead to fundamentally different resource allocation decisions.

The core problem SLOPE addresses is: How can terminal joint-SLO objectives spanning the entire service process be translated into a local resource decision criterion that supports online comparison at each iteration while accounting for execution-time costs? To address this problem, SLOPE starts from terminal joint-SLO risk and establishes a unified local service value that accounts for decode service, prefill progress, and phase completion. It then compares the service gains per unit time of candidate resource investments under a fixed request plan, while directly aligning execution-cost learning with candidate selection.

Therefore, SLOPE’s core contribution is a coherent online decision-making mechanism that aligns terminal service objectives, local resource value, and execution-cost learning.

1.2 Experimental Evidence and Presentation Improvements

Existing experiments validate the practical significance of this design from three perspectives (Hunyuan-A13B/SCROLLS, 20× SLO):

  • Why is dynamic resource selection necessary? Compared with the best tested fixed-budget configuration (ratio = 1.0), SLOPE reduces joint-SLO violation by 17.28 and 30.42 percentage points at 1.0× and 1.5× loads, respectively. This indicates that its benefits do not merely come from selecting a more suitable fixed prefill budget.
  • Why must execution time be considered? When budget selection maximizes service progress alone while ignoring execution-time costs, joint-SLO violation increases by 18.07 and 29.16 percentage points, respectively. This demonstrates that local service progress alone is insufficient to guide resource allocation; the associated time cost must also be considered.
  • Can local resource choices affect terminal outcomes? In six pre-specified controlled single-step replay states, changing only the current resource selection while keeping the subsequent scheduling policy identical, SLOPE achieves more requests satisfying the terminal joint-SLO in five states and ties in the remaining state. This provides further direct experimental evidence that local resource decisions can affect terminal service outcomes.

Together, these results show that dynamically adjusting resource investment is valuable, but adaptivity alone is insufficient. The key is to evaluate the effective service gains of resource investment according to the current service state and jointly account for their execution-time costs. This is precisely SLOPE’s core design principle.

Regarding W2, we appreciate your suggestion to include a minimal worked example. In the revised manuscript, we will add a simplified example involving requests at different prefill stages and requests undergoing decode. The example will explicitly show the service gains, predicted execution times, and final scores of candidate budgets, intuitively explaining why different budgets are selected in different states. In this way, readers can understand the actual decision-making process before examining the corresponding theoretical derivations.

AuthorsOpenReview ↗

Response to Reviewer mQTU (2/4): Targeted Baselines and Evaluation Beyond Overload (W3–W4)

2. Experimental Comparisons and Operating Regimes (W3, W4)

We thank you for your suggestions regarding baseline selection and experimental load ranges. These concerns are directly related to SLOPE’s core research value: whether dynamic resource allocation guided by terminal joint-SLO objectives can demonstrate advantages over more directly relevant existing methods, and whether these advantages exist only under severe overload.

To address these questions, we conducted additional experiments along two dimensions: resource control mechanisms and degrees of resource contention. All new experiments use the same Hunyuan-A13B/SCROLLS setting and a unified 20× joint-SLO evaluation criterion.

2.1 More Directly Relevant Baselines: Validating the Independent Value of Dynamic Resource Allocation (W3)

We added comparisons with Sarathi-Serve and SCORPIO. These two types of baselines evaluate whether SLOPE’s resource allocation design provides additional benefits from the perspectives of resource execution mechanisms and SLO-aware scheduling strategies, respectively.

Table 1: Joint violation under Hunyuan-A13B/SCROLLS with 20× SLO (%, mean ± standard deviation; lower is better).

LoadSLOPESarathi-ServeSCORPIO
1.0×50.25 ± 5.4086.23 ± 1.6377.32 ± 8.15
1.1×52.51 ± 0.3189.37 ± 1.4884.28 ± 0.98
1.2×54.40 ± 2.6491.30 ± 0.8286.97 ± 0.79
1.3×57.97 ± 1.7692.47 ± 0.5088.89 ± 1.13
1.4×59.04 ± 2.5193.52 ± 0.4290.45 ± 1.03
1.5×60.71 ± 1.8994.07 ± 0.6291.47 ± 0.17

SLOPE achieves the lowest joint-SLO violation across all six loads. Compared with SCORPIO, the better-performing of the newly added baselines, SLOPE achieves an average reduction of 30.75 percentage points.

2.2 Non-overloaded Loads: Revealing the Value of Dynamic Resource Allocation Under Different Contention Conditions (W4)

We further extended the experiments to 0.6×–0.9× capacity to directly examine whether SLOPE’s benefits depend on severe overload, while reporting the absolute joint-SLO violation rates of each method.

Table 2: Joint violation under low loads on Hunyuan-A13B/SCROLLS with 20× SLO (%, mean ± standard deviation; lower is better).

LoadSLOPELTR-SJFQoServeSarathi-ServeSCORPIO
0.6×4.26 ± 4.062.76 ± 2.592.25 ± 2.2110.63 ± 8.902.76 ± 2.59
0.7×11.05 ± 2.4114.61 ± 6.1315.85 ± 7.4242.20 ± 27.8211.94 ± 7.00
0.8×22.03 ± 2.7539.03 ± 16.7438.22 ± 13.1267.19 ± 15.9046.04 ± 16.68
0.9×25.50 ± 3.2659.88 ± 16.1162.16 ± 18.2580.94 ± 5.2765.29 ± 17.51

These results show that even when the request arrival rate is below system capacity, resource contention between prefill and decode can still substantially affect joint-SLO attainment, and the advantages of dynamic resource allocation become more pronounced as contention increases.

Together, the two sets of additional experiments show that SLOPE’s benefits can be demonstrated against more directly relevant resource execution and SLO-aware scheduling methods, and are not limited to severely overloaded scenarios. This further supports its core research motivation: dynamically evaluating the effective service gains and execution-time costs of resource investment according to the actual service state is important for improving terminal joint-SLO attainment.

AuthorsOpenReview ↗

Response to Reviewer mQTU (3/4): Theoretical Foundations of Decision-Aligned Resource Allocation (W5–W6)

3. Theoretical Foundations, Decision Space, and Practical Deployment (W5, W6, W7, W8)

We thank you for your thoughtful comments on the theoretical results, candidate decision space, and adaptability of the cost model. We believe that SLOPE’s theoretical value lies not only in the mathematical form of individual propositions, but also in translating terminal joint-SLO objectives into a set of online resource allocation principles that can guide algorithm design and be verified through real execution.

3.1 Theoretical Design: From Mathematical Analysis to Verifiable Decision Principles (W5, W6)

We understand your concerns regarding the mathematical complexity of the theoretical results. The related analyses in SLOPE jointly establish three interconnected design principles.

First, how should local service value be defined in relation to terminal SLO objectives?

Starting from request-level TTFT/E2E joint violation events, SLOPE establishes a pathwise SLO-risk bound. Through an SLO-normalized potential, it provides a unified measure of decode service, continuous prefill progress, and discrete phase completion. Combined with the variable-duration execution process, this analysis derives a service progress criterion per unit time.

The three-term representation in Proposition B.2 relies on separability and additivity assumptions, while the subsequent SLO-normalized analysis further assigns explicit service interpretations to the weights: λ corresponds to prefill shaping that preserves the risk bound, while d₀ represents a reference decode concurrency, establishing an interpretable connection between local service value and terminal TTFT/E2E objectives.

Second, the practical significance of Proposition B.1 for online resource decisions (W5)

Proposition B.1 reveals a limitation that directly affects online resource decisions: when the prefill budget and request plan change simultaneously, even accurate knowledge of the service gains and execution times of two candidates is insufficient to determine whether additional prefill resources themselves are worth the investment. Identical observed endpoints can correspond to different direct budget effects; therefore, improving prediction accuracy alone cannot resolve this issue.

This analysis directly guides SLOPE’s layered design: it compares the service gains of different budgets under a common request plan at each iteration and trains ELP to learn the corresponding relative execution times. Consequently, the resource selection criterion and cost learning target the same budget intervention, giving the comparison of candidate benefits and time costs a clear control interpretation.

Third, how can the prediction model directly serve resource decisions?

Theorem B.3 identifies the key error affecting candidate selection quality: relative execution-time prediction errors within the same state. Based on this result, ELP introduces same-state relative-cost supervision to directly learn the execution-time differences that determine budget selection. The theoretical guarantee applies to planned service rates within the candidate set and does not concern global terminal optimality.

Experiments further validate the above design: relative-cost training reduces candidate selection loss from 18.06% to 13.28%; across five weight configurations, changes in joint-SLO violation relative to the default configuration do not exceed 0.63 and 1.34 percentage points on Hunyuan and Qwen, respectively; and an audit of 329 real-GPU mixed decisions shows a backlog-weighted discrepancy of 3.34% ± 2.64% between planned and realized service.

Together, these analyses establish a design chain from terminal SLO risk and local service value to controlled resource comparisons and decision-relevant cost learning, clarifying what online resource allocation should optimize, compare, and learn.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.