- Rating
- 4(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SLOPE, a dynamic prefill-budget allocation framework for co-located LLM serving, aiming to minimize joint SLO violation rates. Starting from a continuous SLO-risk upper bound, the authors derive a service value numerator that unifies decode coverage, continuous prefill progress, and discrete phase-completion jumps. Efficient scheduling is then achieved through a two-stage decoupling design and a lightweight execution latency predictor (ELP).
Critical strengths
Strengths: 1.Exemplary Mechanism Audits: The authors not only close the loop on the theoretical decomposition via real trace residual audits, but also validate the effectiveness of single-step cost-aware selection through controlled single-step counterfactual replay. 2.High Practical Value: By decoupling request ordering from budget sizing and employing a micro-ELP, SLOPE keeps the p95 scheduling overhead under 2 ms on Llama-70B, demonstrating strong practical utility.
Critical weaknesses
none
Questions for authors
Major Concerns & Questions: 1.Extremely Low Proportion of Core ELP Decision Frames: According to the appendix data, out of 4,747 execution frames generated under 1.5$\times$ load, only 329 (less than 7%) correspond to "normal mixed ELP decisions," while the vast majority are handled by pure phases, single candidates, or memory-pressure fallback rules. This raises a critical question: if the sophisticated ELP budget selector is rarely triggered in practice, how much of the end-to-end performance gain stems from the algorithm itself rather than the underlying base queuing rules (SRPT + predicted SJF)? 2.Paradoxical Hyperparameter Insensitivity vs. Theoretical Calibration: The main text emphasizes the precise physical meaning of mapping weights to SLO-normalized parameters ( and ). However, the sensitivity experiments show that varying and drastically by a factor of 9 results in a mean joint violation rate change of no more than 0.63% and 1.34% for the two models. If the system is overwhelmingly insensitive to these relative weights, the complex theoretical calibration in the paper appears superficial, and operators lose a reliable tuning handle when manual intervention is needed. 3.Severe TTFT Degradation in Long-Context Workloads: In the Hunyuan-A13B (SCROLLS) experiments, while SLOPE improves E2E latency, its TTFT violation rate degrades by up to 23.35 percentage points compared to the best baseline. This exposes that the system aggressively starves the prefill of newly arriving requests to protect the E2E deadlines of ongoing long requests. Combined with the second concern (where increasing weights fails to reverse the trend), it is highly questionable whether SLOPE sacrifices an unacceptable level of user first-response experience in long-text interactive scenarios.
6 comments
Response to Reviewer JtBi (2/4): Theoretical Calibration and Parameter Robustness (Q2)
2. Practical Value of Theoretical Calibration and Parameter Robustness (Q2)
We thank you for your thoughtful consideration of the relationship between theoretical calibration and parameter sensitivity. We understand your question: if joint-SLO performance remains stable even after changes to λ and d₀, does theoretical calibration have practical significance? We believe an effective resource-allocation method should have two important properties: first, it should provide an evaluation basis with clear SLO meaning for resource benefits across different serving phases; second, this evaluation principle should remain effective in practice without requiring fine-grained parameter tuning. SLOPE's theory addresses the former, while the experiments further show that its effectiveness does not depend on fine-grained tuning.
2.1 Value of Theoretical Calibration: Defining the Basis for Evaluating Resource Benefits Rather Than Seeking Sensitive Optimal Weights
Starting from terminal joint-SLO risk, we construct an SLO-normalized potential that unifies decode coverage, continuous prefill progress, and discrete phase completion into a comparable local service value. Here, λ controls continuous prefill shaping while preserving the validity of the risk upper bound, whereas d₀ defines the reference-concurrency normalization of decode coverage. In particular, for any λ ≥ 0, the constructed potential remains an upper bound on the original SLO risk. Thus, theoretical calibration provides a family of resource-evaluation criteria with clear SLO interpretations, rather than claiming that one precise parameter setting is the only effective choice.
This evaluation basis is also supported by evidence from actual execution. In real-GPU audits, the backlog-weighted discrepancy between planned and realized service is only 3.34% ± 2.64%, and the positive service discrepancy is zero for all 258 decisions with decode concurrency no lower than d₀. This further demonstrates that SLO-normalized service evaluation not only has a theoretical definition but can also be linked to actual resource execution.
2.2 Parameter Robustness: An Effective Decision Mechanism Without Fine-Grained Weight Tuning
SLOPE selects a candidate budget based on the ratio of its service benefit to execution time. In this discrete candidate optimization, parameter changes affect the actual resource allocation only if they change the relative ranking of candidates. Thus, having a clear theoretical interpretation does not require terminal performance to be highly sensitive to these parameters.
The experiments validate this design from two complementary perspectives:
| Experiment | Model and load | Comparison condition | Change in joint-SLO violation |
|---|---|---|---|
| Parameter sensitivity | Hunyuan, 1.0×/1.5× | Tested λ and d₀ values span a factor of 9 | Maximum absolute change 0.63 pp |
| Parameter sensitivity | Qwen, 1.0×/1.5× | Tested λ and d₀ values span a factor of 9 | Maximum absolute change 1.34 pp |
| Core mechanism ablation | Hunyuan, 1.0× | Progress-only vs. SLOPE | +18.07 pp |
| Core mechanism ablation | Hunyuan, 1.5× | Progress-only vs. SLOPE | +29.16 pp |
Parameter-sensitivity results report the maximum absolute change from the default across five configurations; mechanism-ablation results report the increase in joint-SLO violation relative to SLOPE after removing cost-aware selection.
These two experiments reveal an important advantage of SLOPE: terminal gains depend on sound resource evaluation and cost-aware decisions, but not on fine-tuning internal weights. The parameter sweep shows that a shared default configuration maintains stable performance across the tested models and loads, while the mechanism ablation confirms that this stability does not mean the core budget-selection mechanism is unimportant. For practical deployment, parameter robustness reduces the need for repeated tuning under different operating conditions. Meanwhile, λ and d₀ calibrate internal service values rather than serving as high-sensitivity knobs for directly controlling user-experience preferences; requirements of different applications should be expressed through explicit TTFT/E2E SLO objectives.
Response to Reviewer JtBi (3/4): Long-Context TTFT/E2E Trade-offs and Joint-SLO Outcomes (Q3)
3. TTFT/E2E Trade-off in Long-Context Scenarios (Q3)
We fully understand your concern about first-response experience in long-context scenarios. Prefill and decode share GPU resources and iteration time. Increasing prefill investment for new requests may delay the decode of ongoing requests, while excessive protection of decode may postpone TTFT for new requests. This cross-request opportunity cost is precisely the central design problem addressed by SLOPE: whether dynamic coordination of resource competition on a shared GPU can allow more requests to meet both TTFT and E2E deadlines, rather than improving only one metric.
3.1 Joint SLO Objective: Meeting Both TTFT and E2E Rather Than Sacrificing the Former for the Latter
SLOPE's joint-SLO violation counts any request that misses either its TTFT or E2E deadline as a violation. Therefore, even if a request finishes on time, it is still counted as a joint violation if its first response is late. SLOPE's objective does not encourage sacrificing the TTFT of a request in exchange for meeting its E2E deadline; instead, it seeks to increase the fraction of requests that meet both deadlines.
At the resource-decision level, SLOPE jointly considers decode coverage, continuous prefill progress, and the discrete benefit of prefill completion, and incorporates execution-time cost when selecting a budget. Increasing prefill investment may accelerate the first response of a new request but may also lengthen decode waiting time for other requests. Therefore, SLOPE does not assign prefill a fixed low priority; instead, it dynamically evaluates whether additional prefill investment justifies its shared execution-time cost given the current resource competition and service progress.
3.2 Experimental Results: The TTFT/E2E Trade-off Changes with Resource Competition
To clarify the request-level outcomes produced by different policies, we further partition joint violations into three mutually exclusive categories: TTFT-only (late first response only), E2E-only (late full completion only), and Both (both deadlines missed).
Table 2: Request-level violation decomposition on Hunyuan-A13B/SCROLLS under the 20× SLO (%, mean ± standard deviation).
| Load | Scheduler | TTFT-only | E2E-only | Both | Joint violation |
|---|---|---|---|---|---|
| 1.0× | SLOPE | 7.45 ± 1.57 | 7.98 ± 12.99 | 34.83 ± 7.79 | 50.25 ± 5.40 |
| FCFS | 7.11 ± 0.93 | 42.30 ± 9.39 | 29.98 ± 21.74 | 79.39 ± 13.61 | |
| 1.2× | SLOPE | 5.98 ± 0.96 | 0.00 ± 0.00 | 48.42 ± 3.49 | 54.40 ± 2.64 |
| FCFS | 3.36 ± 0.78 | 30.98 ± 10.29 | 59.74 ± 14.69 | 94.08 ± 4.57 | |
| 1.5× | SLOPE | 4.57 ± 1.66 | 0.00 ± 0.00 | 56.14 ± 3.55 | 60.71 ± 1.89 |
| FCFS | 2.73 ± 0.83 | 14.92 ± 2.72 | 81.44 ± 4.07 | 99.09 ± 1.11 |
These results reveal two important observations.
First, SLOPE does not always improve E2E at the expense of TTFT. At 1.2×–1.5× load, SLOPE improves both TTFT and E2E: TTFT violation decreases by 8.70 and 23.46 percentage points, respectively, while E2E violation decreases by 42.30 and 40.22 percentage points. This shows that SLOPE's resource selection does not follow a fixed pattern of suppressing prefill for new requests and prioritizing ongoing decode across loads. Instead, its serving outcomes change with resource competition, reflecting dynamic resource coordination.
Second, SLOPE substantially improves the consistency between timely first responses and timely full completion. At 1.2× and 1.5× load, FCFS has E2E-only violation rates of 30.98% and 14.92%, respectively, meaning that a considerable fraction of requests receive their first response on time but fail to complete generation by their E2E deadline.
Overall, the results indicate that SLOPE's E2E gains do not arise from a fixed suppression of prefill for new requests, but from dynamically balancing the service benefits of different phases against shared execution-time costs. This is SLOPE's core value: under resource competition in co-located serving, it does not focus solely on optimizing an individual latency metric, but instead enables more requests to receive both timely first responses and timely full completion.
Response to Reviewer JtBi (1/4): ELP Activation and Independent Budget-Selection Gains (Q1)
We sincerely thank you for your careful evaluation of SLOPE, especially for your positive recognition of the real-GPU mechanism audits, controlled single-step replay, and low scheduling overhead. In response to your further concerns about the frequency and performance contribution of ELP activation, the practical role of the theoretical parameters, and the TTFT/E2E trade-off in long-context scenarios, we provide a focused response based on controlled ablations, theoretical analysis, and request-level serving results. We hope to further clarify SLOPE's core mechanism and practical value.
1. ELP Activation Frequency and Contribution of the Core Mechanism (Q1)
We thank you for your attention to ELP activation frequency and its performance contribution. We believe the central question is: When cost-aware budget selection occurs in only a small fraction of execution frames, how can we confirm that the end-to-end performance gains genuinely come from this mechanism rather than primarily from the underlying request ordering and admission rules? We address this from the perspectives of controlled performance attribution and the actual impact of resource decisions.
1.1 Controlled Ablations: Budget Selection Makes an Independent and Substantial Contribution to Terminal Outcomes
On Hunyuan-A13B/SCROLLS under the 20× SLO, we keep the request ordering, admission, and safety fallback mechanisms unchanged and vary only the resource-budget selection method:
Table 1: Joint-SLO violation under budget-selection ablations (%, lower is better).
Compared with the best tested fixed budget, SLOPE reduces joint-SLO violation by 17.28 and 30.42 percentage points, respectively; compared with progress-only selection, which maximizes service progress alone, it reduces violations by 18.07 and 29.16 percentage points, respectively. The former demonstrates the value of dynamically adjusting resource investment with the system state. The latter retains the underlying planning and candidate-construction rules and removes only the execution-time-aware selection criterion, yet still leads to substantial degradation. Therefore, SLOPE's gains cannot be attributed solely to underlying queuing rules such as SRPT and predicted SJF; cost-aware budget selection itself makes a substantive contribution.
1.2 Selective Activation: A Few Critical Decisions Can Affect the Entire Serving Trajectory
The key to evaluating ELP's actual contribution is whether cost-aware selection delivers independent terminal gains, rather than how frequently it is triggered. As shown in Table 1, keeping the underlying ordering, admission, and candidate-construction rules unchanged, replacing the ELP-supported (N/\hat{T}) selection with progress-only (N) increases joint-SLO violation on Hunyuan by 18.07 and 29.16 percentage points at 1.0× and 1.5× load, respectively. In addition, ELP's relative-cost learning reduces candidate-selection loss from 18.06% to 13.28%. These results demonstrate that SLOPE's gains cannot be explained solely by the underlying queuing rules; ELP-supported cost-aware decisions make a substantial contribution.
A single mixed-budget decision can change the execution time of the current frame and affect concurrent decode requests. It may also complete prefill, thereby changing the request's subsequent serving phase and resource competition. Its impact can extend beyond the current frame and propagate to the terminal deadlines of multiple requests. This effect is also directly supported by controlled single-step replay: with the subsequent scheduling policy held fixed and only the current budget choice changed, SLOPE achieves more terminal joint-SLO successes in five of the six controlled states and ties in the remaining state.
In summary, selective ELP activation does not mean that its contribution is limited to 6.93% of the execution process. Instead, SLOPE concentrates cost-aware resource optimization on critical states where genuine budget choices exist, avoiding unnecessary online computation while influencing subsequent serving trajectories through these decisions. The controlled ablations and single-step replay further validate the substantial terminal benefits of this mechanism.