- Rating
- 2(−1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies performance coupling between prefill and decode despite their placement on separate GPU pools. PHASOR uses Virtual Freeness to estimate prefill pressure from execution time and deadline slack, and decode admission capacity from available KV blocks and projected demand. It coordinates request routing, KV migration, and GPU role conversion to address shortages at different scopes. Experiments on an eight-H100 node cover four model configurations and three workload regimes. The paper reports 6.7–29.0% higher SLO-goodput and 22.0–48.6% lower P99 TTFT than the strongest evaluated baseline in each setting, with separate comparisons of signals and intervention policies.
Critical strengths
-
The design connects useful load information to interventions at different scopes. Token counts and memory occupancy alone can miss differences in queued computation and admission capacity. Combining deadline-aware routing, targeted KV movement, and role conversion gives the controller several ways to address the resulting shortages. This is a meaningful systems contribution even without a guarantee that the signals capture every source of latency.
-
Controlled experiments support the value of coordination. Table 2 holds controllers and actuators fixed when comparing signals, while Figure 3(b) holds the signal and actuators fixed when comparing intervention policies. PHASOR improves goodput while reducing KV movement, accumulated drain time, and conversion counts relative to Global-first and Independent. The paper also acknowledges a decode-heavy case where Bullet achieves better TPOT and end-to-end P99 latency. These results support a useful, appropriately scoped practical contribution.
Critical weaknesses
-
The signal comparison needs more precise specification. RichPhase uses the same underlying information, but its risk transformations, calibration procedure, and cross-phase decision rule are insufficiently described. This makes it difficult to reproduce the comparison and attribute the gain specifically to the common freeness scale. The decode score models KV admission capacity rather than TPOT, as the authors acknowledge; its operating assumptions when execution or bandwidth limits token latency also need clarification.
-
The measurement windows and response costs need clearer interpretation. For Qwen3.5-27B Burst, Table 1 reports goodput of 22.9 req/s, while Table 2 reports 9.70 req/s. Peak-cohort versus whole-trace reporting is a plausible explanation, but the mapping should be explicit. The aggregate migration and drain statistics are useful, yet do not show the latency distribution of individual role conversions. Since conversion waits for the donor's existing requests to finish, this distinction matters for judging responsiveness to short bursts or long remaining outputs.
Questions for authors
-
Clarification: Please provide RichPhase's risk formulas, calibration objective, tuning ranges, and final decision rule. When both methods use the same information, which decisions differ because of the freeness normalization? Under what conditions is this comparison intended to remain useful when decode latency is limited by execution or bandwidth rather than KV capacity?
-
Measurement clarification: Do the Burst attribution tables use the full trace while Table 1 uses the peak cohort? Please specify the windows and request cohorts, and explain the trace, seed, or configuration differences behind the Prefill-heavy Full results in Tables 1 and 2.
-
Targeted supporting evidence: Beyond the reported aggregate drain time and conversion counts, can the authors provide a representative per-conversion latency breakdown and distribution, including draining, switching, and activation? How does the controller handle a donor whose remaining requests take longer to drain than the imbalance persists? Existing measurements or an explicit operating limitation would help assess this concern.
3 comments
Response to Reviewer ubzE (2/3)
Continuing our response to Reviewer ubzE, we address Q2 below and Q3 in (3/3).
Q2. Measurement windows, request cohorts, and Table 1 vs. Table 2 reconciliation.
For Qwen3.5-27B, the two tables align precisely:
- Burst (22.9 vs. 9.70 req/s): Yes—Table 1 evaluates the peak-burst cohort (requests arriving during the 90–120 s high-load shock at 26.3 req/s mean offered load) of the 180-s Burst replay, yielding 13.75 s E2E P99 and 22.9 req/s goodput. By contrast, Tables 2, 7, and 8 report the full 0–180 s trace cohort (7.7->26.3->7.7 req/s). Since the nominal whole-trace arrival rate is req/s, Table 2's 9.70 req/s aligns exactly with Table 7's 89.8% joint SLO attainment ( req/s).
- Prefill-heavy and Decode-heavy Full results: In Table 1 vs. Table 2, Qwen3.5-27B reports 7.56 s / 17.3 req/s vs. 7.00±0.22 s / 16.7±0.2 req/s (Prefill-heavy) and 84.33 s / 3.2 req/s vs. 94.50±2.60 s / 3.18±0.06 req/s (Decode-heavy). In Decode-heavy, 3.18 rounds to 3.2 req/s. Across workloads, Table 1 (the multi-system benchmark) and the attribution suite (Table 2, extended by Tables 7–8) are separate three-run measurement campaigns rather than shared runs—as Table 2's caption explicitly notes, only the two Full rows within Table 2 share the same runs. Within each campaign, all compared variants replay identical request sequences and arrival timestamps (yielding tight intra-table SDs such as ±0.2 req/s and strictly paired comparisons), whereas independent three-run trace realizations between the two campaigns reflect heavy-tailed ShareGPT length variation, shifting tail E2E P99 (notably in long-output Decode-heavy) and saturation goodput slightly without affecting any within-table comparison.
| Qwen3.5-27B Workload | Table 1 Cohort | Table 1 (P99 / GP) | Table 2 Cohort | Table 2 Full (P99 / GP) | Reconciliation |
|---|---|---|---|---|---|
| Burst | 90–120 s peak (26.3 req/s) | 13.75 s / 22.9 req/s | 0–180 s full (10.8 req/s) | 15.40±0.55 s / 9.70±0.20 req/s | Peak vs. full (89.8% att.) |
| Prefill-heavy | Benchmark 3-run mean | 7.56 s / 17.3 req/s | Attribution 3-run mean±SD | 7.00±0.22 s / 16.7±0.2 req/s | Separate 3-run realizations |
| Decode-heavy | Benchmark 3-run mean | 84.33 s / 3.2 req/s | Attribution 3-run mean±SD | 94.50±2.60 s / 3.18±0.06 req/s | 3.18->3.2; separate runs |
We will label these cohorts, windows, and campaign boundaries directly in the captions and setup of Tables 1, 2, 7, and 8.
Response to Reviewer ubzE (3/3)
Q3. Per-conversion latency breakdown, end-to-end cost accounting, and slow-draining donors.
1. Per-conversion latency breakdown across conversion directions: Equation 9 decomposes Macro's conversion response into persistence, draining, switching, and activation: . Phasor uses 50-ms epochs with 200-ms persistence (), a 1-s cooldown, and 20.0 ms readiness. In the 180-s Burst replay, Phasor executes 4.0±0.0 conversions with 6.1±0.5 GPU-s aggregate drain time (~1.53 s average drain per conversion) while achieving 5.0±0.4 s post-shock recovery.
To isolate P-to-D vs. D-to-P components directly, we measured conversion stages in an additional two-node (16-H100) 300-s high-burst replay (6,480 requests after 30 s warm-up):
| Direction (16-H100) | Mean | Mean | Mean | Mean Stop-Admission-to-First-Token | |
|---|---|---|---|---|---|
| P-to-D (expand D) | 200 ms | 383.0 ms | 25.9 ms | 20.0 ms | 428.9 ms |
| D-to-P (expand P) | 200 ms | 3,506.6 ms | 108.8 ms | 20.0 ms | 3,635.4 ms |
The stop-admission-to-first-token clock starts when admission stops and ends at the first served token in the new role, whereas Equation 9 spans persistence to activation. While our current instrumentation captures per-direction component means and multi-run SDs rather than per-event quantiles (which we will add in the revision), the ~9x directional gap (383.0 ms vs. 3,506.6 ms) reflects three properties:
- Workload–direction alignment: P-to-D drains in 383.0 ms (428.9 ms to first token) because prefill chunks are short. Under Decode-heavy (128 input / 2,048 output tokens), prefill is lightly loaded and decode is KV-constrained, so Macro expands decode via fast P-to-D conversion rather than draining long decode sequences.
- Donor selection under D-to-P: When prefill bursts trigger D-to-P (3,506.6 ms mean drain), Algorithm 1 limits tail drain by picking the eligible donor with maximum ()—preferring the decode worker with minimal active KV occupancy and projected demand.
- Drain costs validate minimum-sufficient intervention: All serving results include actual draining and switching costs. In Figure 3(b), Global-first and Independent share the identical draining and conversion implementation as Phasor; precisely because D-to-P draining takes seconds, Phasor's sufficient-repair coordination cuts conversions by 50.0%/33.3%, cuts aggregate drain time by 52.3%/36.5%, and improves goodput by 13.5%/9.0% (9.70 vs. 8.55/8.90 req/s).
2. Handling donors whose drain time exceeds imbalance duration (Mechanism & Explicit Operating Limitation): To avoid aborting in-flight generation, Phasor's Mixed state is non-preemptive: donor stops admitting new work and completes resident sequences cleanly, while Algorithm 1 converts at most one worker before re-evaluating and enforces a 1-s cooldown (). In our 30-s Burst shock (5.0 s recovery), P-to-D finishes in ~0.38 s, well inside the imbalance window. However, we explicitly recognize an operating limitation: when a transient spike subsides faster than the remaining drain time of every eligible decode donor (most relevant for D-to-P under sub-second bursts while decode workers hold long active outputs), Macro cannot deliver capacity before the spike ends, leaving ultra-short transients to Micro and Meso. Supporting abortable Mixed-state rollback or Meso-assisted straggler evacuation is a direct extension we will document in Limitations.
We hope these specifications, cohort reconciliations, and operating boundaries resolve your remaining concerns and support a stronger overall assessment of our paper.
Response to Reviewer ubzE (1/3)
We thank Reviewer ubzE for the constructive review and for recognizing our controlled ablation design and candid scoping of Phasor's operating regime (including the decode-heavy case where Bullet achieves lower TPOT and E2E P99). We address Q1 below, Q2 in (2/3), and Q3 in (3/3).
Q1. RichPhase specification, normalization impact, and decode execution/bandwidth boundary.
1. RichPhase formulation, calibration, and decision rule: RichPhase isolates the benefit of unifying heterogeneous P/D pressure onto a common feasibility scale (Φ=0) from richer phase-local estimation:
2. Why gains stem from normalization, and which decisions differ: Within a single phase (e.g., Micro routing), monotonic transforms largely preserve intra-phase worker ranking. Normalization changes decisions at cross-phase and escalation boundaries:
TRYMESO) to Macro's trigger, RichPhase runs 6.0±0.8 conversions over the 180-s Burst vs. 4.0±0.0 under Full Phi (~50% more from mistimed triggers), trailing by 4.3–11.0% in goodput and 7.0–10.8% in E2E P99 (Table 2). Combined with our sensitivity sweeps (0.92–1.01 normalized capacity across perturbations), gains stem from anchoring Φ=0 to physical feasibility, explaining why the gap is smallest in Decode-heavy (3.05 vs. 3.18 req/s) and largest in Burst (8.74 vs. 9.70 req/s).3. Usefulness under execution- or bandwidth-limited decode: While ΦS models KV admission capacity rather than per-step TPOT, comparing Full Phi vs. RichPhase—and Phasor's decode branch—remains useful under execution- or bandwidth-limited decode in two regimes:
KV-only signals stop differentiating decode pressure only when workers uniformly saturate compute/bandwidth well below KV capacity, where Virtual Freeness extends via Φdecode=min(ΦKV,ΦTPOT) using batch step-time slack to the TPOT SLO. We will document these conditions in the revised paper.