acceptodds
Under review as a conference paper at ICLR 2027

Resource Isolation Is Not Performance Isolation: Virtual Freeness for Disaggregated LLM Serving

Abstract

Phase-disaggregated LLM serving isolates prefill and decode on independently provisioned GPU pools. But does resource isolation provide performance isolation? Production-grounded H100 analyses reveal four findings. First, prefill imbalance can delay decode-ready work and starve separately provisioned decode GPUs, propagating overload across the pipeline. Second, phase-local metrics can miss each phase's limiting resource, creating false headroom that amplifies imbalance. Third, even bottleneck-matched signals omit state needed to decide whether workers can accept additional work within the SLO. Finally, accurate load information alone is insufficient: recovery requires an intervention whose scope matches the imbalance without unnecessary execution cost. We introduce Virtual Freeness, which captures each phase's decision-critical state and expresses remaining worker capacity on a common SLO-feasibility scale. We also propose the minimum-sufficient-intervention principle, which selects the least disruptive action sufficient to restore capacity and escalates only when necessary. Phasor implements these ideas through three coordinated controls: Micro routes requests by deadline-aware prefill risk; Meso restores worker-local KV allocatability through targeted migration; Macro converts donor GPUs under persistent pool-wide imbalance. We evaluate Phasor against five serving baselines on an 8H100 platform, using dense and MoE models from 8B to 110B. Across 12 model-workload combinations, Phasor achieves the highest SLO-goodput and lowest P99 TTFT. Relative to the strongest prior system in each setting, it improves SLO-goodput by 6.7–29.0% and reduces P99 TTFT by 22.0–48.6%. It also achieves the lowest E2E P99 and TPOT in 11 of 12 settings. Holding signals and actuators fixed, its minimum-sufficient intervention policy improves SLO-goodput by 9.0–13.5% and reduces KV movement, GPU drain time, and role conversions by 21.3–52.3% relative to global-first and independent policies. Total runtime overhead is 2.8%.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
LLM serving, prefill-decode disaggregation, SLO-aware scheduling, resource allocation, KV-cache management, multi-scale control

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ubzEOpenReview ↗
Rating
2(−1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies performance coupling between prefill and decode despite their placement on separate GPU pools. PHASOR uses Virtual Freeness to estimate prefill pressure from execution time and deadline slack, and decode admission capacity from available KV blocks and projected demand. It coordinates request routing, KV migration, and GPU role conversion to address shortages at different scopes. Experiments on an eight-H100 node cover four model configurations and three workload regimes. The paper reports 6.7–29.0% higher SLO-goodput and 22.0–48.6% lower P99 TTFT than the strongest evaluated baseline in each setting, with separate comparisons of signals and intervention policies.

Critical strengths
  1. The design connects useful load information to interventions at different scopes. Token counts and memory occupancy alone can miss differences in queued computation and admission capacity. Combining deadline-aware routing, targeted KV movement, and role conversion gives the controller several ways to address the resulting shortages. This is a meaningful systems contribution even without a guarantee that the signals capture every source of latency.

  2. Controlled experiments support the value of coordination. Table 2 holds controllers and actuators fixed when comparing signals, while Figure 3(b) holds the signal and actuators fixed when comparing intervention policies. PHASOR improves goodput while reducing KV movement, accumulated drain time, and conversion counts relative to Global-first and Independent. The paper also acknowledges a decode-heavy case where Bullet achieves better TPOT and end-to-end P99 latency. These results support a useful, appropriately scoped practical contribution.

Critical weaknesses
  1. The signal comparison needs more precise specification. RichPhase uses the same underlying information, but its risk transformations, calibration procedure, and cross-phase decision rule are insufficiently described. This makes it difficult to reproduce the comparison and attribute the gain specifically to the common freeness scale. The decode score models KV admission capacity rather than TPOT, as the authors acknowledge; its operating assumptions when execution or bandwidth limits token latency also need clarification.

  2. The measurement windows and response costs need clearer interpretation. For Qwen3.5-27B Burst, Table 1 reports goodput of 22.9 req/s, while Table 2 reports 9.70 req/s. Peak-cohort versus whole-trace reporting is a plausible explanation, but the mapping should be explicit. The aggregate migration and drain statistics are useful, yet do not show the latency distribution of individual role conversions. Since conversion waits for the donor's existing requests to finish, this distinction matters for judging responsiveness to short bursts or long remaining outputs.

Questions for authors
  1. Clarification: Please provide RichPhase's risk formulas, calibration objective, tuning ranges, and final decision rule. When both methods use the same information, which decisions differ because of the freeness normalization? Under what conditions is this comparison intended to remain useful when decode latency is limited by execution or bandwidth rather than KV capacity?

  2. Measurement clarification: Do the Burst attribution tables use the full trace while Table 1 uses the peak cohort? Please specify the windows and request cohorts, and explain the trace, seed, or configuration differences behind the Prefill-heavy Full results in Tables 1 and 2.

  3. Targeted supporting evidence: Beyond the reported aggregate drain time and conversion counts, can the authors provide a representative per-conversion latency breakdown and distribution, including draining, switching, and activation? How does the controller handle a donor whose remaining requests take longer to drain than the imbalance persists? Existing measurements or an explicit operating limitation would help assess this concern.

3 comments
AuthorsOpenReview ↗

Response to Reviewer ubzE (1/3)

We thank Reviewer ubzE for the constructive review and for recognizing our controlled ablation design and candid scoping of Phasor's operating regime (including the decode-heavy case where Bullet achieves lower TPOT and E2E P99). We address Q1 below, Q2 in (2/3), and Q3 in (3/3).

Q1. RichPhase specification, normalization impact, and decode execution/bandwidth boundary.

1. RichPhase formulation, calibration, and decision rule: RichPhase isolates the benefit of unifying heterogeneous P/D pressure onto a common feasibility scale (Φ=0\Phi=0) from richer phase-local estimation:

  • Identical inputs: RichPhase uses the exact estimators in Eq. (3)–(4): prefill completion T^rem(q∣w,r,t)\hat{T}_{rem}(q|w,r,t) and TTFT slack sq(t)s_q(t)—ratio ρT(w;r,t)=max⁡q∈Aw(r,t)T^rem(q∣w,r,t)/sq(t)\rho_T(w;r,t)=\max_{q\in A_w(r,t)} \hat{T}_{rem}(q|w,r,t)/s_q(t)—plus allocatable blocks Falloc(w,t)F_{alloc}(w,t), budget Bcap(w)B_{cap}(w), and horizon-HH growth B^req(w;r,t)\hat{B}_{req}(w;r,t)—ratio ρS(w;r,t)=(Bcap(w)−Falloc(w,t)+B^req(w;r,t))/Bcap(w)\rho_S(w;r,t)=(B_{cap}(w)-F_{alloc}(w,t)+\hat{B}_{req}(w;r,t))/B_{cap}(w).
  • Extra calibration freedom vs. parameter-free zero: Full Phi sets Φx=1−ρx\Phi_x=1-\rho_x with a parameter-free physical zero (Φ=0  ⟺  ρx=1\Phi=0\iff\rho_x=1, exact TTFT deadline or KV exhaustion). RichPhase instead retains extra degrees of freedom to transform and calibrate P and D risk scales separately on held-out traces to maximize joint SLO-goodput within the same controller tuning budget.
  • Matched controllers and Macro rule: Both share identical Micro, Meso, and Macro state machines, 50-ms epochs, 200-ms persistence (K=4K=4), 1-s cooldown, donor selection, migration limits, and actuators. In RichPhase, Micro routes to minimal local prefill risk; Meso triggers on local decode deficit; Macro compares normalized threshold exceedances across the two separately calibrated scales rather than ΔΦ=ΦˉD−ΦˉP\Delta_\Phi=\bar{\Phi}_D-\bar{\Phi}_P. We will add the explicit calibration formulas and parameter table to the revised appendix.

2. Why gains stem from normalization, and which decisions differ: Within a single phase (e.g., Micro routing), monotonic transforms largely preserve intra-phase worker ranking. Normalization changes decisions at cross-phase and escalation boundaries:

  • Cross-phase pressure shifts: Despite extra freedom to calibrate P and D independently, RichPhase's separate scales lose comparability away from calibration. Under Burst, pressure-shift decision latency degrades from 0.26±0.03 s (Full Phi) to 0.44±0.04 s (RichPhase), and recovery slows from 5.0±0.4 s to 6.2±0.5 s (Appendix G).
  • Escalation selectivity: Without a shared zero linking Meso's sufficiency check (TRYMESO) to Macro's trigger, RichPhase runs 6.0±0.8 conversions over the 180-s Burst vs. 4.0±0.0 under Full Phi (~50% more from mistimed triggers), trailing by 4.3–11.0% in goodput and 7.0–10.8% in E2E P99 (Table 2). Combined with our sensitivity sweeps (0.92–1.01 normalized capacity across perturbations), gains stem from anchoring Φ=0\Phi=0 to physical feasibility, explaining why the gap is smallest in Decode-heavy (3.05 vs. 3.18 req/s) and largest in Burst (8.74 vs. 9.70 req/s).

3. Usefulness under execution- or bandwidth-limited decode: While ΦS\Phi_S models KV admission capacity rather than per-step TPOT, comparing Full Phi vs. RichPhase—and Phasor's decode branch—remains useful under execution- or bandwidth-limited decode in two regimes:

  • Prefill starvation and TTFT-sensitive joint SLOs: On Llama3.1-8B Decode-heavy (short prompts, long outputs, ample HBM headroom), Bullet yields 4.1% lower TPOT (33.1 vs. 34.5 ms) and 4.8% lower E2E P99 (57.69 vs. 60.58 s). Yet Phasor cuts TTFT by 24.4% (0.155 vs. 0.205 s), raises joint SLO attainment by 8.0 pp, and achieves 9.9% higher goodput (21.1 vs. 19.2 req/s), keeping its 12/12 goodput lead in Table 1.
  • Bandwidth-bound TPOT tied to KV volume: When step time scales with per-worker KV reads, Meso's allocatability balancing prevents skewed KV buildup and per-worker bandwidth bottlenecks.

KV-only signals stop differentiating decode pressure only when workers uniformly saturate compute/bandwidth well below KV capacity, where Virtual Freeness extends via Φdecode=min⁡(ΦKV,ΦTPOT)\Phi_{decode}=\min(\Phi_{KV},\Phi_{TPOT}) using batch step-time slack to the TPOT SLO. We will document these conditions in the revised paper.

AuthorsOpenReview ↗

Response to Reviewer ubzE (2/3)

Continuing our response to Reviewer ubzE, we address Q2 below and Q3 in (3/3).

Q2. Measurement windows, request cohorts, and Table 1 vs. Table 2 reconciliation.

For Qwen3.5-27B, the two tables align precisely:

  • Burst (22.9 vs. 9.70 req/s): Yes—Table 1 evaluates the peak-burst cohort (requests arriving during the 90–120 s high-load shock at 26.3 req/s mean offered load) of the 180-s Burst replay, yielding 13.75 s E2E P99 and 22.9 req/s goodput. By contrast, Tables 2, 7, and 8 report the full 0–180 s trace cohort (7.7->26.3->7.7 req/s). Since the nominal whole-trace arrival rate is (7.7×150+26.3×30)/180=10.8(7.7\times150+26.3\times30)/180=10.8 req/s, Table 2's 9.70 req/s aligns exactly with Table 7's 89.8% joint SLO attainment (10.8×0.898=9.7010.8\times0.898=9.70 req/s).
  • Prefill-heavy and Decode-heavy Full results: In Table 1 vs. Table 2, Qwen3.5-27B reports 7.56 s / 17.3 req/s vs. 7.00±0.22 s / 16.7±0.2 req/s (Prefill-heavy) and 84.33 s / 3.2 req/s vs. 94.50±2.60 s / 3.18±0.06 req/s (Decode-heavy). In Decode-heavy, 3.18 rounds to 3.2 req/s. Across workloads, Table 1 (the multi-system benchmark) and the attribution suite (Table 2, extended by Tables 7–8) are separate three-run measurement campaigns rather than shared runs—as Table 2's caption explicitly notes, only the two Full rows within Table 2 share the same runs. Within each campaign, all compared variants replay identical request sequences and arrival timestamps (yielding tight intra-table SDs such as ±0.2 req/s and strictly paired comparisons), whereas independent three-run trace realizations between the two campaigns reflect heavy-tailed ShareGPT length variation, shifting tail E2E P99 (notably in long-output Decode-heavy) and saturation goodput slightly without affecting any within-table comparison.
Qwen3.5-27B WorkloadTable 1 CohortTable 1 (P99 / GP)Table 2 CohortTable 2 Full (P99 / GP)Reconciliation
Burst90–120 s peak (26.3 req/s)13.75 s / 22.9 req/s0–180 s full (10.8 req/s)15.40±0.55 s / 9.70±0.20 req/sPeak vs. full (89.8% att.)
Prefill-heavyBenchmark 3-run mean7.56 s / 17.3 req/sAttribution 3-run mean±SD7.00±0.22 s / 16.7±0.2 req/sSeparate 3-run realizations
Decode-heavyBenchmark 3-run mean84.33 s / 3.2 req/sAttribution 3-run mean±SD94.50±2.60 s / 3.18±0.06 req/s3.18->3.2; separate runs

We will label these cohorts, windows, and campaign boundaries directly in the captions and setup of Tables 1, 2, 7, and 8.

AuthorsOpenReview ↗

Response to Reviewer ubzE (3/3)

Q3. Per-conversion latency breakdown, end-to-end cost accounting, and slow-draining donors.

1. Per-conversion latency breakdown across conversion directions: Equation 9 decomposes Macro's conversion response into persistence, draining, switching, and activation: Tresponse=KΔt+Tdrain+Tswitch+TactivateT_{response}=K\Delta t+T_{drain}+T_{switch}+T_{activate}. Phasor uses 50-ms epochs with 200-ms persistence (K=4K=4), a 1-s cooldown, and 20.0 ms readiness. In the 180-s Burst replay, Phasor executes 4.0±0.0 conversions with 6.1±0.5 GPU-s aggregate drain time (~1.53 s average drain per conversion) while achieving 5.0±0.4 s post-shock recovery.

To isolate P-to-D vs. D-to-P components directly, we measured conversion stages in an additional two-node (16-H100) 300-s high-burst replay (6,480 requests after 30 s warm-up):

Direction (16-H100)KΔtK\Delta tMean TdrainT_{drain}Mean TswitchT_{switch}Mean TactivateT_{activate}Mean Stop-Admission-to-First-Token
P-to-D (expand D)200 ms383.0 ms25.9 ms20.0 ms428.9 ms
D-to-P (expand P)200 ms3,506.6 ms108.8 ms20.0 ms3,635.4 ms

The stop-admission-to-first-token clock starts when admission stops and ends at the first served token in the new role, whereas Equation 9 spans persistence to activation. While our current instrumentation captures per-direction component means and multi-run SDs rather than per-event quantiles (which we will add in the revision), the ~9x directional gap (383.0 ms vs. 3,506.6 ms) reflects three properties:

  • Workload–direction alignment: P-to-D drains in 383.0 ms (428.9 ms to first token) because prefill chunks are short. Under Decode-heavy (128 input / 2,048 output tokens), prefill is lightly loaded and decode is KV-constrained, so Macro expands decode via fast P-to-D conversion rather than draining long decode sequences.
  • Donor selection under D-to-P: When prefill bursts trigger D-to-P (3,506.6 ms mean drain), Algorithm 1 limits tail drain by picking the eligible donor with maximum Φ\Phi (v←arg⁡max⁡u∈XeligibleΦ(u,t)v \leftarrow \arg\max_{u\in \mathcal{X}_{eligible}} \Phi(u,t))—preferring the decode worker with minimal active KV occupancy and projected demand.
  • Drain costs validate minimum-sufficient intervention: All serving results include actual draining and switching costs. In Figure 3(b), Global-first and Independent share the identical draining and conversion implementation as Phasor; precisely because D-to-P draining takes seconds, Phasor's sufficient-repair coordination cuts conversions by 50.0%/33.3%, cuts aggregate drain time by 52.3%/36.5%, and improves goodput by 13.5%/9.0% (9.70 vs. 8.55/8.90 req/s).

2. Handling donors whose drain time exceeds imbalance duration (Mechanism & Explicit Operating Limitation): To avoid aborting in-flight generation, Phasor's Mixed state is non-preemptive: donor vv stops admitting new work and completes resident sequences cleanly, while Algorithm 1 converts at most one worker before re-evaluating ΔΦ\Delta_\Phi and enforces a 1-s cooldown (t−tconv≥Tcoolt-t_{conv}\ge T_{cool}). In our 30-s Burst shock (5.0 s recovery), P-to-D finishes in ~0.38 s, well inside the imbalance window. However, we explicitly recognize an operating limitation: when a transient spike subsides faster than the remaining drain time of every eligible decode donor (most relevant for D-to-P under sub-second bursts while decode workers hold long active outputs), Macro cannot deliver capacity before the spike ends, leaving ultra-short transients to Micro and Meso. Supporting abortable Mixed-state rollback or Meso-assisted straggler evacuation is a direct extension we will document in Limitations.

We hope these specifications, cohort reconciliations, and operating boundaries resolve your remaining concerns and support a stronger overall assessment of our paper.

Reviewer_DfdUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that separating the prefill phase and decode phase into different GPU pools does not provide performance separation. The paper studies four perspectives: 1) a prefill backlog starves decode workers 2) phase-local metrics miss each phase's actual bottleneck 3) well-matched local signals omit queue nonlinear work, TTFT slack and allocatable KV blocks 4) similar SLO losses need different scopes of repair. Therefore, the authors propose Virtual Freeness and the minimum-sufficient-intervention principles and implement their system, Phasor, with Micro, Meso, and Macro, and show better performance compared to all baselines in all 12 settings. Additional ablation studies attribute gains to the signal and escalation policy.

Critical strengths
  1. Clear problem framing. The paper splits cross-phase control into two decisions: how much work a worker can still accept, and which intervention to use. This is a clean decomposition. The oracle study (Appendix A, Fig. 4) shows convincingly that similar SLO losses can require different repair scopes.

  2. A principled signal. Φ keeps each phase’s real constraint (deadline-bound compute for prefill, allocatable space for decode) while giving both a shared zero point and direction. The RichPhase comparison is a thoughtful control: it isolates the value of the common scale from the value of richer information, which ablations usually don’t do.

  3. Intervention-policy ablation. Holding the signal and actuators fixed and comparing Global-first, Independent, Local-only and Phasor (Fig. 3b) directly tests the minimum-sufficient-intervention principle. The paper also reports actuation costs (KV moved, drain time, conversions), not just goodput.

Critical weaknesses
  1. Novelty compared to Llumnix. "Virtual Freeness" seems similar to Llumnix's virtual-usage/freeness metrics, which unify dispatching, migration, and scaling. Llumnix is discussed in Appendix H but is neither cited in the main text nor compared against empirically. Given that the novelty rests on the signal and the escalation policy, the main paper should discuss its differences from Llumnix and provide an empirical comparison.

  2. Not an apple-to-apple comparison. Phasor can convert workers between pools at runtime, whereas baselines are kept at a fixed 4P-4D split throughout the run. Table 8 shows that Macro alone demonstrates large gains, especially during prefill-heavy scenarios. A simple baseline would be a reactive role-switching baseline, while one triggered by queue length could help separate the benefit of decision rules from the role conversion.

  3. Number mismatch across tables. For Qwen3.5-27B, tables 1 E2E P99 / goodput: 7.56 s / 17.3 (prefill-heavy), 84.33 / 3.2 (decode-heavy), 13.57 s / 22.9 (Burst). If Table 1 uses the full setting as in Table 2 Full Φ, Table 2 E2E P99 / goodput has 7.00 s / 16.7 (Prefill-heavy), 94.50 s / 3.18 (decode-heavy), 15.40 s / 9.7 (Burst). Further explanations from the authors are needed.

  4. Macro's latency in decode-heavy regimes is not quantified. A decode donor must have finished its activation sequences before it can convert. With roughly 2,048-token outputs at about 40 ms per token, draining could take tens of seconds. Eq. 9 breaks Macro’s response time into components, but those components are never reported per workload. Fig. 3b reports only Burst. Please report T_drain and T_response for decode-heavy traces and for prefill→decode conversions.

Questions for authors
  1. Please address the number mismatch between Table 1 and Tables 2,7,8. Were the ablations run with a different setting?
  2. How does Phasor compare with a Llumnix-style virtual-usage/freeness policy empirically, and with a simple reactive role-switching baseline, given the same conversion mechanism?
  3. What are T_drain and T_response for Macro under decode-heavy traffic, and how often does Macro fire there?
  4. How accurate is the TTFT completion-time predictor online, e.g., the error distribution under Burst? How accurate is the output-length quantile in practice?
  5. How much does migration copy traffic interfere with ongoing decode (TPOT impact during migrations), as opposed to the handoff pause alone?
3 comments
AuthorsOpenReview ↗

Response to Reviewer DfdU (1/3)

Thank you for your careful review and for recognizing the distinction between estimating feasible capacity and choosing a sufficient intervention. The controlled comparisons support a clear conclusion: Phasor's decision rules improve goodput even when the comparison policies have the same conversion capability. The additional Tuned-static experiment further shows that the gain persists after the fixed P/D split is calibrated. These results connect the two contributions you identified to measurable serving improvements.

Q2 / W1-W2. Phasor changes both the capacity model and the decision that follows from it.

Llumnix balances requests across complete serving instances, each executing both prefill and decode. Virtual usage incorporates queued demand, priority headroom and draining to coordinate dispatch, migration and autoscaling. Phasor addresses a different problem: recovering usable capacity in separate prefill and decode pools when latency targets are missed despite apparently available resources. The key decision is whether work redistribution or KV relocation can restore service within the current P/D allocation, or capacity should move between phases.

This requires different information. Phasor compares affected requests' predicted prefill completion with their remaining TTFT slack, and allocatable KV blocks with additional demand projected from sequence progress. For example, workers with equal memory headroom and batch size can face different future KV growth. Under normal priorities and without queuing or draining, Llumnix gives them equal freeness; Phasor can distinguish their projected allocation risk. The decode model therefore changes decisions in its own right.

The same model evaluates a proposed repair: the source must reach its projected headroom target, and the destination must remain feasible after receiving the resident KV and its projected growth. A sufficient repair, including one already in flight, justifies deferring decode-pool expansion. Phasor thus makes the expected outcome of local repair part of the phase-capacity decision. This is a core contribution that distinguishes its control decisions from Llumnix's. For a detailed comparison of the decode model and the sufficient-repair decision, please also see Sections 1-2 of our "Response to Reviewer yLt6 (1/3)". Appendix H develops this distinction, which we will also make explicit in the main text.

The controlled experiments distinguish information, coordination and provisioning.

Question testedMatched comparisonResult
Does the common scale add value with the same detailed information?Full Phi vs. RichPhase; same phase information, controllers and actuators4.3-11.0% higher goodput
Does coordination add value when every policy can convert workers?Phasor vs. Global-first / Independent; same Full Phi and all actuators13.5% / 9.0% higher goodput

The intervention study you highlight in Strength 3 is particularly relevant here. In Fig. 3(b), Global-first converts before local repair, while Independent permits separate triggers. Under Burst, Phasor achieves 9.70 req/s versus 8.55/8.90, moves 21.3%/33.3% less KV, and incurs 52.3%/36.5% less drain time. The advantage comes from how the available actions are coordinated.

The additional Tuned-static control tests whether Phasor's gain persists against a calibrated fixed split in these replay experiments. On Qwen3.5-27B (FP16, TP=1; eight H100s), it selects the best P:D split from 1:7 through 7:1 on separate calibration windows, freezes that split, and retains Micro/Meso with Macro disabled. We replay typical/high-burst BurstGPT windows, preserving arrival order and input/output pairing, scaled to 6,480 requests over 600 s after 60 s warm-up. SLO multipliers apply to isolated single-worker P90 TTFT and TPOT.

ReplaySLO multiplierSelected static P:DPhasor goodput gain
Typical3x4:428.9%
High-burst5x4:417.8%
Typical7.5x3:58.5%

The 4:4 split is selected by calibration in the first two cases, and Phasor still improves goodput substantially; the gain also persists against the selected 3:5 split. Appendix B.2 describes the original baselines' provisioning policies, including DistServe's SLO-aware provisioning. The additional experiment makes the static selection procedure and its selected allocations explicit. Table 2 tests the shared capacity model, Fig. 3(b) tests coordination with the same actuators, and Tuned-static tests adaptation against a calibrated fixed allocation.

AuthorsOpenReview ↗

Response to Reviewer DfdU (2/3)

Q1 / W3. The Burst rows report different request cohorts.

Table 1 reports requests arriving during the 90-120 s peak of the 180-s Burst replay. Tables 2/7/8 report the full attribution replay. The nominal whole-trace arrival rate is (7.7×150+26.3×30)/180=10.8(7.7\times150+26.3\times30)/180=10.8 req/s. Full's 9.70 req/s is therefore consistent with Table 7's 89.8% attainment. The peak goodput of 22.9 req/s and whole-trace goodput of 9.70 req/s describe different request cohorts. We will label those windows directly in the tables.

For Qwen3.5-27B, the reported E2E P99 and goodput values are:

WorkloadTable 1: E2E P99 / goodputTable 2 Full: E2E P99 / goodput
Prefill-heavy7.56 s / 17.3 req/s7.00 s / 16.7 req/s
Decode-heavy84.33 s / 3.2 req/s94.50 s / 3.18 req/s
Burst13.75 s / 22.9 req/s15.40 s / 9.70 req/s

The Decode-heavy goodput pair agrees after rounding 3.18 to 3.2 req/s. The two Full rows in Table 2 share the same measurements, and Tables 7/8 extend those measurements.

Q3 / W4. Expanding decode drains a prefill donor; long decode sequences affect the reverse conversion.

The conversion direction is decisive here. When decode needs more capacity, Macro performs P-to-D conversion and drains a prefill worker. D-to-P conversion waits for the selected decode worker's active sequences. In either direction, the donor stops accepting new work and completes its existing work, so drain time depends on what remains at selection.

Equation 9 separates the response into persistence, draining, switching and activation:

Tresponse=KΔt+Tdrain+Tswitch+Tactivate.T_{response}=K\Delta t+T_{drain}+T_{switch}+T_{activate}.

The configured persistence is 200 ms. The 1-s cooldown regulates conversion starts. Additional two-node measurements on sixteen H100s replay 6,480 high-burst requests over 300 s after 30 s warm-up:

ConversionMean donor drainMean stop-admission-to-first-token time
P-to-D383.0 ms428.9 ms
D-to-P3,506.6 ms3,635.4 ms

In these high-burst trials, P-to-D conversion finishes substantially sooner than D-to-P conversion. Readiness alone is 20.0 ms; the table includes draining and service resumption. Its clock starts when admission stops and ends at the first served token, whereas Equation 9 starts with persistence and ends at activation.

For frequency, the submitted 180-s Burst experiment records 4.0 +/- 0.0 conversions and 6.1 +/- 0.5 GPU-s of aggregate drain time.

These costs are included in the serving results. Under the same actuation implementation, Phasor's higher goodput and lower aggregate drain time in Fig. 3(b) show the benefit of coordinating recovery after accounting for its execution cost.

AuthorsOpenReview ↗

Response to Reviewer DfdU (3/3)

Q4. Execution-cost calibration and controlled biases show how estimation affects scheduling.

The online prefill estimate uses the current queue, batch/chunk configuration and execution progress to predict remaining completion time. Shared batches are counted once and completed work is removed. Decode demand is projected over horizon HH from observed generation progress and a conservative remaining-output estimate, then rounded to additional allocator blocks (Appendix F.4). An active request remains active after exceeding the output-length quantile.

Offline profiling over input lengths 128-8,192 yields an execution-cost fit with R2>0.99R^2>0.99. For the online controller, we additionally test how errors in the estimates affect its decisions. We jointly multiply prefill-time and KV-growth estimates by 0.5 or 2 on high-burst replay at 7.5x SLO, holding arrivals and execution fixed. The multipliers apply during 360-560 s; goodput retention covers the 600-s replay, while Meso and allocation-failure counts cover the perturbation interval. Values are mean +/- SD.

Prefill / KV multiplierGoodput retentionMeso activationsAllocation failures
0.5 / 0.599.65 +/- 8.79%50 +/- 10.61,201 +/- 485
2 / 297.92 +/- 8.78%40 +/- 10.60
0.5 / 299.24 +/- 8.84%39 +/- 9.60
2 / 0.596.84 +/- 9.67%70 +/- 13.22,331 +/- 560

Unperturbed Full achieves 10.343 +/- 0.068 req/s with 19 +/- 6.2 Meso activations and zero allocation failures. Both KV-overestimation settings record zero allocation failures. Halving the KV estimate produces 1,201 or 2,331 failures, depending on the prefill bias. Mean Meso activations rise from 19 to 39-70 across the four perturbations. These results explain the choice of conservative KV growth estimates and show how repair activity changes when estimates are biased.

Q5. We measure complete migration time and serving performance with transfers active.

Meso copies the snapshot and subsequent deltas while the source continues decoding, then pauses for final synchronization and ownership transfer. Table 4's 8.1-ms P99 measures that final pause. The additional two-node, 8K-context test under remote contention measures the complete transfer as well:

Measured quantityResult
Total migration duration, P9562.40 ms
Final handoff pause, P9517.41 ms
Reference decode iteration32.43 ms

The serving experiment measures joint TTFT/TPOT SLO-goodput while migration and controller actuation are active. With an initial 4P:4D allocation on each of two eight-H100 nodes, we compare both clean and contended networks at 3x isolated-P90 TTFT/TPOT SLOs:

Network conditionEffective inter-node bandwidthPhasor goodput gain over Mooncake
Clean40 GB/s10.4%
Bidirectional background transfers20 GB/s2.3%

Phasor retains higher goodput in both network conditions, including the contended case where effective bandwidth is halved. The gain is measured with transfer, draining and conversion costs present.

We welcome further discussion and would be happy to clarify any remaining questions. If the clarifications and evidence above address your concerns, we respectfully ask you to reconsider your overall assessment and rating.

Reviewer_yLt6OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies prefill/decode (P/D) disaggregated LLM serving and argues that placing the two phases on separate GPUs does not isolate their performance. For example, a prefill burst can starve the decode pool. Meanwhile, metrics like SM utilization or HBM occupancy which are used to detect and address issues give a misleading picture of headroom. The authors propose Virtual Freeness, a perworker signal that expresses remaining capacity as a normalized margin to each phase's feasibility limit. The authors introduce their system Phasor to choose the narrowest sufficient action: routing new requests (Micro), migrating KV state between decode workers (Meso), or converting a GPU between prefill and decode roles (Macro). On a single H100 node (with 8GPUs) with four models and three workloads. Phasor achieves the best SLO-goodput and P99 TTFT in all 12 settings against five reimplemented baselines, improving goodput by 6.7–29.0% over the strongest baseline in each setting.

Critical strengths
  1. The motivation section is clear and well supported, with concrete measurements showing how prefill overload propagates to decode and how utilization metrics mislead in each phase.
  2. The evaluation is broad for a single node, covering dense and MoE models from 8B to 110B, three workload types, five baselines, and controlled comparisons that hold the actuators fixed while varying the signal or the escalation policy.
  3. The design is practical, with low reported overhead (2.8% total, ~12 µs routing decisions, ~8 ms migration pause) and a careful migration protocol with explicit ownership and failure handling.
Critical weaknesses
  1. The paper builds heavily on Llumnix (OSDI '24) without saying so in the main text. Llumnix defines "virtual usage" and a "freeness" metric (free memory minus anticipated demand, normalized, with negative values meaning overload) and uses it for dispatching, migration and scaling. Phasor's decode-side Virtual Freeness has the same structure. Meso's migration protocol (copy existing KV during decoding, copy deltas in rounds, then a short stop-and-copy with a confirmation handshake) is essentially Llumnix's live migration, yet neither Section 3.2 nor Appendix E.2 cites it; Llumnix appears only in Appendix H. The genuinely new parts appear to be the prefill-side deadline signal, the shared prefill/decode scale, and trying migration before role conversion. The paper should credit Llumnix in the main text and compare against it at least at the component level. For example, it could replace Meso and the decode-side signal with Llumnix's freeness and migration policy while keeping the rest of Phasor fixed. The current Meso evidence (Figure 3a, Table 6) compares only against systems that do not migrate at all.

  2. Dynamic P/D role switching is also not new, and closely related systems are missing. "Arrow: Adaptive Scheduling Mechanisms for Disaggregated LLM Inference Architecture" flips instances between prefill and decode using transitional pools much like Phasor's Mixed state, and DynaServe and DOPD also adapt the P/D split at runtime. None of these is cited or compared.

  3. It is not clear how much of the gain comes from simply fixing a poorly chosen 4P:4D split. Two of the three workloads are deliberately lopsided (~2K input / 100 output tokens and the reverse), Phasor starts at 4P:4D and can convert GPUs between roles, and the paper does not say what split the baselines ran at, or whether DistServe, which searches over provisioning, was allowed to pick its own. The ablation suggests this matters. In the cumulative ablation (Table 2), adding Macro, which only changes the split (I think?), appears to account for much of Phasor's goodput gain over Base in every workload, including the steady Poisson ones.

Questions for authors
  1. Beyond the prefill-side signal and the cross-phase comparison, what does Phasor add over Llumnix's freeness metric and live migration? How does Phasor perform if Meso and the decode-side signal are replaced with Llumnix's freeness and migration policy, and how does it compare against a role-switching system such as Arrow?
  2. I have to ask, given how close this is to a strong OSDI paper why not consider sending this to a systems venue that would provide more space to explain the method in the paper and would be better equiped to evaluate this paper?
3 comments
AuthorsOpenReview ↗

Response to Reviewer yLt6 (1/3)

Thank you for the time and effort you have devoted to reviewing our paper. We respectfully disagree with the characterization that Phasor "builds heavily on Llumnix." Phasor and Llumnix differ fundamentally in both the scientific questions they address and the solutions they develop. Llumnix balances requests across instances that each execute both prefill and decode, coordinating dispatch, migration and autoscaling through virtual load. Phasor asks why nominally available resources fail to deliver SLO-compliant service in separate prefill and decode pools, and which intervention can recover that capacity. The decisive question is whether redistributing work or KV state can resolve the bottleneck within the current P/D allocation, or capacity needs to move between phases. Answering it requires assessing whether a proposed repair will be sufficient. Our models of prefill deadline feasibility and projected decode KV headroom support that assessment; our minimum-sufficient intervention principle uses it to choose the scope of intervention. This is a core contribution of Phasor that distinguishes its control decisions from those of Llumnix. Appendix H already develops this distinction; we will make it explicit in the main text.

1. The decode model changes decisions even before cross-phase control is considered.

You ask what the decode branch adds beyond prefill and the common scale. It changes which workers are predicted to need repair. Llumnix's FL=(M−∑iVi)/BF_L=(M-\sum_i V_i)/B includes physical usage, queued demand, priority headroom and draining; division by batch size accounts for memory consumption per iteration. Phasor's ΦS=(Falloc−B^req)/Bcap\Phi_S=(F_{alloc}-\widehat B_{req})/B_{cap} compares immediately allocatable KV blocks with additional demand projected over horizon HH. We estimate that growth from sequence progress and conservative remaining-output estimates, round to allocator blocks, and count the increments beyond already allocated state (Section 3.1; Appendix F.4).

Take two workers with equal KV capacity, allocatable blocks, current occupancy and batch size, serving normal-priority requests with no queue or draining. Llumnix assigns them equal freeness. Yet their requests can have different projected growth: one worker may retain enough headroom while the other faces an allocation deficit. Phasor distinguishes these states and can trigger repair accordingly. A virtual-usage implementation would need the added progress-dependent demand model to make that distinction. The difference therefore reaches the migration decision itself, even in this decode-only example.

For prefill, we evaluate predicted completion against the remaining TTFT slack of every request affected by an assignment. We then express both phases as normalized margins to their respective constraints, with a common zero and direction. This lets the controller compare how close each phase is to its limit. Table 2 tests the value of this shared interpretation against RichPhase, which receives the same detailed phase information.

2. Minimum-sufficient intervention decides whether local repair justifies keeping the current P/D allocation.

The phrase "trying migration before role conversion" leaves out the decisive step: judging whether migration will be enough. Routing changes where new work goes, relocation changes where KV resides, and conversion transfers capacity between phases. Choosing the right scope matters. In Appendix A's offline diagnostic, routing recovers 18% of lost goodput under KV-placement loss, rising to 91% with relocation. Under a pool-wide deficit, routing plus relocation recovers 16%, rising to 93% with conversion. Similar service losses therefore call for different remedies.

We turn that observation into a decision rule. Meso evaluates the state after a proposed repair: the source must reach projected ΦS≥θS\Phi_S\geq\theta_S, and the destination must retain positive headroom after receiving the KV and its projected growth. A repair expected to meet that target, or such a repair already in flight, justifies deferring decode-pool expansion. Otherwise, persistent cross-pool pressure can trigger Macro (Algorithm 1; Appendix E.2).

This is why minimum-sufficient intervention is a contribution in its own right. Llumnix balances instance freeness using migration thresholds and changes the number of serving instances through autoscaling. Phasor uses the expected outcome of a local repair to decide whether to preserve the current P/D allocation. That links worker-level feasibility to phase-capacity decisions within the GPU budget.

We implemented Meso independently in vLLM and will cite Llumnix's bulk-copy/delta/handoff technique in Section 3.2 and Appendix E.2. The experiments below hold execution mechanisms fixed to test the capacity model and recovery rule that distinguish Phasor.

AuthorsOpenReview ↗

Response to Reviewer yLt6 (2/3)

3. The gains persist with the same information, the same actions, and tuned static provisioning.

Your concern is that the gains may mainly come from a poor static split or from allowing worker conversion. We tested both explanations, alongside the contribution of the phase model:

ComparisonHeld fixedPhasor goodput gain
Full Phi vs. RichPhase (Table 2)Detailed phase information, controllers and actions4.3–11.0%
Phasor vs. Global-first / Independent (Fig. 3b, Burst)Full Phi and all routing, migration, draining and conversion implementations13.5% / 9.0%

RichPhase receives the same nonlinear prefill-work estimate, TTFT slack, projected KV demand and allocatable capacity, but calibrates its phase risks separately. Full Phi gives the two phases a common interpretation. Under Burst, pressure-shift decision latency falls from 0.44 to 0.26 s and recovery from 6.2 to 5.0 s (Appendix G). In Fig. 3(b), all three policies can convert workers: Global-first converts before local repair, Independent permits separate triggers, and Phasor selects a sufficient recovery scope. Phasor also moves 21.3%/33.3% less KV and incurs 52.3%/36.5% less GPU drain time. The choice of signal and coordination rule improves service even with the same information and available actions.

On Qwen3.5-27B (FP16, TP=1; eight H100s), Tuned-static selects the best P:D split from 1:7 through 7:1 on separate calibration windows, freezes it, and retains Micro/Meso with Macro disabled. We replay typical/high-burst BurstGPT windows preserving arrival order and input/output pairing, scaled to 6,480 requests/600 s after 60 s warm-up. SLO multipliers apply to isolated single-worker P90 TTFT and TPOT:

ReplaySLO multiplierSelected static P:DPhasor goodput gain
Typical3x4:428.9%
High-burst5x4:417.8%
Typical7.5x3:58.5%

Here, 4:4 is selected by calibration, yet Phasor still gains 28.9% and 17.8%; it also gains 8.5% against the selected 3:5 split. The gain therefore survives a tuned static split in these replay tests. For the original baselines, Appendix B.2 specifies provisioning policies, including DistServe's, rather than the selected splits. The additional control makes the static choice explicit while retaining Phasor's routing and migration.

AuthorsOpenReview ↗

Response to Reviewer yLt6 (3/3)

Title: Response to Reviewer yLt6 (3/3)

4. Phasor uses the expected result of local repair to decide whether to expand decode.

Section 5 and Appendix H already cover dynamic allocation and state migration through TokenScale, FlowKV and BanaServe. Phasor contributes the decision about whether existing capacity can be recovered within the current P/D allocation or capacity must move between phases.

Arrow: the condition for changing phase capacity differs. Arrow profiles TTFT/TPOT, routes requests through SLO checks, and considers role flipping when no suitable instance is found or pool-load conditions trigger reassignment. Phasor asks whether a concrete KV relocation can restore decode headroom before expanding the pool.

Suppose one decode worker has a projected KV deficit and another has spare capacity. Phasor accounts for resident KV and future growth removed from the source and added to the destination. If the source reaches its repair target and the destination remains feasible, it defers decode expansion. Otherwise, persistent pressure can trigger Macro, subject to donor eligibility and cooldown. The expected repair outcome therefore changes phase allocation. Arrow overlaps phase execution during transitions; Phasor drains before conversion.

DynaServe: request partitioning and repair scope are different decisions. DynaServe splits requests into micro-requests at token boundaries and places them across unified instances using predicted execution costs and current load. Its local scheduler adjusts batches under memory and time-between-token constraints. Phasor selects a sufficient change to existing service: redirect incoming work, relocate resident KV, or transfer a worker between phase pools, using the predicted capacity after that action.

DOPD: provisioning and reactive overload control differ from a sufficient-repair test. DOPD combines workload forecasts and profiled TTFT/TPOT to select P/D counts and parallelism, and reacts immediately to overload through prefill-queue depth and KV occupancy. Phasor evaluates affected requests' predicted prefill completion against remaining TTFT slack and a specific KV repair against projected source/destination headroom. These worker-level checks determine whether the current allocation can be retained.

Fig. 3(b) measures the coordination rule with the same Full Phi signal and identical routing, migration, draining and conversion implementations. Phasor improves goodput by 9.0-13.5% over Independent and Global-first while reducing KV movement and drain time. All three can convert workers; coordination changes the outcome. We will add these comparisons to Section 5 and Appendix H.

5. Inference modeling and scheduling are established ICLR research topics.

We chose ICLR because Phasor studies how to deliver more latency-compliant inference from a fixed GPU budget. The ICLR 2027 call explicitly includes infrastructure, software libraries and hardware. Its main-conference program reflects this scope: Preble (2025) coordinates distributed prompt scheduling, KV reuse and load balancing; Semantic Parallelism (2026) jointly schedules MoE experts and tokens to reduce communication. These are inference scheduling contributions with an established ICLR audience.

Phasor advances this research direction by studying why separate prefill/decode pools can miss latency targets despite apparent spare resources, and which intervention can recover usable capacity. Its phase-constraint model and minimum-sufficient intervention principle provide a concrete answer. The same-information and same-actuator comparisons test the value of those decisions, while Tuned-static tests adaptation against a calibrated fixed allocation. This connects a modeling question to controlled evidence about efficient model execution.

We welcome further discussion of the technical distinctions and experimental evidence presented above. In light of these clarifications, we respectfully ask you to reassess Phasor’s contributions and reconsider your overall evaluation of the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.