- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper presents a serving system for autoregressive streaming video generation, where long-lived sessions alternate between active and idle periods and each chunk has a latency target. The scheduler combines a placement controller that migrates sessions off the bottleneck GPU with an HPA-style autoscaler whose target utilization depends on workload volatility. The runtime supports CPU offloading of idle sessions and GPU-to-GPU session migration. On production traces with 1.3B and 7B models on H20 and B300 clusters, the paper reports about 37% lower worst-case chunk latency at matched cost and about 37% lower GPU cost at matched latency against fixed-allocation baselines, as well as near-oracle placement and autoscaling.
Critical strengths
- The workload is new and well motivated. Stateful video sessions really are different from both stateless video generation and LLM request serving. Section 2 does a good job of showing on real traces that placement imbalance and static provisioning both hurt.
- It's a real system on production clusters. Session migration takes 23–30 ms, about 2–3% of a chunk's latency, so the runtime side looks practical.
Critical weaknesses
- The baselines don't isolate the proposed scheduler. None of them autoscales or migrates, so the headline gains mostly show that autoscaling beats a fixed allocation. In the paper's own ablation, removing migration raises cost by 15%, versus 43% for removing autoscaling. HPA with least-loaded placement and a Llumnix-style migration baseline are missing.
- The evaluation metrics are limited. The paper doesn't explain how cost or latency was matched between a time-varying GPU budget and a fixed allocation, and it reports only the max chunk latency per trace, with no tail percentiles or stall rate. Without the chunk playback length, it's unclear whether any of the systems runs in real time.
- The formulation isn't what the algorithm optimizes. Eq. (1) trades cost against latency through λ(t), but neither controller uses λ, and the latency model behind the migration decisions is not specified.
- Several evaluation details are hard to verify. The traces are only 5 minutes long, although the motivation is about sessions lasting tens of minutes. The placement oracle is described as exhaustive search at up to 256 GPUs, which isn't feasible as stated, and the autoscaling-oracle numbers in the appendix don't line up with the reported cost gap.
Questions for authors
- How did you choose the baseline GPU counts for matched cost, and how was latency matched in the cost comparison?
- What are the chunk duration and the latency target for each model? Please report P95/P99 latency and the fraction of chunks over target.
- How long are the evaluation traces? Please reconcile the 5-, 12- and 30-minute windows. Also, every arrival and departure count in Table 3 is even. Were the traces duplicated or rescaled?
- What exactly does the placement oracle compute at 128 and 256 GPUs? How does the online autoscaler end up with target utilizations outside the profiled table?
- Could you add HPA with least-loaded placement, the same with periodic rebalancing, and a Llumnix-style migration policy?
4 comments
W3. The formulation isn't what the algorithm optimizes. Eq. (1) trades cost against latency through , but neither controller uses , and the latency model behind the migration decisions is not specified.
Role of . is a control parameter of the scheduler that is profiled offline together with the target utilization by grid search (Appendix E), and it takes effect through : a larger corresponds to a smaller , which reserves more latency headroom. As stated in Section 3.2.2, the controllers do not solve Eq. (1) explicitly at every event but regulate the load toward . In our traces the profiled is constant at 0.2 (Table 6), so the adaptation is carried by alone.
Latency model. The bottleneck latency of a placement is estimated as , where is the number of sessions on GPU and is the offline-profiled per-chunk latency of a GPU serving sessions.
Thank you for pointing this out. We will state the role of and the latency model explicitly in Section 3 of the revised version.
W4 + Q3 + Q4. How long are the evaluation traces? Please reconcile the 5-, 12- and 30-minute windows. Every arrival and departure count in Table 3 is even. Were the traces duplicated or rescaled? What exactly does the placement oracle compute at 128 and 256 GPUs? The autoscaling-oracle numbers in the appendix don't line up with the reported cost gap. How does the online autoscaler end up with target utilizations outside the profiled table?
Trace length. The different durations serve different purposes. The main Figure 5 latency experiments use a 5-minute replay window from the 30-minute trace files. The 12-minute window in Table 11 illustrates scheduling decisions and is not the window used for those latency measurements. Our additional experiments replay the full 30-minute traces and report cumulative worst-case latency at 5, 10, and 30 minutes. We will clarify these distinctions in the revised paper.
Longer traces. We ran additional experiments on three 30-minute production traces with 8 B300 GPUs. The table below reports the cumulative worst-case per-chunk latency after 5, 10, and 30 minutes of the same run. TurboServe is below LAG at every checkpoint, and after 30 minutes its worst-case latency is 17.9% to 33.4% lower.
| Trace | Duration | TurboServe | LAG |
|---|---|---|---|
| T4 | 5 min | 1.361 s | 1.696 s |
| T4 | 10 min | 1.361 s | 1.696 s |
| T4 | 30 min | 1.454 s | 2.183 s |
| T5 | 5 min | 1.411 s | 1.578 s |
| T5 | 10 min | 1.411 s | 2.050 s |
| T5 | 30 min | 1.684 s | 2.050 s |
| T6 | 5 min | 1.244 s | 1.883 s |
| T6 | 10 min | 1.306 s | 1.954 s |
| T6 | 30 min | 1.642 s | 2.362 s |
Even counts in Table 3. For the 2× scaling step, we duplicated each session’s event sequence, and the arrival, active/idle transitions, and departure were resampled. We will clarify the replication factors and timing perturbations in the revised paper.
Placement oracle. We assume GPUs are homogeneous and sessions are heterogeneous. The placement oracle is computed by sorting sessions by their base latency and using dynamic programming to divide them into at most contiguous groups, each containing at most sessions. Under the model , where is the fixed latency cost of session , and is the extra cost of serving sessions, the DP minimizes the maximum group latency in time after sorting, avoiding enumeration of all session-to-GPU assignments.
Autoscaling oracle. The two sets of numbers come from two different experiments. Table 1 compares TurboServe with an offline oracle that knows the full trace and plans the GPU budget for each 1-minute slot by dynamic programming, on three production traces on Cluster 2, where the cost gap is 4.7% to 8.3%. Appendix E compares only the online parameter mapping with an oracle that selects the target utilization for each 30 s window, with at most 16 GPUs, where the cost gap is 0.73% to 3.99%.
Target utilizations outside the profiled table. When the current workload is not covered by a profiled entry, the online autoscaler uses the closest one. At each event, it measures the recent volatility, finds the profiled level closest to it, and applies that level's target utilization (Appendix E). This is sufficient in practice: the unseen trace in Appendix E is such a case, as its optimal targets found by the oracle (0.55 to 0.95, Table 8) fall outside the profiled table, and the lookup still costs only 0.73% more than the oracle, which knows the future trace.
Thank you for the careful reading. We will add the 30-minute results and clarify the trace and oracle descriptions in the revised version.
We thank the reviewer once again for the detailed and constructive feedback. Should our responses and clarifications resolve the concerns raised, we would be grateful if the reviewer would consider raising the score accordingly.
Dear Reviewer NivQ,
As the discussion period closes today, we would like to kindly ask whether our responses have addressed your concerns, and whether there are any follow-up questions or remaining points we could clarify before the deadline. We thank you again for your time and thoughtful review. We wish you a wonderful day.
We sincerely thank the reviewer for the detailed and constructive comments. Below, we respond to each concern with new experiments and explanations, and we would be pleased to provide further evidence upon request.
Additional baselines. We ran additional experiments comparing TurboServe with least-loaded placement (LAG), HPA+LAG, which combines autoscaling with periodic rebalancing, and a Llumnix-style migration policy, on 8 B300 GPUs and three 30-minute production traces at the same GPU cost. The table below reports the worst-case per-chunk latency with migration gaps included. TurboServe has the lowest latency on all three traces, and lowers it on average by 27.2% compared with LAG, 22.5% compared with HPA+LAG, and 12.8% compared with the Llumnix-style policy.
Reason for the gain. As sessions pause and resume, the initial placement may become imbalance. LAG keeps all session on its assigned GPU, so the imbalance persists. HPA+LAG rebalances only at fixed intervals, so a latency bottleneck remains until the next rebalance. The Llumnix-style policy triggers migration using thresholds on memory room rather than predicted chunk latency. A latency bottleneck may therefore remain unaddressed when thresholds are not met. In contrast, TurboServe rebalances and migrates sessions at every session event, resolving the latency bottleneck as soon as it appears.
How cost or latency was matched:
The table below shows the resulting latency–cost points for 1.3B on T4 with B300. TurboServe meets the 3 s target at about 22% lower cost than the cheapest baseline that meets it, and LAG and MAG exceed the target even at the highest cost.
Chunk duration and latency target. The chunk duration and the latency target for each model are listed in the table below.
Tail latency and chunks over target. We ran additional experiments comparing the per-chunk latency distributions of TurboServe and the baselines for 1.3B on T4 with B300. The table below reports the mean, P50, P95, P99, and maximum over all N chunks, and the number of chunks over the 3 s target. TurboServe lowers P95 by 31% to 33% and P99 by 42% to 44% compared with the baselines, and none of its 61,282 chunks exceeds the target.
Reason for the lower tail. The baselines keep each session on its initially assigned GPU under a fixed GPU allocation, so load imbalance accumulates as sessions switch between active and idle, and the overloaded GPUs produce the tail. TurboServe removes this tail in two ways: it migrates sessions off the bottleneck GPU at every session event, and it scales out the GPU budget when the load rises.
Thank you for these suggestions. We will add the matching procedure, the chunk durations and latency targets, and the tail metrics to the revised version.