acceptodds
Under review as a conference paper at ICLR 2027

TurboServe: Serving Streaming Video Generation Efficiently and Economically

Abstract

Streaming video generation is emerging as a new serving workload in which users interact with long-lived sessions that generate video progressively, chunk by chunk. Unlike offline video generation or typical LLM serving, streaming video generation must preserve session state across active and idle periods, repeatedly schedule ongoing sessions, and deliver each chunk under a tight latency target. This creates two key serving challenges in multi-user, multi-GPU environments: session duration heterogeneity, where long-running sessions make placement decisions suboptimal over time, and temporal user-demand heterogeneity, where the number of active sessions fluctuates sharply across bursts and idle periods. We present TurboServe, the first serving system designed specifically for streaming video generation workloads. TurboServe formulates serving as an online scheduling problem that jointly coordinates session placement and GPU provisioning. Its closed-loop scheduling algorithm combines a migration-aware placement controller, which rebalances sessions across GPUs to reduce the maximum per-chunk latency, with a load-driven autoscaling controller, which adapts the GPU budget to workload variation for improved cost efficiency. To support these decisions at runtime, TurboServe implements coalesced chunk processing for batching concurrent active sessions on the same GPU, GPU-CPU offloading for session suspension and resumption, and GPU-GPU migration for online rebalancing. We evaluate TurboServe on real-world production traces from Shengshu Technology across multiple model sizes and GPU clusters with up to 64 NVIDIA B300 GPUs. Compared with baseline serving configurations, TurboServe reduces worst-case per-chunk latency by 37.5% and total GPU operating cost by 37.2% on average.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Streaming Video Generation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_NivQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents a serving system for autoregressive streaming video generation, where long-lived sessions alternate between active and idle periods and each chunk has a latency target. The scheduler combines a placement controller that migrates sessions off the bottleneck GPU with an HPA-style autoscaler whose target utilization depends on workload volatility. The runtime supports CPU offloading of idle sessions and GPU-to-GPU session migration. On production traces with 1.3B and 7B models on H20 and B300 clusters, the paper reports about 37% lower worst-case chunk latency at matched cost and about 37% lower GPU cost at matched latency against fixed-allocation baselines, as well as near-oracle placement and autoscaling.

Critical strengths
  1. The workload is new and well motivated. Stateful video sessions really are different from both stateless video generation and LLM request serving. Section 2 does a good job of showing on real traces that placement imbalance and static provisioning both hurt.
  2. It's a real system on production clusters. Session migration takes 23–30 ms, about 2–3% of a chunk's latency, so the runtime side looks practical.
Critical weaknesses
  1. The baselines don't isolate the proposed scheduler. None of them autoscales or migrates, so the headline gains mostly show that autoscaling beats a fixed allocation. In the paper's own ablation, removing migration raises cost by 15%, versus 43% for removing autoscaling. HPA with least-loaded placement and a Llumnix-style migration baseline are missing.
  2. The evaluation metrics are limited. The paper doesn't explain how cost or latency was matched between a time-varying GPU budget and a fixed allocation, and it reports only the max chunk latency per trace, with no tail percentiles or stall rate. Without the chunk playback length, it's unclear whether any of the systems runs in real time.
  3. The formulation isn't what the algorithm optimizes. Eq. (1) trades cost against latency through λ(t), but neither controller uses λ, and the latency model behind the migration decisions is not specified.
  4. Several evaluation details are hard to verify. The traces are only 5 minutes long, although the motivation is about sessions lasting tens of minutes. The placement oracle is described as exhaustive search at up to 256 GPUs, which isn't feasible as stated, and the autoscaling-oracle numbers in the appendix don't line up with the reported cost gap.
Questions for authors
  1. How did you choose the baseline GPU counts for matched cost, and how was latency matched in the cost comparison?
  2. What are the chunk duration and the latency target for each model? Please report P95/P99 latency and the fraction of chunks over target.
  3. How long are the evaluation traces? Please reconcile the 5-, 12- and 30-minute windows. Also, every arrival and departure count in Table 3 is even. Were the traces duplicated or rescaled?
  4. What exactly does the placement oracle compute at 128 and 256 GPUs? How does the online autoscaler end up with target utilizations outside the profiled table?
  5. Could you add HPA with least-loaded placement, the same with periodic rebalancing, and a Llumnix-style migration policy?
4 comments

We sincerely thank the reviewer for the detailed and constructive comments. Below, we respond to each concern with new experiments and explanations, and we would be pleased to provide further evidence upon request.


W1 + Q5. The baselines don't isolate the proposed scheduler, since none of them autoscales or migrates. Could you add HPA with least-loaded placement, the same with periodic rebalancing, and a Llumnix-style migration policy?

Additional baselines. We ran additional experiments comparing TurboServe with least-loaded placement (LAG), HPA+LAG, which combines autoscaling with periodic rebalancing, and a Llumnix-style migration policy, on 8 B300 GPUs and three 30-minute production traces at the same GPU cost. The table below reports the worst-case per-chunk latency with migration gaps included. TurboServe has the lowest latency on all three traces, and lowers it on average by 27.2% compared with LAG, 22.5% compared with HPA+LAG, and 12.8% compared with the Llumnix-style policy.

TraceTurboServeLAGHPA+LAGLlumnix-style
T41.454 s2.183 s1.699 s2.037 s
T51.684 s2.050 s2.120 s1.684 s
T61.642 s2.362 s2.428 s1.823 s

Reason for the gain. As sessions pause and resume, the initial placement may become imbalance. LAG keeps all session on its assigned GPU, so the imbalance persists. HPA+LAG rebalances only at fixed intervals, so a latency bottleneck remains until the next rebalance. The Llumnix-style policy triggers migration using thresholds on memory room rather than predicted chunk latency. A latency bottleneck may therefore remain unaddressed when thresholds are not met. In contrast, TurboServe rebalances and migrates sessions at every session event, resolving the latency bottleneck as soon as it appears.


W2 + Q1 + Q2. The paper doesn't explain how cost or latency was matched, and it reports only the max chunk latency. How did you choose the baseline GPU counts for matched cost, and how was latency matched in the cost comparison? What are the chunk duration and the latency target for each model? Please report P95/P99 latency and the fraction of chunks over target.

How cost or latency was matched:

  • For the matched-cost comparison, each baseline uses the fixed GPU count whose total GPU cost over the trace is closest to that of TurboServe.
  • For the matched-latency comparison, each baseline uses the smallest fixed GPU count that meets the same latency target.

The table below shows the resulting latency–cost points for 1.3B on T4 with B300. TurboServe meets the 3 s target at about 22% lower cost than the cheapest baseline that meets it, and LAG and MAG exceed the target even at the highest cost.

MethodCostWorst-case latencyMeets 3 s target
TurboServe_base7200≈3.105 sNo
TurboServe_base8400≈2.940 sYes
TurboServe_base96002.895 sYes
TurboServe_base+LAG96003.206 sNo
TurboServe_base+MAG96003.138 sNo
TurboServe6000≈3.030 sNo
TurboServe≈6520≈2.800 sYes
TurboServe≈7390≈2.650 sYes
TurboServe≈86902.302 sYes

Chunk duration and latency target. The chunk duration and the latency target for each model are listed in the table below.

ModelGPULatency target
1.3BH203 s
7BH204.5 s
1.3BB3003 s
7BB3004.5 s

Tail latency and chunks over target. We ran additional experiments comparing the per-chunk latency distributions of TurboServe and the baselines for 1.3B on T4 with B300. The table below reports the mean, P50, P95, P99, and maximum over all N chunks, and the number of chunks over the 3 s target. TurboServe lowers P95 by 31% to 33% and P99 by 42% to 44% compared with the baselines, and none of its 61,282 chunks exceeds the target.

MethodNMeanP50P95P99MaxChunks over 3 s
TurboServe_base472410.980 s0.916 s1.900 s2.270 s2.895 s0 (0.000%)
TurboServe_base+LAG494690.980 s0.914 s1.850 s2.348 s3.206 s31 (0.063%)
TurboServe_base+MAG486540.975 s0.917 s1.867 s2.318 s3.138 s17 (0.035%)
TurboServe612820.933 s0.922 s1.281 s1.320 s2.302 s0 (0.000%)

Reason for the lower tail. The baselines keep each session on its initially assigned GPU under a fixed GPU allocation, so load imbalance accumulates as sessions switch between active and idle, and the overloaded GPUs produce the tail. TurboServe removes this tail in two ways: it migrates sessions off the bottleneck GPU at every session event, and it scales out the GPU budget when the load rises.

Thank you for these suggestions. We will add the matching procedure, the chunk durations and latency targets, and the tail metrics to the revised version.

W3. The formulation isn't what the algorithm optimizes. Eq. (1) trades cost against latency through λ(t)\lambda(t), but neither controller uses λ\lambda, and the latency model behind the migration decisions is not specified.

Role of λ\lambda. λ(t)\lambda(t) is a control parameter of the scheduler that is profiled offline together with the target utilization ρ^(t)\hat{\rho}(t) by grid search (Appendix E), and it takes effect through ρ^(t)\hat{\rho}(t): a larger λ(t)\lambda(t) corresponds to a smaller ρ^(t)\hat{\rho}(t), which reserves more latency headroom. As stated in Section 3.2.2, the controllers do not solve Eq. (1) explicitly at every event but regulate the load toward ρ^(t)\hat{\rho}(t). In our traces the profiled λ\lambda is constant at 0.2 (Table 6), so the adaptation is carried by ρ^(t)\hat{\rho}(t) alone.

Latency model. The bottleneck latency of a placement is estimated as max⁡jf(nj)\max_j f(n_j), where njn_j is the number of sessions on GPU gjg_j and f(n)f(n) is the offline-profiled per-chunk latency of a GPU serving nn sessions.

Thank you for pointing this out. We will state the role of λ\lambda and the latency model explicitly in Section 3 of the revised version.


W4 + Q3 + Q4. How long are the evaluation traces? Please reconcile the 5-, 12- and 30-minute windows. Every arrival and departure count in Table 3 is even. Were the traces duplicated or rescaled? What exactly does the placement oracle compute at 128 and 256 GPUs? The autoscaling-oracle numbers in the appendix don't line up with the reported cost gap. How does the online autoscaler end up with target utilizations outside the profiled table?

Trace length. The different durations serve different purposes. The main Figure 5 latency experiments use a 5-minute replay window from the 30-minute trace files. The 12-minute window in Table 11 illustrates scheduling decisions and is not the window used for those latency measurements. Our additional experiments replay the full 30-minute traces and report cumulative worst-case latency at 5, 10, and 30 minutes. We will clarify these distinctions in the revised paper.

Longer traces. We ran additional experiments on three 30-minute production traces with 8 B300 GPUs. The table below reports the cumulative worst-case per-chunk latency after 5, 10, and 30 minutes of the same run. TurboServe is below LAG at every checkpoint, and after 30 minutes its worst-case latency is 17.9% to 33.4% lower.

TraceDurationTurboServeLAG
T45 min1.361 s1.696 s
T410 min1.361 s1.696 s
T430 min1.454 s2.183 s
T55 min1.411 s1.578 s
T510 min1.411 s2.050 s
T530 min1.684 s2.050 s
T65 min1.244 s1.883 s
T610 min1.306 s1.954 s
T630 min1.642 s2.362 s

Even counts in Table 3. For the 2× scaling step, we duplicated each session’s event sequence, and the arrival, active/idle transitions, and departure were resampled. We will clarify the replication factors and timing perturbations in the revised paper.

Placement oracle. We assume GPUs are homogeneous and sessions are heterogeneous. The placement oracle is computed by sorting sessions by their base latency and using dynamic programming to divide them into at most MM contiguous groups, each containing at most KK sessions. Under the model L(S)=max⁡i∈Sai+h(∣S∣)L(S)=\max_{i\in S}a_i+h(|S|), where aia_i is the fixed latency cost of session ii, and h(n)h(n) is the extra cost of serving nn sessions, the DP minimizes the maximum group latency in O(MNK)O(MNK) time after sorting, avoiding enumeration of all session-to-GPU assignments.

Autoscaling oracle. The two sets of numbers come from two different experiments. Table 1 compares TurboServe with an offline oracle that knows the full trace and plans the GPU budget for each 1-minute slot by dynamic programming, on three production traces on Cluster 2, where the cost gap is 4.7% to 8.3%. Appendix E compares only the online parameter mapping with an oracle that selects the target utilization for each 30 s window, with at most 16 GPUs, where the cost gap is 0.73% to 3.99%.

Target utilizations outside the profiled table. When the current workload is not covered by a profiled entry, the online autoscaler uses the closest one. At each event, it measures the recent volatility, finds the profiled level closest to it, and applies that level's target utilization (Appendix E). This is sufficient in practice: the unseen trace in Appendix E is such a case, as its optimal targets found by the oracle (0.55 to 0.95, Table 8) fall outside the profiled table, and the lookup still costs only 0.73% more than the oracle, which knows the future trace.

Thank you for the careful reading. We will add the 30-minute results and clarify the trace and oracle descriptions in the revised version.

We thank the reviewer once again for the detailed and constructive feedback. Should our responses and clarifications resolve the concerns raised, we would be grateful if the reviewer would consider raising the score accordingly.

Dear Reviewer NivQ,

As the discussion period closes today, we would like to kindly ask whether our responses have addressed your concerns, and whether there are any follow-up questions or remaining points we could clarify before the deadline. We thank you again for your time and thoughtful review. We wish you a wonderful day.

Reviewer_q5vSOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TurboServe combines session migration and GPU autoscaling for streaming video generation. On 1.3B/7B models, six production traces, and H20/B300 clusters, it reports 37.5% lower worst-case chunk latency at matched GPU cost and 37.2% lower GPU cost at matched worst-case latency, averaged over comparisons with static placement baselines.

Critical strengths
  1. A useful system for a relevant workload. TurboServe supports long-lived video sessions with idle/resume transitions. Chunk-boundary migration and the separation of shared weights from session state provide a coherent design for managing these sessions (§§2–3; Appendix F).

  2. Evaluation across workloads and hardware. Figure 5 compares latency and cost across two model sizes, six traces, and two GPU platforms. Figure 6 ablates migration and autoscaling, and Table 12 reports mean migration overheads of 23–30 ms.

Critical weaknesses
  1. The anonymous artifact excludes active-session migration interruptions from reported latency. Migration clears the preceding output timestamp, and the first post-migration output restarts the clock (turboserve/scheduling/metrics.py, lines 103–116, 162–170). With synthetic timestamps, a CPU test reports a 100 ms maximum and zero deadline misses despite a 10.1 s migration-crossing gap. Whether Figures 5–6 used this implementation is unspecified. Excluding these gaps can bias both matched-cost latency and matched-latency cost, directly affecting the streaming-latency claims (§4.1; Appendix D.1).

  2. The benefit of the proposed control policy is not isolated. Contribution 1 introduces migration-aware placement and load-driven autoscaling, but the main baselines disable both mechanisms. A3 combines periodic rebalancing and autoscaling (§2.3; Appendix C.2), yet is absent from the main evaluation. The reported comparison therefore mixes the benefit of adding these capabilities with that of the proposed control policy.

Questions for authors
  1. Which timing implementation produced Figures 5–6 and Appendix E? Provide consecutive output timestamps for active sessions spanning migration. If the artifact's gap exclusions were used, report the matched-cost and matched-latency results with those gaps included.
  2. How does TurboServe compare with the baselines on median and tail activation-to-first-chunk and resume-to-first-chunk latency under bursty demand and scale-out? Include queueing, state restoration, and GPU initialization when they delay the first output.
  3. How much does full TurboServe improve over tuned periodic rebalancing plus autoscaling, such as A3, using identical runtime primitives, scaling overheads, and streaming-latency accounting?
4 comments

We sincerely thank the reviewer for the careful and thorough review of our work. Below, we address each concern with additional experiments and clarifications, and we remain available to provide any further information the reviewer may require.


W1 + Q1. The artifact excludes active-session migration interruptions from reported latency. Which timing implementation produced Figures 5–6 and Appendix E? Provide consecutive output timestamps for active sessions spanning migration. If the artifact's gap exclusions were used, report the matched-cost and matched-latency results with those gaps included.

Timing implementation. Figures 5–6 and Appendix E were produced with the timing implementation in the artifact, which excludes the gaps that span a migration. We have corrected it so that every gap between consecutive outputs of a session is counted.

Consecutive timestamps. The table below lists the consecutive output timestamps of active sessions around a migration.

SessionLast output before migrationFirst output after migrationGap
138 (7B, T4)125.056 s128.145 s3.089 s
5164 (7B, T6)132.818 s136.479 s3.660 s
3184 (1.3B, T5)45.827 s47.996 s2.169 s

Results with gaps included. We recomputed the worst-case per-chunk latency of TurboServe for all 12 configurations of Figure 5 with the gaps included. The two tables below report the latency before and after the correction. The latency is unchanged in 10 of the 12 configurations and increases in two, both with the 7B model on B300: by 16.4% on T4 and by 18.6% on T6. The baselines do not migrate, so their results are unchanged.

  • H20 (gaps excluded → gaps included):
ModelT1T2T3
1.3B2.096 s → 2.096 s1.793 s → 1.793 s2.869 s → 2.869 s
7B2.288 s → 2.288 s2.339 s → 2.339 s2.571 s → 2.571 s
  • B300 (gaps excluded → gaps included):
ModelT4T5T6
1.3B2.302 s → 2.302 s2.169 s → 2.169 s2.469 s → 2.469 s
7B2.653 s → 3.089 s2.869 s → 2.869 s3.087 s → 3.660 s

Why 10 configurations are unchanged. The worst-case latency is the maximum over all gaps between consecutive outputs. In these 10 configurations, every gap that spans a migration is shorter than the existing maximum, which comes from a chunk that does not follow a migration. Including the gaps therefore does not change the reported value.

Why 2 configurations change. With the gaps included, a migrated session may wait in the queue of its target GPU before its next chunk is generated. In these two configurations, this gap between consecutive outputs is longer than the previous maximum, so it becomes the new worst-case latency.

Advantage in the 2 changed configurations. Even with the gaps included, TurboServe still lowers the worst-case per-chunk latency by 30.3% to 39.5% compared with the three baselines in these two configurations at matched GPU cost, as shown in the table below. The percentage in parentheses is the reduction achieved by TurboServe relative to that baseline.

ConfigurationTurboServe_baseTurboServe_base+LAGTurboServe_base+MAGTurboServe (gaps included)
7B, T4, B3004.909 s (37.1%)4.590 s (32.7%)4.564 s (32.3%)3.089 s
7B, T6, B3006.051 s (39.5%)5.989 s (38.9%)5.248 s (30.3%)3.660 s

Thank you for pointing this out. We will update Figures 5–6, Appendix E, and the artifact to use the gap-inclusive latency in the revised version.

Q2. How does TurboServe compare with the baselines on median and tail activation-to-first-chunk and resume-to-first-chunk latency under bursty demand and scale-out? Include queueing, state restoration, and GPU initialization when they delay the first output.

First-chunk latency. We ran additional experiments measuring both latencies for TurboServe, A3, and LAG on 8 B300 GPUs and three 30-minute production traces, end to end from the event to the first delivered chunk, including queueing, state restoration, and GPU initialization. The two tables below report the median, P99, and maximum. TurboServe is comparable to the baselines on both metrics: its P99 is within 3% of theirs in all cases except activation on T6, where it is 11% lower, and its medians are within 0.4 s. On the same runs, TurboServe lowers the worst-case per-chunk latency by 14.4% to 32.4% compared with A3 and by 17.9% to 33.4% compared with LAG.

  • Activation-to-first-chunk latency:
TraceSchedulerP50P99Max
T4TurboServe1.940 s54.020 s73.739 s
T4A31.913 s53.866 s72.881 s
T4LAG2.003 s53.377 s73.194 s
T5TurboServe7.660 s69.035 s92.446 s
T5A37.643 s70.048 s92.313 s
T5LAG7.669 s69.774 s92.076 s
T6TurboServe1.520 s47.852 s97.633 s
T6A31.591 s53.671 s97.325 s
T6LAG1.688 s53.661 s97.622 s
  • Resume-to-first-chunk latency:
TraceSchedulerP50P99Max
T4TurboServe2.232 s23.446 s54.096 s
T4A32.142 s23.668 s54.002 s
T4LAG1.860 s23.643 s53.594 s
T5TurboServe2.553 s41.325 s99.182 s
T5A32.477 s41.341 s99.347 s
T5LAG2.364 s40.278 s99.958 s
T6TurboServe2.394 s38.715 s76.509 s
T6A32.324 s38.515 s76.498 s
T6LAG2.209 s38.973 s76.486 s

Results analysis. The first chunk is delayed mainly by queueing for a GPU during bursts, which all three schedulers share, so the tail is nearly identical even for LAG. The benefit of TurboServe appears after the first chunk, where rebalancing lowers the worst-case latency of the subsequent chunks.

Thank you for suggesting these metrics. We will add the first-chunk latency results to the revised version.


W2 + Q3. The benefit of the proposed control policy is not isolated, and A3 is absent from the main evaluation. How much does full TurboServe improve over tuned periodic rebalancing plus autoscaling, such as A3, using identical runtime primitives, scaling overheads, and streaming-latency accounting?

Comparison with A3. We ran additional experiments comparing TurboServe with A3 on 8 B300 GPUs and three 30-minute production traces at the same GPU cost, using the 1.3B model and identical runtime primitives, scaling overheads, and gap-inclusive latency accounting. The table below reports the worst-case per-chunk latency. TurboServe lowers it by 22.5% on average and by up to 32.4% compared with A3.

Trace (30 min)A3TurboServeReduction
T41.699 s1.454 s14.4%
T52.120 s1.684 s20.6%
T62.428 s1.642 s32.4%

Reason for the gain. TurboServe corrects imbalance at every session event, migrates only when the latency gain exceeds the transfer cost, and adapts its scaling target to workload volatility. In contrast, A3 rebalances only every 10 s, ignores the transfer cost when migrating, and scales with a fixed utilization target.

Thank you for the suggestion. We will add A3 to the main evaluation in the revised version.


We thank the reviewer once again for the careful and thorough review. We hope that the additional results and clarifications resolve the concerns raised, and we would be grateful if the reviewer would consider raising the score accordingly.

Thanks for your reply, most of my concerns are addressed, so I will raise the score.

Thank you for your constructive feedback and for taking the time to consider our response. We are pleased that most of your concerns have been addressed, and we sincerely appreciate your decision to raise the score.

Reviewer_knk5OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how to serve streaming video sessions as users pause, resume, and leave. TurboServe combines session migration with GPU autoscaling: migration balances the load across GPUs, while autoscaling adjusts capacity as demand changes. It builds on an existing runtime that batches chunks and offloads idle sessions to CPU memory. The authors evaluate it by replaying production traces with two model sizes on H20 and B300 clusters. Compared with round-robin, least-loaded, and least-memory placement, they report lower worst-case chunk latency at the same cost and lower cost at the same worst-case latency. Ablations show that both migration and autoscaling help reduce cost. The paper also compares the scheduler with oracles and measures the overhead of scheduling and migration.

Critical strengths
  1. The paper makes a convincing case for coordinating session placement and GPU scaling. A balanced placement can become uneven as users pause and resume generation, so scaling based on the current load alone may allocate more GPUs than needed. The motivating experiment helps explain the design: migration reduces latency, while combining it with autoscaling allows the system to release more GPUs and lower cost.

  2. The evaluation uses production-trace replays and shows consistent improvements across the tested model sizes and GPU clusters. The baselines share the same runtime, which helps isolate the benefits of the scheduling changes. The migration and autoscaling ablations show that both mechanisms contribute to cost efficiency, supporting the value of the combined system beyond a single favorable configuration.

Critical weaknesses
  1. The experiments show that migration and autoscaling are useful, but I am less convinced that they establish the benefit of the particular control policy. Figure 5 uses placement baselines, and Figure 6 disables one mechanism at a time. A3 already combines periodic rebalancing with autoscaling (Appendix C.2), yet the paper does not clearly explain how it differs from TurboServe or compare their performance. The small gaps to the oracles are encouraging, but a simpler policy might also achieve small gaps. A direct comparison with A3, after clarifying its implementation, on a few main workloads would help show what the proposed controllers add.

  2. I could not find how the scheduler computes the latency of a candidate placement. Section 3.2.1 uses this quantity to assign sessions and decide whether to migrate them, through L′L' in Eq. (3). The paper specifies a model for migration cost, but not for the expected latency after migration. It also does not report how KK, the per-GPU concurrency limit, is calibrated for the main experiments. Both quantities directly affect the decisions: whether a move is worthwhile and how many GPUs to provision. Without these details, it is difficult to reproduce the scheduler or judge how reliable its decisions are. Reporting the latency estimation method, the latency targets and KK values, and a comparison of predicted versus measured migration benefits would address this concern.

Questions for authors
  1. How does A3 differ from TurboServe in deciding which sessions to migrate and when to scale? Have you compared them directly under the matched-cost or matched-latency conditions in Figure 5? I would like to understand how much the specific policy adds beyond using migration and autoscaling together.

  2. How is the bottleneck latency estimated for a candidate placement, both when assigning a session and when computing L′L' in Eq. (3)? What latency targets and KK values did you use for each model and GPU configuration in the main experiments, and how did you choose them? Have you checked whether the predicted L′L' matches the bottleneck latency measured after migration?

3 comments

We thank the reviewer for the constructive comments. We answer each weakness together with its matching question.


W1 + Q1. The experiments show that migration and autoscaling are useful, but not the benefit of the particular control policy. A3 already combines periodic rebalancing with autoscaling, yet the paper does not explain how it differs from TurboServe or compare their performance. How does A3 differ in deciding which sessions to migrate and when to scale? Have you compared them directly under the matched-cost or matched-latency conditions in Figure 5?

How they differ. A3 and TurboServe use the same runtime primitives but differ in their control decisions, as shown in the table below.

DecisionA3TurboServe
When to migrateEvery 10 s, and around each scaling actionOn every arrival, departure, and active/idle transition
Which sessionsMinimizes the maximum per-GPU load, transfer cost not consideredOnly from the bottleneck GPU, only if latency gain exceeds transfer cost (Eq. 3)
When to scaleFollows current demand with a fixed utilization targetHysteresis band, proportional tracking, target adapted to workload volatility

Direct comparison. We ran additional experiments comparing A3 and TurboServe directly on 8 B300 GPUs using the 1.3B model, on three 30-minute production traces at the same GPU cost. As shown in the table below, TurboServe lowers the worst-case per-chunk latency by 22.5% on average and by up to 32.4% compared with A3. This is consistent with the differences above.

Trace (30 min)A3 worst-case latencyTurboServeReduction
T41.699 s1.454 s14.4%
T52.120 s1.684 s20.6%
T62.428 s1.642 s32.4%

Thank you for suggesting this comparison. We will add A3 to the main evaluation in the revised version.


W2 + Q2. The paper does not specify how the scheduler computes the latency of a candidate placement, nor report KK or the latency targets. How is the bottleneck latency estimated, both when assigning a session and when computing L′L' in Eq. (3)? What latency targets and KK values did you use, and how did you choose them? Have you checked whether the predicted L′L' matches the bottleneck latency measured after migration?

Estimation. In both cases, the bottleneck latency of a candidate placement is estimated as max⁡jf(nj)\max_j f(n_j), where njn_j is the number of sessions on GPU gjg_j under that placement and f(n)f(n) is the offline-profiled per-chunk latency of a GPU serving nn sessions.

Targets and KK. The latency target and KK for each model and GPU configuration are listed in the table below. The target is the playback duration of one chunk, and KK is the largest nn with f(n)≤f(n) \le target.

ModelGPULatency targetKK
1.3BH203 s4
7BH204.5 s4
1.3BB3003 s4
7BB3004.5 s4

How we chose KK. KK trades GPU utilization against latency. If KK is too small, the GPU is underutilized. If KK is too large, the per-chunk latency of each session exceeds the latency target. We therefore choose a KK that keeps the GPU well utilized while keeping the per-chunk latency within the target required for streaming video generation.

Predicted versus measured. The table below compares the predicted latencies for different sessions in a single prediction step with their measured latencies after migration. Across these 10 sessions, the mean absolute error is 0.046 s and the mean absolute percentage error is 9.5%. Although session 713’s interval is underestimated by 6.3%, the prediction still correctly identifies it as the slowest session and GPU 1 as the bottleneck. The predicted session ranking largely agrees with the measured ranking in this case.

SessionGPUPredictedMeasured
74400.2940.299
71310.6510.695
73710.6450.688
74620.2590.289
73330.3250.309
74340.3420.313
74150.3680.293
73660.3510.295
71670.5470.592
74270.5420.662

Thank you for raising these points. We will add the latency estimation method, the latency targets and KK values, and the predicted-versus-measured results to the revised version.


We will add all of these additional details and results to the revised version. We thank the reviewer again for the constructive and insightful comments.

Thank you for the response and additional experiments. They have resolved several of my concerns, so I will keep my rating as accept.

Thank you for your feedback and for taking the time to review our responses. Your suggestions have helped us improve the paper, and we sincerely appreciate your continued support for acceptance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.