acceptodds
Under review as a conference paper at ICLR 2027

FlashRT: Agent Harness for Guiding Agents to Deploy Real-Time Multimodal Applications

Abstract

Real-time multimodal applications, such as voice agents and interactive video generation, compose heterogeneous models into pipelines whose efficient deployment requires application-specific decisions about placement, streaming, and intra-model parallelism. Existing serving systems and auto-parallelism compilers commit to limited transformations and fixed workload assumptions, so each new application requires a hand-crafted implementation to achieve high performance. We present FlashRT, an agent harness that guides a generic coding agent to lift a simple developer-written reference implementation into optimized multi-GPU deployments that flexibly weigh target metrics such as latency and throughput. Following a new chain-of-program paradigm, FlashRT directs the agent to convert the reference into an intermediate representation (IR) that captures data dependencies and persistent state, validate the IR with a sequential interpreter, and statically analyze it to identify candidate transformations, which the agent then implements, verifies, and benchmarks in a measurement-gated optimization loop. Across five applications, including video world models and multimodal LLMs, FlashRT delivers up to 70 latency reduction and 2.8 throughput improvement on NVIDIA B200 GPUs and up to 3.6 throughput improvement on AMD MI355X GPUs, and it reduces Qwen3-Omni's response latency by up to 65% relative to the expert-built vLLM-Omni deployment.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
agentic systems, multimodal serving, inference optimization

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_AAHQOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents FlashRT, an agent harness for generating a high-performance inference engine for real-time multimodal applications. FlashRT implements a chain-of-program paradigm that directs the agent to work on an intermediate representation that captures dependencies between model components. Evaluation across five applications shows that FlashRT achieves up to ~70x latency reduction and 2.8x throughput improvement.

Critical strengths
  1. The paper proposes a chain-of-program to make it easier for agents to write a real-time and multimodal inference engine, which is a reasonable approach for this kind of problem.
Critical weaknesses
  1. The paper evaluates only the single-user case and does not consider the multi-user case. For the hardware setting (e.g., 8xB200), it is more common to serve muliple users concurrently, and that makes the system implementation much more complicated. It seems to me that the single-user case has very limited design space (e.g., compilation, sharding) and is not a very difficult implementation task.
  2. The paper is unclear about how to determine if streaming between model components is possible. Streaming can alter the output content depending on the model architecture, such as based on whether the attention layer uses a causal mask or not. Systems like FlashRT would need to reason about it.
Questions for authors
  1. How does FlashRT support workloads with concurrent users?
  2. How does FlashRT determine if streaming is possible between two graph nodes?
6 comments

We are grateful to the reviewer for their helpful comments and for raising some valuable questions about our work. We hope the following addresses the key concerns.

How does FlashRT support workloads with concurrent users?

The examples we have shown in the paper are indeed for single-session use cases, but there is nothing that prevents FlashRT from being used to optimize applications for concurrent requests.

We present an additional experiment in R.1 (Concurrent user evaluation) in the common response. In summary, we perform an independent run of FlashRT on the concurrent request setting for Qwen3-Omni, and the agent is given a general target of jointly optimizing per-request latency and aggregate throughput. Compared to vLLM-Omni (which also supports this multi-request setting), FlashRT’s best deployment achieves 5% higher throughput and 2.6x lower median time-to-first-output (TTFO) latency. This experiment demonstrates that the FlashRT agent can flexibly balance different deployment considerations, including throughput under concurrent user workloads.

For our other applications, serving concurrent users on the same GPUs is not practical. Unlike LLM decoding, which is memory-bound and gains throughput from batching, a video DiT processes a whole chunk of frames and is already compute-bound for one stream, so serving a second stream on the same GPUs would roughly halve each stream's frame rate. So for these models a node's multi-user capacity is set by single-stream efficiency, which is what our results show can be optimized by FlashRT.

It seems to me that the single-user case has very limited design space (e.g., compilation, sharding)

We respectfully disagree that single-user serving reduces to compilation and sharding.

First, we manually sweep different compilation and model placements and directly apply them to the reference implementation for the four video generation applications. The results are shown in R.8 (Manual configuration-only tuning) in the common response. In summary, compilation setting and model placement sweeps performed directly on a sequential reference implementation account for minimal gain, if any.

Second, our results in the paper demonstrate that the strongest gains come from fundamentally modifying the execution structure of the reference implementation, such as to enable streaming or to choose application-specific model parallelisms (e.g., pipeline parallelism within a single request). These cannot be generally reasoned about procedurally from an arbitrary reference implementation since their legality depends on application-specific semantics, and the existing ablation studies already demonstrate that missing these critical deployment opportunities can result in highly suboptimal deployments.

How does FlashRT determine if streaming is possible between two graph nodes?

Streaming is actually one of the key contributions of FlashRT, and it is an important motivating example for why we use an agentic approach: while a rule-based compiler will struggle to automatically apply streaming from an arbitrary prototype implementation (especially if it composes with third-party software), FlashRT employs an agent to reason about streaming legality. For example, the agent can identify that a causal mask enables streaming while a bidirectional mask requires blocking (something that a human would have to manually encode into a rule-based compiler on a per-model, per-attention-backend basis). Then, during the IR stage, the agent can organize the IR graph to express the pipeline operations in a streaming manner and explicitly mark each edge as streaming or non-streaming, with an associated streaming chunk size (which depends on the specific modules that the edge connects and which modality of data is being transferred).

The legality of the streaming is first determined when the agent validates its IR using the IR interpreter. When the interpreter runs the IR graph (sequentially in topological order), inputs will be fed to the graph pipeline in a chunked manner (to simulate real-time input), meaning that if the graph is structured to express streaming, then these streaming paths will be exercised by the interpreter; if streaming is illegal along a certain edge, the interpreter output will fail the required correctness check against the reference implementation, so the agent will revisit its IR to identify the issue. Furthermore, even if an illegal streaming edge improperly makes it past the IR validation, the agent will end up implementing a candidate that uses the illegal streaming behavior, and since each candidate must itself be validated against the reference implementation, the correctness check will fail due to invalid consumer computations induced by illegal streaming. The FlashRT harness then flexibly allows the agent to return to the IR stage to correct the IR as well as any downstream analysis or deployment candidates.

Thank you again for your time in the reviewing process. As the discussion period closes today, we wanted to check whether our response addressed your concerns about concurrent users and streaming legality. We have also uploaded a revised PDF (new text in blue) that adds the new results in Appendix G.1 and G.8 and explains in Section 4.2 how streaming legality is checked. We would be happy to clarify anything further.

Thank you very much for your rebuttal, and apologies for the late response.

I would like to push back on the following two points:

Unlike LLM decoding, which is memory-bound and gains throughput from batching, a video DiT processes a whole chunk of frames and is already compute-bound for one stream, so serving a second stream on the same GPUs would roughly halve each stream's frame rate.

While this should be true for video DiT models, I was not sure if all the components in other systems are compute-bound. For example, components like the ASR model and streaming VAE feature processing can be memory-bound. In general, for scenarios with concurrency >1, I think an important challenge is efficiently placing and scheduling both compute-bound and memory-bound components, which seems to be missing from the paper.

First, we manually sweep different compilation and model placements and directly apply them to the reference implementation for the four video generation applications. The results are shown in R.8 (Manual configuration-only tuning) in the common response. In summary, compilation setting and model placement sweeps performed directly on a sequential reference implementation account for minimal gain, if any.

Could you elaborate on the optimizations you tried for each scenario, including the torch.compile configuration, the granularity at which you applied CUDA graphs, the placement design space, and the attention kernels you tested? Moreover, how much of these implementations are already implemented in the baseline systems? While I understand that streaming would be the major contributor to the performance improvement, it was a bit surprising that these optimizations bring little performance improvement.

I apologize again for the late response. I understand that the timeline for the discussion period is very tight, but it would be very helpful to get further clarification about the points above.

Reviewer_RDEEOpenReview ↗
Rating
3
Confidence
1
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

FlashRT is a harness that guides a coding agent from a synchronous reference implementation to an optimized multi-GPU multimodal deployment. Its chain-of-program workflow first constructs a hierarchical IR with persistent-state and streaming annotations, then uses graph analysis and a measurement-driven implementation loop to explore placement, streaming, and parallelism. The evaluation covers five applications on NVIDIA B200 and AMD MI355X GPUs. It reports substantial latency and frame-rate improvements over sequential references, and lower Qwen3-Omni response latency than vLLM-Omni at the same three-GPU allocation. Ablations and repeated runs on LiveAvatar examine the roles of the IR and optimization loop.

Critical strengths
  1. The harness makes a useful connection between program structure and agent exploration. Persistent-state annotations expose when denoising steps can be pipelined, while streaming annotations identify overlap between models. The LiveAvatar ablation is informative: the loop without the IR finds full streaming but misses step-level pipeline parallelism; the IR without the loop finds that strategy but stops at an inefficient implementation. The final combination reaches 133.48 FPS versus 44.69 FPS for No Loop at the same six-GPU deployment size. This is a concrete example of the representation and the implementation loop contributing in different ways.

  2. The system produces working deployments across several applications and both accelerator platforms. The strongest comparison is Qwen3-Omni against an expert-built system with the same GPU count: latency falls from 0.433 to 0.323 s on B200 and from 0.779 to 0.276 s on MI355X, while all reported deployments retain RTF below one. The independent validation harness, kept separate from agent sessions, is also valuable. These results support practical utility beyond improvements over deliberately sequential references.

Critical weaknesses
  1. The evidence for the harness’s general advantage over a generic optimization agent is still narrow. The component ablations, three independent runs, and alternative agent configuration all use LiveAvatar. Other applications have one main run per hardware platform, and the ablations match the available GPUs but do not report matched optimization time or token budgets. A generic agent that is explicitly kept optimizing for the same budget is an important control for the claim that the structured IR and queue, rather than simply additional exploration, explain the gains. Also, the approximately 70x latency headline compares full-response buffering with first-frame streaming; Table 2 shows that No IR already obtains essentially the same latency. That number should therefore not be read as evidence for the full harness’s advantage over its ablations.

  2. Independent validation does not fully establish behavior preservation during unrestricted streaming execution. Appendix D provides useful fixed-input checks, but WorldPlay and Krea-Realtime use teacher forcing or diffusion-timestep forcing specifically to prevent error accumulation. These tests establish local numerical agreement, while the deployed systems feed their own generated states forward. The PDF does not report free-running quality or error growth over long streams, and its quantitative independent-validation table covers B200 deployments only. Since the system generates scheduling and state-management code, this gap matters to the claim that the optimized deployments preserve the reference behavior.

Questions for authors
  1. Harness attribution: Can you compare FlashRT with a persistent generic optimization loop under the same wall-clock or token budget, report costs for the ablations, and repeat the key comparison on one application other than LiveAvatar? This would help distinguish the IR’s contribution from additional search effort.
  2. Deployment correctness: Can you report independent free-running validation for WorldPlay and Krea-Realtime over multiple prompts and stream lengths, including an AMD deployment? Please distinguish tests of local numerical equivalence from tests of end-to-end output quality and state continuity.
  3. Latency and sustained service: For the principal deployments, can you report time to full response or stream-stall statistics alongside TTFO, and clarify how LiveAvatar’s theoretical sustainable FPS relates to measured frame delivery? This would make the practical meaning of the latency and throughput gains clearer.
3 comments

We appreciate the reviewer for their thorough and thoughtful feedback, and we hope the following will address their concerns.

Can you report independent free-running validation for WorldPlay and Krea-Realtime over multiple prompts and stream lengths

We use teacher-forcing and diffusion-timestep-forcing to perform framewise evaluation for WorldPlay and Krea-Realtime because small implementation differences (e.g., sequence parallelism causing different reduction orders) can cause numerical differences that diverge the output. However, this does not mean that the optimized implementation is invalid as the end-to-end output can still be high quality. As you suggest, we conduct a long-running quality validation without any teacher-forcing or diffusion-timestep-forcing. The results are shown in R.3 (Independent long-running quality evaluation) in the common response: in summary, FlashRT’s deployments closely match the long-horizon quality metrics produced by the reference implementation, including for the AMD deployments.

For the principal deployments, can you report time to full response … and clarify how LiveAvatar’s theoretical sustainable FPS relates to measured frame delivery

These additional results are reported in R.2 (End-to-end latency for LiveAvatar deployments) in the common response. The results show that streaming alone provides the 70x latency reduction since S2V pipeline-parallelism (PP) does not modify the critical path.

So the reviewer is correct in noting that the 70x latency reduction is also delivered by the No IR agent, but only FlashRT is able to jointly deliver the 70x latency reduction along with the highest observed frame rate in the same deployment. To clarify this, we will modify the language in the paper to state “FlashRT delivers up to 70x latency reduction while jointly sustaining the highest observed throughput”, rather than only stating “FlashRT delivers up to 70x latency reduction”.

In the paper, “theoretical frame rate” indeed means the measured frame rate. We just stated “theoretical” because the LiveAvatar model itself is trained to produce videos at a lower frame rate, so in a practical deployment the true playback rate will be clamped down to this value. So “theoretical” here just means that FlashRT’s best deployment can sustain real-time output for any version of the LiveAvatar model trained to produce videos at up to 133.48 FPS.

Can you compare FlashRT with a persistent generic optimization loop under the same wall-clock or token budget

Already in our ablation study, variants that do not use the IR (Naive and No IR) spend almost as much time if not more time than FlashRT.

VariantAPI costE2E time (h)
Naive$68.476.01
No Loop$44.233.57
No IR$70.404.92
FlashRT$166.984.99

Still, the API cost is not matched above, so we conduct an additional experiment: we run a version of the Naive agent (called “Persistent”) where, every time it stops, the agent is automatically prompted to continue exploring additional optimizations. The re-prompting is repeated until both the wall clock and API budget of the original FlashRT run are exceeded. The results are provided in R.4 (Matched budget ablation for LiveAvatar) in the common response. In summary, even the Persistent agent fails to identify key optimization opportunities (namely pipeline parallelism across denoising steps of the S2V model), resulting in substantially lower frame rate than FlashRT’s best deployment. Therefore, a generic optimization loop with extra search does not reproduce the benefits that FlashRT’s IR provides in helping identify key deployment candidates.

repeat the key comparison on one application other than LiveAvatar

We also repeat the comparison between FlashRT and the Persistent agent on the Krea-Realtime (video background editor) application. The results are provided in R.5 (Matched budget ablation for Krea-Realtime) in the common response. In summary, even on this additional application, the Persistent agent fails to identify key optimization candidates (namely DiT / VAE disaggregation), resulting in lower throughput than FlashRT’s best deployment. This provides further evidence that a generic optimization loop with extra search does not reproduce FlashRT’s benefits.

Thank you again for your time in the reviewing process. As the discussion period closes today, we wanted to check whether our response addressed your questions. We have also uploaded a revised PDF (new text in blue) with the new results in Appendix G.2 to G.5 and the "70x latency reduction" wording change. We would be happy to clarify anything further.

Dear Authors,

Thank you for the further explanation. I maintain my positive score.

Reviewer_XrYSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FlashRT is an agent harness that lifts a synchronous, single-GPU reference implementation of a real-time multimodal application into optimized multi-GPU deployments. It structures the agent's work with a chain-of-program paradigm: build a hierarchical IR with persistent-state and streaming annotations, validate it with a sequential interpreter, derive candidate transformations via static analysis, and refine implementations in a measurement-gated variant queue. Evaluation spans five applications on NVIDIA B200 and AMD MI355X, with formal latency/throughput analysis in the appendix.

Critical strengths
  1. The paper brings unusual analytical rigor to agentic systems work: the deployment problem is formalized as a task graph with intra-/cross-batch edges, latency and throughput are given clean operational definitions, NP-hardness is established, and provable trade-offs (co-location minimizes latency, disaggregation minimizes period; SP vs. stage-PP) are derived and shown to match the measured deployments. The bounds on non-agentic baselines also give the reader a principled yardstick for what the agent achieves.

  2. It extends agentic optimization from kernel- or file-level edits to full end-to-end pipeline engineering — the agent produces working multi-process, multi-GPU deployments with NCCL communication and streaming coordination — and does so at a transparently reported cost ($75–167 and a few hours per application), which makes the economic argument for replacing expert effort concrete.

  3. The reliability evidence is stronger than typical for this literature: three independent runs converge to the same deployment family on the same exploration path, a second frontier coding agent reproduces the result under an unchanged prompt, and re-optimization on AMD without hardware hints recovers the same deployment classes.

Critical weaknesses
  1. The claim of application diversity is broader than the evidence. Four of the five applications reduce to the same archetype (an autoregressive DiT followed by a streaming VAE, possibly with one independent side branch), and one is a three-stage LLM chain; the transformations exercised (streaming, SP, PP, co-location vs. disaggregation) form a small recurring set. Generality to genuinely different structures — bidirectional-attention models where chunking is unsafe, pipelines with feedback or variable-rate components — is untested.

  2. The results are tightly coupled to proprietary frontier models (Claude Code + Opus 4.8, with a single Codex/GPT-5.6 trial). Because these models are not reproducible by third parties and evolve over time, the demonstrated gains are confounded with the base agent's general coding ability, and the paper's own disclaimer that it does not claim model-agnosticism limits how much the community can build on or verify the approach.

  3. For four of the five applications, the comparison set contains only the deliberately sequential single-GPU reference and analytical bounds; the single expert-built baseline (vLLM-Omni) provides just one data point. The magnitude of improvement relative to practical state-of-the-art engineering therefore remains unknown for most of the reported workloads.

Questions for authors

Can you provide more baseline comparisons, for example, some other coding agents or harnesses.

4 comments

We thank the reviewer for their careful assessment and constructive questions, and we hope the following addresses the main concerns.

The claim of application diversity is broader than the evidence. Four of the five applications reduce to the same archetype

The reviewer is correct that four of the applications share a DiT-VAE core, but their task graphs differ in key ways that change which deployment is right. For example:

  • Video background editor: two independent branches join at the output, so giving each branch its own set of GPUs improves both metrics, as Appendix B.5 predicts.
  • LiveAvatar and LongLive: both applications contain variable-rate components (ASR, LLM, etc.), which often require careful scheduling decisions to distinguish blocked from streaming edges.
  • WorldPlay contains cross-chunk feedback: generated latents are re-encoded into the DiT's KV cache every chunk.
  • Qwen3-Omni involves a three-model chain with rate conversion and no DiT.
  • Components with bidirectional attention keep blocking edges, as discussed in our response to Reviewer AAHQ.

In addition, we conduct another experiment on a multi-user serving case for Qwen3-Omni, as provided in R.1 (Concurrent user evaluation) in the common response. This presents a new workload setting for FlashRT previously not explored. In summary, FlashRT’s best deployment achieves 5% higher throughput and 2.6x lower median time-to-first-output (TTFO) latency compared to vLLM-Omni (which also supports this multi-request setting). This experiment demonstrates that the FlashRT agent can flexibly balance different deployment considerations, including throughput under concurrent user workloads.

The results are tightly coupled to proprietary frontier models

To alleviate this concern, we conduct an additional trial with Kimi K3.1 Pro on the LiveAvatar application. The results are provided in R.6 (Further model ablation) in the common response. In summary, we observe that, with FlashRT, Kimi’s attempted deployments are fairly consistent with those written by Opus 4.8 in our original experiments. FlashRT enhances a model’s capabilities to discover key optimization techniques, and the same benefits extend to weaker open-weight models based on this experiment. However, as explained in the common response, the quality of the final deployments still depends on the model’s native programming ability, supporting our statement that FlashRT is not model-agnostic.

For four of the five applications, the comparison set contains only the deliberately sequential single-GPU reference and analytical bounds

No existing non-agent system deploys our LiveAvatar, WorldPlay, Krea-Realtime, or LongLive applications, so we cannot compare against one directly. Cornserve, vLLM-Omni, and ModServe do not fit our setting since they take a fundamentally different input than FlashRT: while FlashRT takes as input any arbitrary reference program (even if it composes with third-party software), these existing systems only accept a graph of stages that a developer writes manually. Therefore, their performance relies on the developer’s expertise in organizing the graph in a useful manner for automatic transformations. FlashRT’s main benefit is that it eliminates reliance on this expertise.

Still, we conduct an approximate comparison: typically, developers will write graph stages on the granularity of full models, i.e., each model forward is a node. This decomposition does not always admit the best deployment candidates, as we show in R.7 (Approximate comparison to non-agentic baselines) in the common response. This strengthens one of our claims in the paper, which is that FlashRT’s key benefit is that it can decompose a pipeline at heterogeneous levels of granularity, which (a) keeps the search tractable despite the problem's NP-hardness, while also (b) allowing the IR graph to be able to express key optimization opportunities that would otherwise be hidden when using a uniform model-level granularity.

Furthermore, as an approximation of configuration autotuning, we manually sweep different compilation and model placements and directly apply them to the reference implementation for the four video generation applications. The results are shown in R.8 (Manual configuration-only tuning) in the common response. In summary, compilation setting and model placement sweeps performed directly on a sequential reference implementation account for minimal gain, if any.

Can you provide more baseline comparisons, for example, some other coding agents or harnesses

We provide additional baselines in R.4 (Matched budget ablation for LiveAvatar) and R.5 (Matched budget ablation for Krea-Realtime) in the common response. These experiments run a version of the Naive agent (called “Persistent”) where, every time it stops, the agent is automatically prompted to continue exploring additional optimizations. The re-prompting is repeated until both the wall clock and API budget of the original FlashRT run are exceeded. In summary, even the Persistent agent fails to identify key optimization opportunities, resulting in suboptimal throughput compared to FlashRT’s best deployment for both the LiveAvatar and Krea-Realtime applications. The result shows that a generic optimization loop with extra search does not reproduce the benefits that FlashRT’s IR provides in helping identify key deployment candidates.

Other coding agents are ablated in R.6 (Further model ablation) as discussed previously.

Thank you again for your time in the reviewing process. As the discussion period closes today, we wanted to check whether our response addressed your concerns. We have also uploaded a revised PDF (new text in blue) with the new results in Appendix G.1 and G.4 to G.8. We would be happy to clarify anything further.

Thanks for the authors' rebuttal. Most of my concerns are properly addressed. I will keep my score and vote for acceptance of this paper.

Reviewer_ttpwOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

FlashRT is an agent harness that automatically lifts a synchronous single-GPU reference implementation into an optimized multi-GPU deployment for real-time multimodal applications. It contributes (1) chain-of-program — a hierarchical dataflow IR with persistent-state annotations, validated by a sequential interpreter and statically analyzed to derive legal transformations — and (2) a measurement-gated validation loop that implements, equivalence-checks, and benchmarks candidate transformations in a queue until no profitable one remains. Validation on five applications (video world models, multimodal LLMs, talking avatar, voice agent) reports up to ~70x lower time-to-first-output and 2.8x (B200) / 3.6x (MI355X) throughput versus reference, plus up to 65% lower Qwen3-Omni latency versus the expert-built vLLM-Omni deployment; ablations reproduce the two failure modes of naive prompting. Validation uses five applications (LiveAvatar face-to-face conversational agent, Qwen3-Omni multimodal LLM, Krea-Realtime + SAM 3 video background editor, WorldPlay and LongLive video world models) on an 8x B200 node and an 8x MI355X node, driven by Claude Code with Opus 4.8 (plus a Codex/GPT-5.6 Sol cross-check and three independent runs on one application). Reported results: up to ~70x TTFO latency reduction and 2.8x throughput on B200, up to 3.6x throughput on MI355X, 25% (B200) and 65% (MI355X) lower latency than the expert-built vLLM-Omni deployment of Qwen3-Omni at RTF < 1, an ablation showing each of the two components is necessary, an independent correctness check against the references (PSNR/MAE thresholds), and a cost of $75-167 and a few hours of agent time per application.

Critical strengths

Strength 1 - The gains are demonstrated against a strong, non-trivial baseline, and they generalize across hardware and across the underlying agent. Strength 2 - The two design principles are separately justified, by an ablation that maps onto the observed failure modes and by formal analysis that explains why the problem cannot be solved by a fixed policy.

Critical weaknesses

Weakness 1 - The headline numbers are measured against a weak reference, reported only as time-to-first-output, and are not compared empirically against the non-agent baselines the paper itself argues against. This is the weakness that most limits my ability to endorse the central claim. Weakness 2 - Statistical strength of the evaluation, and the gap between the claimed behavior preservation and what is actually validated.

Questions for authors

Q1 (clarification). Please report time-to-last-output (total response time) alongside TTFO for the LiveAvatar deployments. Under an equal-output-length comparison, how much of the ~70x comes from the semantic change to streaming rather than from placement/parallelism? Q2 (clarification). Can you add an end-to-end measured comparison against at least one non-agent baseline - Cornserve or ModServe, or an auto-parallelism framework plus configuration autotuning - on one application? Appendix B.5's analytic argument currently carries the whole claim that non-agent automation cannot reach these points.

2 comments

We thank the reviewer for their thoughtful consideration of this work, and we hope the following clarifications and additional experiments address the main concerns.

Please report time-to-last-output (total response time) alongside TTFO for the LiveAvatar deployments.

We have reported these results in R.2 (End-to-end latency for LiveAvatar deployments) in the common response.

Under an equal-output-length comparison, how much of the ~70x comes from the semantic change to streaming rather than from placement/parallelism?

The results from R.2 show that the 70x latency reduction does not generally come from placement/parallelism but instead primarily comes from the semantic change to streaming. This demonstrates a critical feature of FlashRT: that it can fundamentally modify the execution structure of the original reference program to deliver substantial gains. Meanwhile, traditional rule-based approaches are typically unable to correctly derive such opportunities and instead remain limited to placement and intra-stage parallelism.

Can you add an end-to-end measured comparison against at least one non-agent baseline

No existing non-agent system deploys our LiveAvatar, WorldPlay, Krea-Realtime, or LongLive applications, so we cannot compare against one directly. Cornserve, vLLM-Omni, and ModServe do not fit our setting since they take a fundamentally different input than FlashRT: while FlashRT takes as input any arbitrary reference program (even if it composes with third-party software), these existing systems only accept a graph of stages that a developer writes manually. Therefore, their performance relies on the developer’s expertise in organizing the graph in a useful manner for automatic transformations. FlashRT’s main benefit is that it eliminates reliance on this expertise.

Still, we conduct an approximate comparison: typically, developers will write graph stages on the granularity of full models, i.e., each model forward is a node. This decomposition does not always admit the best deployment candidates, as we show in R.7 (Approximate comparison to non-agentic baselines) in the common response. This strengthens one of our claims in the paper, which is that FlashRT’s key benefit is that it can decompose a pipeline at heterogeneous levels of granularity, which (a) keeps the search tractable despite the problem's NP-hardness, while also (b) allowing the IR graph to be able to express key optimization opportunities that would otherwise be hidden when using a uniform model-level granularity.

Furthermore, as an approximation of configuration autotuning, we manually sweep different compilation and model placements and directly apply them to the reference implementation for the four video generation applications. The results are shown in R.8 (Manual configuration-only tuning) in the common response. In summary, compilation setting and model placement sweeps performed directly on a sequential reference implementation account for minimal gain, if any.

Statistical strength of the evaluation, and the gap between the claimed behavior preservation and what is actually validated

Our previous experiments demonstrate that FlashRT finds the same key deployments across independent runs on the same application, as discussed in Section 5.6. Furthermore, in Appendix E we find that, not only is the final deployment consistent, but the search path used by FlashRT agent to arrive at the final deployment also remains fairly consistent between these independent runs.

While the FlashRT agent is required to maintain correctness of its deployments, we also independently evaluate the deployment correctness and find that the independent evaluation succeeds on the key deployments, as discussed in Appendix D. To strengthen this evaluation, we conduct a long-running quality validation on video generation applications: the results are provided in R.3 (Independent long-running quality evaluation) in the common response.

Thank you again for your time in the reviewing process. As the discussion period closes today, we wanted to check whether our response addressed your questions. We have also uploaded a revised PDF (new text in blue) with the new results in Appendix G.2, G.3, G.7 and G.8. We would be happy to clarify anything further.

Comments on the paper

6 comments

Common response

We thank all the reviewers for their feedback. Here we provide a set of new experiments we conducted, which we will refer to in the responses for each individual review.

R.1: Concurrent user evaluation

The examples we have shown in the paper are for single-session use cases. To demonstrate a concurrent user example, we perform an independent run of FlashRT on the Qwen3-Omni application and instruct the agent to jointly target multi-user throughput and response latency. We use the same settings as in the paper (Claude Code with Opus 4.8 at max reasoning effort, run completely fresh with no prior information provided). To match vLLM-Omni’s GPU budget, we provide the agent with 3 GPUs.

To evaluate the throughput, we follow the same benchmarking protocol already implemented by vLLM-Omni: using the first 640 prompts from the Seed-TTS [1] English test set, send 64 requests to the backend and, any time a request completes, send another request so that 64 requests are always in flight, until all 640 requests are exhausted. Note that this benchmarking protocol was not provided to the FlashRT agent ahead of time, and the agent was instead instructed generically to optimize for latency/throughput. We compare FlashRT’s best produced deployment against vLLM-Omni’s standard deployment.

Note that, due to limited hardware access, this experiment is conducted on H200 rather than B200. Should our paper be accepted, we will redo the experiment on B200 (once we regain access) and report those results in the camera-ready version.

Requests / secAudio sec / real-time secMedian TTFO (s)P95 TTFO (s)
FlashRT12.4352.91.568.38
vLLM-Omni11.7150.43.987.69

As shown in the table above, FlashRT’s deployment achieves both higher throughput and lower median time-to-first-output (TTFO) latency, with only slightly higher p95 latency. FlashRT identifies the Talker LLM as the bottleneck and reduces its latency with TP-2, and it properly balances the audio codec chunk size: too low of a chunk size results in low hardware utilization, and too high of a chunk size limits concurrency between the Talker and Vocoder. Meanwhile, vLLM-Omni’s deployment places the Talker on a single GPU without model parallelism and uses a higher audio codec chunk size. Therefore, this experiment demonstrates that the FlashRT agent can flexibly balance different deployment considerations, including throughput under concurrent user workloads.

For our other applications, serving concurrent users on the same GPUs is not practical. Unlike LLM decoding, which is memory-bound and gains throughput from batching, a video DiT processes a whole chunk of frames and is already compute-bound for one stream, so serving a second stream on the same GPUs would roughly halve each stream's frame rate. So for these models a node's multi-user capacity is set by single-stream efficiency, which is what our results show can be optimized by FlashRT.

[1] Anastassiou et al. 2024. Seed-TTS: A Family of High-Quality Versatile Speech Generation Models.

R.2: End-to-end latency for LiveAvatar deployments

The table below shows time-to-last-output (TTLO) latency alongside time-to-first-output (TTFO) and frame-rate for the key LiveAvatar deployments from the runs reported in the paper (these numbers are from the original runs themselves, collected on B200). As shown, using pipeline parallelism (PP) on the S2V model significantly boosts throughput: the combined streaming and PP deployment achieves 4.1x lower TTLO and 4.3x higher frame rate compared to the streaming-only deployment.

Deployment# GPUsTTFO (s)TTLO (s)Frame rate (FPS)
Reference1117.10117.10 (single output burst)–
Naive (best: S2V output streaming)210.45116.1230.99
No IR (best: full streaming)21.68108.4830.67
No Loop (best: full streaming + S2V PP)62.3775.7044.69
FlashRT (full streaming)21.69107.6230.92
FlashRT (S2V PP-only)636.2936.29 (single output burst)–
FlashRT (best: streaming + S2V PP)61.7226.26133.48

R.3: Independent long-running quality evaluation

In addition to the independent frame-level quality evaluations we provide in Appendix D, here we conduct an additional long-running quality validation. As the LongLive and LiveAvatar applications are already validated on frame-level error without teacher forcing in Appendix D, we perform this evaluation on the WorldPlay and Krea-Realtime (video background editor) applications. We do this using VBench, a standard video quality estimation benchmark, for various deployments.

Below is a table for WorldPlay, where we collect 48-second videos using 4 input scenes and 3 randomly generated action sequences per scene. Below we observe that both the CUDA and ROCm deployments remain close in VBench quality score (sometimes even slightly higher), which demonstrates that free-running quality is maintained by these deployments. To further confirm that quality maintained over the full horizon, we separately compute the quality score for the first half and second half of the videos: as shown, the deployments closely match the reference in quality score for both halves; the second half scores degrades from the first half scores, but this occurs in the reference as well, so the degradation is due to the generation quality of the model itself rather than the deployment.

PlatformDeployment# GPUsVBench Quality Score (full)VBench Quality Score (first half)VBench Quality Score (second half)
–Reference179.9784.6675.27
CUDALatency optimized279.9584.4975.40
CUDAFrame-rate optimized279.9984.6875.29
CUDALatency optimized479.7884.3975.17
CUDAFrame-rate optimized479.9584.4975.40
CUDAFrame-rate + latency optimized679.7784.3975.16
ROCmLatency optimized279.9684.4975.43
ROCmFrame-rate optimized280.0384.6975.36
ROCmLatency optimized480.0384.4875.58

Below is the same evaluation on Krea-Realtime, where we sweep across 7 different talking-head videos, each on two different style prompts. Again we observe the VBench quality score closely matches the reference, including the full video and the independent first/second evaluations (the Krea-Realtime quality does not degrade in the second half).

PlatformDeployment# GPUsVBench Quality Score (full)VBench Quality Score (first half)VBench Quality Score (second half)
–Reference184.8584.9684.74
CUDAParallel284.4084.4384.37
CUDALatency optimized484.7084.9984.41
CUDAFrame-rate optimized584.6685.0884.24
ROCmLatency optimized484.7885.1984.38
ROCmFrame-rate optimized584.6284.8984.36

R.4: Matched budget ablation for LiveAvatar

To augment our ablation study in Section 5.5, we supply an additional “matched budget” ablation that we call the Persistent agent: this is a version of the Naive agent where, every time it stops, the agent is automatically prompted to continue exploring additional optimizations. The re-prompting is repeated until both the wall clock and API budget of the original FlashRT run are exceeded. This puts FlashRT at a slight disadvantage when compared to Persistent, but it allows us to more clearly distinguish the benefits of FlashRT from simply using extra search effort.

The results are shown in the table below, on the LiveAvatar application. Note that, due to limited hardware access, this experiment is conducted on H200 rather than B200. Should our paper be accepted, we will redo the experiment on B200 (once we regain access) and report those results in the camera-ready version.

AgentBest Deployment# GPUsTTFO (s)TTLO (s)Frame rate (FPS)E2E time (h)API Cost
–Reference1158.18158.18 (single output burst)–––
PersistentStreaming + S2V SP-351.8187.1033.615.27$225.87
FlashRTStreaming + S2V PP62.3736.1085.034.99$166.98

In the run above, we stopped re-prompting the Persistent agent once it had exceeded the wall-clock budget, during which it used 1.35x the API cost. Like the No IR ablation, the Persistent agent discovered streaming to provide substantial latency gains, but the agent failed to discover S2V PP and instead deployed the S2V model across three GPUs using sequence parallelism (SP). The Persistent agent used the additional time and tokens ineffectively by trying to increase the SP degree (which did not provide any additional benefit due to overhead) and trying to approximate the S2V model with fewer diffusion steps (which the agent itself identified as invalid due to failing the correctness requirement). Ultimately, both agents reduce TTFO to about 2 s, but the Persistent agent’s best deployment has a 2.5x lower frame rate and 2.4x higher TTLO than FlashRT’s. This demonstrates that, even under matched wall-clock and API budgets, a generic optimization loop with extra search does not reproduce the benefits that the IR provides in helping identify key deployment candidates.

R.5: Matched budget ablation for Krea-Realtime (video background editor)

We repeat the matched-budget ablation experiment on the Krea-Realtime application to demonstrate that the gains of FlashRT extend beyond the LiveAvatar application. The results are shown in the table below. Again, the “matched-budget” Persistent agent is re-prompted to continue every time it stops until both the wall clock and API cost budgets of the FlashRT run are exceeded.

Again the results are collected on H200 rather than B200 due to limited hardware access, so we will provide B200 results in a camera-ready version should our paper be accepted.

AgentDeployment# GPUsLatency (s)Frame rate (FPS)E2E time (h)API Cost
–Reference12.504.76––
PersistentSP-340.9111.793.44$81.46
FlashRTDiT SP-3 + disagg. VAE50.9614.852.64$74.53

Based on the table above, the Persistent agent and FlashRT achieve essentially the same latency, but FlashRT achieves 1.3x higher frame rate than the Persistent agent. This is because, while both agents place the SAM 3 model on a separate GPU, only FlashRT identifies the VAE as an independent stage from the DiT while constructing the IR, so the agent deploys it in a disaggregated manner on a separate GPU, which allows boosting throughput without noticeably affecting latency. Therefore, even under matched budgets, a generic optimization loop with extra search is unable to reproduce the benefits of FlashRT, including on this additional application.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.