- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper presents FlashRT, an agent harness for generating a high-performance inference engine for real-time multimodal applications. FlashRT implements a chain-of-program paradigm that directs the agent to work on an intermediate representation that captures dependencies between model components. Evaluation across five applications shows that FlashRT achieves up to ~70x latency reduction and 2.8x throughput improvement.
Critical strengths
- The paper proposes a chain-of-program to make it easier for agents to write a real-time and multimodal inference engine, which is a reasonable approach for this kind of problem.
Critical weaknesses
- The paper evaluates only the single-user case and does not consider the multi-user case. For the hardware setting (e.g., 8xB200), it is more common to serve muliple users concurrently, and that makes the system implementation much more complicated. It seems to me that the single-user case has very limited design space (e.g., compilation, sharding) and is not a very difficult implementation task.
- The paper is unclear about how to determine if streaming between model components is possible. Streaming can alter the output content depending on the model architecture, such as based on whether the attention layer uses a causal mask or not. Systems like FlashRT would need to reason about it.
Questions for authors
- How does FlashRT support workloads with concurrent users?
- How does FlashRT determine if streaming is possible between two graph nodes?
6 comments
Thank you again for your time in the reviewing process. As the discussion period closes today, we wanted to check whether our response addressed your concerns about concurrent users and streaming legality. We have also uploaded a revised PDF (new text in blue) that adds the new results in Appendix G.1 and G.8 and explains in Section 4.2 how streaming legality is checked. We would be happy to clarify anything further.
Thank you very much for your rebuttal, and apologies for the late response.
I would like to push back on the following two points:
Unlike LLM decoding, which is memory-bound and gains throughput from batching, a video DiT processes a whole chunk of frames and is already compute-bound for one stream, so serving a second stream on the same GPUs would roughly halve each stream's frame rate.
While this should be true for video DiT models, I was not sure if all the components in other systems are compute-bound. For example, components like the ASR model and streaming VAE feature processing can be memory-bound. In general, for scenarios with concurrency >1, I think an important challenge is efficiently placing and scheduling both compute-bound and memory-bound components, which seems to be missing from the paper.
First, we manually sweep different compilation and model placements and directly apply them to the reference implementation for the four video generation applications. The results are shown in R.8 (Manual configuration-only tuning) in the common response. In summary, compilation setting and model placement sweeps performed directly on a sequential reference implementation account for minimal gain, if any.
Could you elaborate on the optimizations you tried for each scenario, including the torch.compile configuration, the granularity at which you applied CUDA graphs, the placement design space, and the attention kernels you tested? Moreover, how much of these implementations are already implemented in the baseline systems? While I understand that streaming would be the major contributor to the performance improvement, it was a bit surprising that these optimizations bring little performance improvement.
I apologize again for the late response. I understand that the timeline for the discussion period is very tight, but it would be very helpful to get further clarification about the points above.
We are grateful to the reviewer for their helpful comments and for raising some valuable questions about our work. We hope the following addresses the key concerns.
The examples we have shown in the paper are indeed for single-session use cases, but there is nothing that prevents FlashRT from being used to optimize applications for concurrent requests.
We present an additional experiment in R.1 (Concurrent user evaluation) in the common response. In summary, we perform an independent run of FlashRT on the concurrent request setting for Qwen3-Omni, and the agent is given a general target of jointly optimizing per-request latency and aggregate throughput. Compared to vLLM-Omni (which also supports this multi-request setting), FlashRT’s best deployment achieves 5% higher throughput and 2.6x lower median time-to-first-output (TTFO) latency. This experiment demonstrates that the FlashRT agent can flexibly balance different deployment considerations, including throughput under concurrent user workloads.
For our other applications, serving concurrent users on the same GPUs is not practical. Unlike LLM decoding, which is memory-bound and gains throughput from batching, a video DiT processes a whole chunk of frames and is already compute-bound for one stream, so serving a second stream on the same GPUs would roughly halve each stream's frame rate. So for these models a node's multi-user capacity is set by single-stream efficiency, which is what our results show can be optimized by FlashRT.
We respectfully disagree that single-user serving reduces to compilation and sharding.
First, we manually sweep different compilation and model placements and directly apply them to the reference implementation for the four video generation applications. The results are shown in R.8 (Manual configuration-only tuning) in the common response. In summary, compilation setting and model placement sweeps performed directly on a sequential reference implementation account for minimal gain, if any.
Second, our results in the paper demonstrate that the strongest gains come from fundamentally modifying the execution structure of the reference implementation, such as to enable streaming or to choose application-specific model parallelisms (e.g., pipeline parallelism within a single request). These cannot be generally reasoned about procedurally from an arbitrary reference implementation since their legality depends on application-specific semantics, and the existing ablation studies already demonstrate that missing these critical deployment opportunities can result in highly suboptimal deployments.
Streaming is actually one of the key contributions of FlashRT, and it is an important motivating example for why we use an agentic approach: while a rule-based compiler will struggle to automatically apply streaming from an arbitrary prototype implementation (especially if it composes with third-party software), FlashRT employs an agent to reason about streaming legality. For example, the agent can identify that a causal mask enables streaming while a bidirectional mask requires blocking (something that a human would have to manually encode into a rule-based compiler on a per-model, per-attention-backend basis). Then, during the IR stage, the agent can organize the IR graph to express the pipeline operations in a streaming manner and explicitly mark each edge as streaming or non-streaming, with an associated streaming chunk size (which depends on the specific modules that the edge connects and which modality of data is being transferred).
The legality of the streaming is first determined when the agent validates its IR using the IR interpreter. When the interpreter runs the IR graph (sequentially in topological order), inputs will be fed to the graph pipeline in a chunked manner (to simulate real-time input), meaning that if the graph is structured to express streaming, then these streaming paths will be exercised by the interpreter; if streaming is illegal along a certain edge, the interpreter output will fail the required correctness check against the reference implementation, so the agent will revisit its IR to identify the issue. Furthermore, even if an illegal streaming edge improperly makes it past the IR validation, the agent will end up implementing a candidate that uses the illegal streaming behavior, and since each candidate must itself be validated against the reference implementation, the correctness check will fail due to invalid consumer computations induced by illegal streaming. The FlashRT harness then flexibly allows the agent to return to the IR stage to correct the IR as well as any downstream analysis or deployment candidates.