- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2(+1)
Read review
Paper summary
PhyAI is a unified inference runtime for robot policies spanning VLA and world action models. It separates model-specific scheduling and execution state from shared operators, uses a capability- and priority-based kernel dispatcher, and supports DP, TP, and classifier-free guidance parallelism. CUDA Graph replay, kernel fusion, and cache precomputation accelerate five model families across edge and server GPUs. The paper reports 1.40–4.65× model-runner speedups over official implementations, near-linear replica scaling, and a 1.17× overall RL-step speedup. It also introduces a Control Rate Roofline that relates request latency to a fixed action-execution window under sequential execution or ideal overlap. Matched LIBERO evaluations provide behavioral validation for π0.5 and GR00T N1.7.
Critical strengths
-
The system integrates a useful breadth of models and deployment modes. Table 2 covers five model families, including a WAM using parallism across H20 GPUs. Figure 7 extends the evaluation to replica scaling and an RL workflow, where overall step time falls from 772.45 to 657.79 seconds. This demonstrates practical integration beyond isolated kernel benchmarks.
-
The stage-level batching analysis provides useful deployment insights. Section 5.3 shows that batching shifts π0.5's bottleneck toward the VLM, identifies diminishing throughput returns around batch 8 for GR00T, and explains Cosmos3's limited batching gains through its already compute-intensive generation stage. These findings help identify where batching is useful and where it primarily increases latency.
Critical weaknesses
-
The edge-performance advantage is unclear. Figure 2 shows no consistent latency advantage over FlashRT on overlapping configurations. The headline speedups are measured against weaker official implementations rather than established optimized runtimes, making them insufficient evidence of an advance over existing systems.
-
The benefit of the unified abstraction is insufficiently demonstrated. Table 1 lists both PhyAI and FlashRT as supporting VLA and WAM models across edge and cloud. The paper does not demonstrate broader model coverage or quantify reduced integration effort, leaving the value of its abstraction beyond parallel-execution support unclear.
-
Static batching gains are not validated under actual cloud workloads. Section 5.3 usefully identifies diminishing returns, but excludes batch formation and queueing. Without request arrival patterns or latency budgets, it remains unclear whether beneficial batch sizes can be formed in practice or how PhyAI selects them.
-
The cloud results do not establish gains beyond FlashRT-style inference acceleration. The demonstrated benefits center on graph replay, kernel fusion, and caching—the same class of optimizations used for edge inference. Although PhyAI supports DP/TP/CFG, the experiments do not establish an advantage over conventional implementations of these parallelism strategies, such as lower communication overhead, better computation–communication overlap, or fewer scheduling bubbles. The RLinf comparison likewise reports faster prediction without demonstrating orchestration or scheduling gains. In this experiment, “Other time” still accounts for 61.1% of the optimized RL step, nearly six times the prediction time. Without a breakdown of simulation, rendering, communication, and synchronization costs, the evaluation neither identifies the dominant bottleneck nor demonstrates that PhyAI addresses it.
Questions for authors
-
What accounts for the approximately 41-second reduction in non-prediction time in the RLinf experiment?
-
What concrete capability or performance benefit does PhyAI provide over FlashRT on their overlapping models? Does the unified abstraction enable additional architectures or deployment modes, or reduce integration effort in a way the authors can substantiate?
-
How does PhyAI select batch sizes and waiting limits under actual request workloads, and what throughput–latency benefits remain after batch formation and queueing? For cloud deployments serving multiple models concurrently, how does the runtime allocate GPU resources and coordinate per-model batching and scheduling, given their different memory demands and latency requirements?
-
Compared with conventional DP/TP/CFG implementations on matched hardware, does PhyAI reduce communication overhead, improve computation–communication overlap, or reduce pipeline bubbles? Can these benefits be quantified?
8 comments
Q3: Static batching, queueing, and multi-model scheduling
How does PhyAI select batch sizes and waiting limits under actual request workloads
In the fixed robot deployments and batch simulation rollouts considered in this paper, the number of robots or environments and the task configuration are typically fixed in advance, and requests arrive primarily at the pace of the control loop or the rollout, keeping the load relatively stable. PhyAI therefore currently employs static batching. Batch size is configured ahead of time by the user based on available hardware resources and the latency budget. It neither forms batches dynamically from online requests nor offers policies for choosing a batching wait-time cap. Section 5.3 reports throughput and inference latency measured at given batch sizes, providing a basis for selecting the configuration for these deployments.
All configurations use BF16. Each cell reports full-batch inference latency (ms) / throughput (action chunks/s).
| Batch size | ,H200 | GR00T,RTX 5090 | Cosmos3,H200 |
|---|---|---|---|
| 1 | 22.59 / 44.26 | 20.49 / 48.80 | 1132.13 / 0.883 |
| 2 | 31.52 / 63.46 | 27.64 / 72.40 | 2251.90 / 0.888 |
| 4 | 50.95 / 78.51 | 37.86 / 105.70 | 4157.16 / 0.962 |
| 8 | 89.37 / 89.51 | 62.49 / 128.00 | 8026.96 / 0.997 |
| 16 | 168.85 / 94.76 | 118.32 / 135.20 | 15846.72 / 1.010 |
What throughput–latency benefits remain after batch formation and queueing?
We evaluate RL rollout, where batches are defined in advance from parallel environments. Section 5.3 reports model-runner throughput and latency for these preformed, excluding batch formation and queueing. These measurements characterize batched inference efficiency. For Throughput and end-to-end request latency under online arrivals were not evaluated in this experiment.
For cloud deployments serving multiple models concurrently, how does the runtime allocate GPU resources and coordinate per-model batching and scheduling, given their different memory demands and latency requirements?
Multi-model support in PhyAI means that the same inference infrastructure can execute different VLA/WAM architectures. Deployment settings specify the replica groups and the parallel modes supported by each model. The scheduling mechanisms discussed here route requests among complete model replicas and coordinate execution within the configured parallel groups. Multiple robots can be served by replicas of the same model. Global GPU resource allocation and coordinated batching and scheduling across different models are beyond the scope of this work.
W4/Q4: DP/TP/CFG efficiency relative to conventional implementations
PhyAI builds on conventional DP, TP, and CFG implementations. Its contribution is a unified parallel-execution abstraction for embodied models, including VLAs and WAMs. This abstraction exposes each model’s supported parallel modes through a common interface, simplifying their integration into inference workflows and allowing model implementations to be reused across deployment settings.
We do not claim new techniques for reducing communication overhead, improving overlap between computation and communication, or reducing pipeline bubbles. Our contribution lies in the shared abstraction and its integration with model-specific execution.
Thank you for the response. After reading it and briefly inspecting the repository, I feel that my earlier concerns have not been adequately addressed.
-
What accounts for the approximately 20 ms latency difference from FlashRT on Thor? As a newly proposed unified framework, you should explain the specific implementation differences behind this result.
-
What is the real cost of adapting a new model? For a unified framework, users would expect a 'convert.cpp' tool like in llama.cpp. The rebuttal mentions support for additional models, but how much time and effort did these integrations require, including token usage if AI agents were used? Could a user bring a new model, such as DM0.5 or WALL-OSS, and obtain performance improvements through your framework?
-
Could you provide the RLinf commit used and explain what accounts for the 41 seconds reduction in non-prediction time? The reduction in “Trajectory packing & sending” is less than 10s.
My opinion is that reporting a latency difference without explaining its source, while using native implementations as the headline speedup baseline, does not establish its contribution over existing frameworks. If a detailed comparison is difficult, claiming the supported parallelism strategies in training workloads for models such as Cosmos could offer another concrete contribution. I would appreciate a clearer explanation of the contribution over existing systems, or evidence of a more automated path to model optimization.
I would raise my score if these concerns are addressed.
Dear reviewer S9Uo,
We are grateful to the reviewer for the insightful feedback, which has helped improve the quality of this paper.
W4/Q1: RL timing breakdown and the reduction in non-prediction time
The 41-second reduction in non-prediction time in the RLinf experiment stems from an optimization in how we process trajectory data, which we elaborate on below.
The table below breaks down a single RL step for the paper experiment and the two repeated experiments. Each value is the maximum across ranks, averaged over steps 2–5 and reported in seconds.
The table below further splits the rollout stage into the rollout-worker side and the environment side. Each value in the table below is computed as the average over all ranks.
As shown in the table, we observe two key improvements. (1) Our prediction time decreases substantially. This represents our primary contribution and is consistent with expectations. (2) We also optimize the handling of trajectory data: whereas the HF backend stores a deep copy of the raw observations, PhyAI stores ready-to-use preprocessed tensors, thereby eliminating redundant work in trajectory packing, transmission, and repeated preprocessing on the actor side. Consequently, we achieve considerable time savings beyond the prediction stage itself.
W1/Q2: Edge performance and comparisons with FlashRT
During rebuttal, we further optimized PhyAI's edge inference, reducing the latency of π0.5 on Jetson Thor to 64.69 ms, compared with 86.87 ms for FlashRT reported in Figure 2. This corresponds to 25.5% lower latency.
W2/Q2: Model coverage, deployment modes, and integration effort
What concrete capability or performance benefit does PhyAI provide over FlashRT on their overlapping models?
PhyAI supports data-parallel replica scaling and, for models such as Cosmos3, TP/CFG execution within a replica. These modes share the same inference runtime. In the π0.5 experiment in Figure 7(a), scaling from one to eight H800 GPUs, with one complete replica per GPU, increases aggregate throughput from 6.785 to 53.379 requests/s. The 7.87× throughput gain reaches 98.3% of the linear reference. Each replica reuses the same single-GPU model implementation, allowing throughput to scale without a separate distributed model path.
Does the unified abstraction enable additional architectures or deployment modes, or reduce integration effort in a way the authors can substantiate?
During the rebuttal period, we extended PhyAI to support Xiaomi Robotic U0, LingBot-VLA, HumanWAM-Goal, and DreamZero, spanning both VLA and WAM paradigms. Each model preserves its specific generation procedure and state management logic, yet reuses the runtime operators and execution services through a unified interface. These additions demonstrate that PhyAI's abstraction scales to additional models without the need for model-specific inference infrastructure.
We have fully open-sourced our code at https://anonymous.4open.science/r/phyai-8602/ . Please take a look. We invested considerable time in building and thoroughly testing this repository, and we would sincerely appreciate it if you could review it carefully.