acceptodds
Under review as a conference paper at ICLR 2027

PhyAI: A Unified Inference Runtime for Robot Policies Across Edge and Cloud

Abstract

Recent breakthroughs in foundational models have driven significant progress in Robot Learning Models (RLMs), spanning Vision-Language-Action (VLA) and World Action Model (WAM) architectures. However, unlike the mature serving ecosystems for LLMs, RLM inference infrastructure remains highly fragmented across diverse deployment scales, from batch-1 real-time device control to massive batch-N edge serving, evaluationa and RL post-training rollouts. We present PhyAI, a unified, high-performance inference runtime for RLMs. By cleanly decoupling model semantics from execution, PhyAI dynamically adapts its parallelism (e.g., unbatched, DP, TP) and kernel dispatch strategies to the target hardware, seamlessly bridging edge and cloud environments. Through system-level optimizations including CUDA graph replay, kernel fusion, and state reuse, PhyAI achieves 1.40x to 4.65x latency speedups over official implementations across 11 benchmark configurations. To investigate how these inference speedups translate to physical control frequencies, we introduce the Control Rate Roofline model, demonstrating that our optimizations either directly boost control rates or unlock time margins for deploying larger models.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Infrastructure, Robot Learning Models, Robot

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_S9UoOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

PhyAI is a unified inference runtime for robot policies spanning VLA and world action models. It separates model-specific scheduling and execution state from shared operators, uses a capability- and priority-based kernel dispatcher, and supports DP, TP, and classifier-free guidance parallelism. CUDA Graph replay, kernel fusion, and cache precomputation accelerate five model families across edge and server GPUs. The paper reports 1.40–4.65× model-runner speedups over official implementations, near-linear replica scaling, and a 1.17× overall RL-step speedup. It also introduces a Control Rate Roofline that relates request latency to a fixed action-execution window under sequential execution or ideal overlap. Matched LIBERO evaluations provide behavioral validation for π0.5 and GR00T N1.7.

Critical strengths
  1. The system integrates a useful breadth of models and deployment modes. Table 2 covers five model families, including a WAM using parallism across H20 GPUs. Figure 7 extends the evaluation to replica scaling and an RL workflow, where overall step time falls from 772.45 to 657.79 seconds. This demonstrates practical integration beyond isolated kernel benchmarks.

  2. The stage-level batching analysis provides useful deployment insights. Section 5.3 shows that batching shifts π0.5's bottleneck toward the VLM, identifies diminishing throughput returns around batch 8 for GR00T, and explains Cosmos3's limited batching gains through its already compute-intensive generation stage. These findings help identify where batching is useful and where it primarily increases latency.

Critical weaknesses
  1. The edge-performance advantage is unclear. Figure 2 shows no consistent latency advantage over FlashRT on overlapping configurations. The headline speedups are measured against weaker official implementations rather than established optimized runtimes, making them insufficient evidence of an advance over existing systems.

  2. The benefit of the unified abstraction is insufficiently demonstrated. Table 1 lists both PhyAI and FlashRT as supporting VLA and WAM models across edge and cloud. The paper does not demonstrate broader model coverage or quantify reduced integration effort, leaving the value of its abstraction beyond parallel-execution support unclear.

  3. Static batching gains are not validated under actual cloud workloads. Section 5.3 usefully identifies diminishing returns, but excludes batch formation and queueing. Without request arrival patterns or latency budgets, it remains unclear whether beneficial batch sizes can be formed in practice or how PhyAI selects them.

  4. The cloud results do not establish gains beyond FlashRT-style inference acceleration. The demonstrated benefits center on graph replay, kernel fusion, and caching—the same class of optimizations used for edge inference. Although PhyAI supports DP/TP/CFG, the experiments do not establish an advantage over conventional implementations of these parallelism strategies, such as lower communication overhead, better computation–communication overlap, or fewer scheduling bubbles. The RLinf comparison likewise reports faster prediction without demonstrating orchestration or scheduling gains. In this experiment, “Other time” still accounts for 61.1% of the optimized RL step, nearly six times the prediction time. Without a breakdown of simulation, rendering, communication, and synchronization costs, the evaluation neither identifies the dominant bottleneck nor demonstrates that PhyAI addresses it.

Questions for authors
  1. What accounts for the approximately 41-second reduction in non-prediction time in the RLinf experiment?

  2. What concrete capability or performance benefit does PhyAI provide over FlashRT on their overlapping models? Does the unified abstraction enable additional architectures or deployment modes, or reduce integration effort in a way the authors can substantiate?

  3. How does PhyAI select batch sizes and waiting limits under actual request workloads, and what throughput–latency benefits remain after batch formation and queueing? For cloud deployments serving multiple models concurrently, how does the runtime allocate GPU resources and coordinate per-model batching and scheduling, given their different memory demands and latency requirements?

  4. Compared with conventional DP/TP/CFG implementations on matched hardware, does PhyAI reduce communication overhead, improve computation–communication overlap, or reduce pipeline bubbles? Can these benefits be quantified?

8 comments

Dear reviewer S9Uo,

We are grateful to the reviewer for the insightful feedback, which has helped improve the quality of this paper.

W4/Q1: RL timing breakdown and the reduction in non-prediction time

The 41-second reduction in non-prediction time in the RLinf experiment stems from an optimization in how we process trajectory data, which we elaborate on below.

The table below breaks down a single RL step for the paper experiment and the two repeated experiments. Each value is the maximum across ranks, averaged over steps 2–5 and reported in seconds.

RunFull stepWeight syncPredictionRest of rolloutLog-prob recomputationActor training
Paper HF772.450.58 (0.1%)143.13 (18.5%)431.94 (55.9%)87.95 (11.4%)108.83 (14.1%)
Paper PhyAI657.791.27 (0.2%)69.27 (10.5%)401.22 (61.0%)83.99 (12.8%)102.02 (15.5%)
Repeat 1 HF777.810.81 (0.1%)146.06 (18.8%)434.32 (55.8%)88.15 (11.3%)108.45 (13.9%)
Repeat 1 PhyAI663.321.23 (0.2%)68.69 (10.4%)405.93 (61.2%)84.44 (12.7%)103.01 (15.5%)
Repeat 2 HF779.860.79 (0.1%)146.44 (18.8%)438.52 (56.2%)88.13 (11.3%)105.95 (13.6%)
Repeat 2 PhyAI678.081.27 (0.2%)72.04 (10.6%)415.87 (61.3%)84.50 (12.5%)104.38 (15.4%)

The table below further splits the rollout stage into the rollout-worker side and the environment side. Each value in the table below is computed as the average over all ranks.

RunPredictionRollout waiting for env observationsEnv resetSimulator steppingTrajectory packing & sendingEnv waiting for policy outputTail
Paper HF142.05386.78195.32164.4110.25168.5335.96
Paper PhyAI67.83375.28190.76164.344.6587.4922.72
Repeat 1 HF145.44389.32196.58166.1110.39171.4935.20
Repeat 1 PhyAI67.52381.74195.19165.964.6787.5620.67
Repeat 2 HF144.93394.19201.23166.799.23170.4636.60
Repeat 2 PhyAI70.04388.16200.42166.706.5590.4823.14

As shown in the table, we observe two key improvements. (1) Our prediction time decreases substantially. This represents our primary contribution and is consistent with expectations. (2) We also optimize the handling of trajectory data: whereas the HF backend stores a deep copy of the raw observations, PhyAI stores ready-to-use preprocessed tensors, thereby eliminating redundant work in trajectory packing, transmission, and repeated preprocessing on the actor side. Consequently, we achieve considerable time savings beyond the prediction stage itself.

W1/Q2: Edge performance and comparisons with FlashRT

During rebuttal, we further optimized PhyAI's edge inference, reducing the latency of π0.5\pi_{0.5} on Jetson Thor to 64.69 ms, compared with 86.87 ms for FlashRT reported in Figure 2. This corresponds to 25.5% lower latency.

W2/Q2: Model coverage, deployment modes, and integration effort

What concrete capability or performance benefit does PhyAI provide over FlashRT on their overlapping models?

PhyAI supports data-parallel replica scaling and, for models such as Cosmos3, TP/CFG execution within a replica. These modes share the same inference runtime. In the π0.5\pi_{0.5} experiment in Figure 7(a), scaling from one to eight H800 GPUs, with one complete replica per GPU, increases aggregate throughput from 6.785 to 53.379 requests/s. The 7.87× throughput gain reaches 98.3% of the linear reference. Each replica reuses the same single-GPU model implementation, allowing throughput to scale without a separate distributed model path.

Does the unified abstraction enable additional architectures or deployment modes, or reduce integration effort in a way the authors can substantiate?

During the rebuttal period, we extended PhyAI to support Xiaomi Robotic U0, LingBot-VLA, HumanWAM-Goal, and DreamZero, spanning both VLA and WAM paradigms. Each model preserves its specific generation procedure and state management logic, yet reuses the runtime operators and execution services through a unified interface. These additions demonstrate that PhyAI's abstraction scales to additional models without the need for model-specific inference infrastructure.

We have fully open-sourced our code at https://anonymous.4open.science/r/phyai-8602/ . Please take a look. We invested considerable time in building and thoroughly testing this repository, and we would sincerely appreciate it if you could review it carefully.

Q3: Static batching, queueing, and multi-model scheduling

How does PhyAI select batch sizes and waiting limits under actual request workloads

In the fixed robot deployments and batch simulation rollouts considered in this paper, the number of robots or environments and the task configuration are typically fixed in advance, and requests arrive primarily at the pace of the control loop or the rollout, keeping the load relatively stable. PhyAI therefore currently employs static batching. Batch size is configured ahead of time by the user based on available hardware resources and the latency budget. It neither forms batches dynamically from online requests nor offers policies for choosing a batching wait-time cap. Section 5.3 reports throughput and inference latency measured at given batch sizes, providing a basis for selecting the configuration for these deployments.

All configurations use BF16. Each cell reports full-batch inference latency (ms) / throughput (action chunks/s).

Batch sizeπ0.5\pi_{0.5},H200GR00T,RTX 5090Cosmos3,H200
122.59 / 44.2620.49 / 48.801132.13 / 0.883
231.52 / 63.4627.64 / 72.402251.90 / 0.888
450.95 / 78.5137.86 / 105.704157.16 / 0.962
889.37 / 89.5162.49 / 128.008026.96 / 0.997
16168.85 / 94.76118.32 / 135.2015846.72 / 1.010

What throughput–latency benefits remain after batch formation and queueing?

We evaluate RL rollout, where batches are defined in advance from parallel environments. Section 5.3 reports model-runner throughput and latency for these preformed, excluding batch formation and queueing. These measurements characterize batched inference efficiency. For Throughput and end-to-end request latency under online arrivals were not evaluated in this experiment.

For cloud deployments serving multiple models concurrently, how does the runtime allocate GPU resources and coordinate per-model batching and scheduling, given their different memory demands and latency requirements?

Multi-model support in PhyAI means that the same inference infrastructure can execute different VLA/WAM architectures. Deployment settings specify the replica groups and the parallel modes supported by each model. The scheduling mechanisms discussed here route requests among complete model replicas and coordinate execution within the configured parallel groups. Multiple robots can be served by replicas of the same model. Global GPU resource allocation and coordinated batching and scheduling across different models are beyond the scope of this work.

W4/Q4: DP/TP/CFG efficiency relative to conventional implementations

PhyAI builds on conventional DP, TP, and CFG implementations. Its contribution is a unified parallel-execution abstraction for embodied models, including VLAs and WAMs. This abstraction exposes each model’s supported parallel modes through a common interface, simplifying their integration into inference workflows and allowing model implementations to be reused across deployment settings.

We do not claim new techniques for reducing communication overhead, improving overlap between computation and communication, or reducing pipeline bubbles. Our contribution lies in the shared abstraction and its integration with model-specific execution.

Thank you for the response. After reading it and briefly inspecting the repository, I feel that my earlier concerns have not been adequately addressed.

  1. What accounts for the approximately 20 ms latency difference from FlashRT on Thor? As a newly proposed unified framework, you should explain the specific implementation differences behind this result.

  2. What is the real cost of adapting a new model? For a unified framework, users would expect a 'convert.cpp' tool like in llama.cpp. The rebuttal mentions support for additional models, but how much time and effort did these integrations require, including token usage if AI agents were used? Could a user bring a new model, such as DM0.5 or WALL-OSS, and obtain performance improvements through your framework?

  3. Could you provide the RLinf commit used and explain what accounts for the 41 seconds reduction in non-prediction time? The reduction in “Trajectory packing & sending” is less than 10s.

My opinion is that reporting a latency difference without explaining its source, while using native implementations as the headline speedup baseline, does not establish its contribution over existing frameworks. If a detailed comparison is difficult, claiming the supported parallelism strategies in training workloads for models such as Cosmos could offer another concrete contribution. I would appreciate a clearer explanation of the contribution over existing systems, or evidence of a more automated path to model optimization.

I would raise my score if these concerns are addressed.

Reviewer_n3aGOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

In this paper, the authors introduce a shared inference runtime (PhyAI) for robot learning models, e.g., VLAs and WAMs. Their PhyAI integrates model-specific runners with shared kernel dispatch, operator fusion, CUDA graph replay, caching, data configuration and parallelism framework. The claimed contribution is a unified runtime for both edge and cloud deployment with 1.4x to 4.65x lower inference latency across 11 configurations. They conduct empirical evaluations mainly on LIBERO benchmark, as well as RoboTwin.

Critical strengths
  1. The motivation of the work is practically useful. Since the current VLAs and WAMs are widely used as robot policies for the deployment on device, edge and cloud, however, they do not have a unified runtime for inference stacks. With the proposed unified runtime framework, it is possible to reduce duplicated integration and maintenance work.

  2. As claimed by the authors, their PhyAI could support many models and parallel execution modes with lower model-runner latency than the baselines' officially released implementations.

Critical weaknesses
  1. Considering the main contributions of this work are engineering practice. It is important to evaluate the proposed runtime framework on wide range of benchmarks. However, the main experiments are conducted on LIBERO. RoboTwin is only an integration check with no reported task success rate comparison. Evaluations on the large benchmarks, e.g., RoboCase and RoboDojo, are missing, which significantly weaken the claims contributions.

  2. The whole paper reads like a technical report with no clear novelty. The proposed runtime simply integrates standard tools for CUDA Graph replay, kernel fusion, caching and parallelism. More importantly, the paper does not isolate what is technically novel or demonstrate clearly that its unified framwork outperforms a well-tuned combination of existing models.

  3. The definition of the latency metric is not clear. From my understanding, the authors' define latency as a warmed-up model-runner time for one action chunk. This does not consider observation processing, transport, queueing and action handoff, which take more significant latency. The paper also lacks sufficient detail on matched precision and timing protocols, as well as variability and tail latency, to support fair comparisons.

  4. The authors do not provide implementation details and code for reproducibility check. It reads more like a company technical report without sufficient details for the readers to reproduce their framework. Although the runtime, benchmark harness and figure-source data are promised for future release, without providing them for the review, I have no idea how to evaluate their contributions, espicially considering its contributions are more engineering sides. Without implementation details, reviewers cannot verify the implementation, reproduce the speedups, or assess the claimed contributions.

  5. Figure 5 assumes inference can overlap perfectly with current action execution. But in practice, it does not. Please provide sufficient implementation details so that reviewers can better assess the claimed contributions.

Questions for authors

See my concerns in Weaknesses.

6 comments

Dear reviewer n3aG,

Thank you for your constructive feedback and for recognizing the value of our work. We address each of your comments below.

Q1: Benchmark coverage, output fidelity, and downstream task validation

Our work focuses on inference infrastructure. We accelerate model inference through a series of lossless optimizations. For each model implementation, we compare its outputs against those produced by the inference code released by the model provider.

ModelCosine similarity
π0\pi_00.999982
π0.5\pi_{0.5}0.999941
GR00T N1.70.999967
Cosmos3-Nano0.999991
DreamZero0.999632
MiniCPM-RobotBitwise identical
HumanWAM-Goal (added during rebuttal)Bitwise identical
XiaoMi Robotic U0 (added during rebuttal)0.999977
LingBot-VLA (added during rebuttal)0.999978
VPP2 (added during rebuttal)0.999995

We aim for bitwise-identical outputs. Minor differences in operator implementations can introduce small numerical discrepancies, but these differences are negligible.

Following the reviewer's suggestion, we also evaluated π0.5\pi_{0.5} on RoboCasa in BF16 using four inference seeds. The mean success rate was 43.88% for PhyAI and 45.00% for LeRobot.

Inference seedPhyAI (BF16)LeRobot (BF16)
043.8%44.8%
1046.5%46.9%
2041.7%42.3%
3043.5%46.0%
Mean43.88%45.00%

We have not yet evaluated π0.5\pi_{0.5} on RoboDojo. We could not identify a suitable checkpoint for this benchmark, and fine-tuning one was not feasible within the rebuttal period.

Q2: Technical contribution of the unified runtime and comparison with existing approaches

A core contribution of PhyAI is to integrate optimized VLA and WAM inference with configurable DP, TP, and CFG parallelism in a single reusable runtime. Our comparison with FlashRT, vla.cpp, and realtime-vla therefore considers both inference latency and the range of supported execution modes.

We organize model-specific tuned implementations within a common execution framework, distilling reusable execution components while preserving each model's generation process and state-management logic. This makes these optimizations applicable across a range of models and deployment settings.

As our submission falls within the Infrastructure, Software Libraries, Hardware, Systems, etc. track, we believe that the practical engineering contributions described above should also be given due consideration.

Q3: Latency boundaries, matched measurement protocols, variability, and tail latency

The definition of the latency metric: Table 2 and Figure 2 report the model-runner latency for generating a single complete action chunk after warmup. The measured interval covers the model stages and iterative generation steps executed under the benchmark configuration, and excludes model loading and warmup, observation acquisition and preprocessing outside the model runner, client-server transport, request queueing, action handoff, and physical action execution.

This does not consider observation processing, transport, queueing and action handoff, which take more significant latency:

Our work focuses on accelerating the inference stage. As such, the additional latencies in question are orthogonal to our contributions and lie outside the scope of our work.

detail on matched precision and timing protocols:

ModelPrecisionBatch sizeInput tensor sizesAction chunk lengthDenoising stepsCUDA GraphWarm-up iterationsTimed iterations
pi0BF16(FP8 for FlashRT, which did not support BF16 for pi0 at the time of benchmarking)1(B, 2, 3, 224, 224)5010enabled10100
pi0.5BF161(B, 3, 3, 224, 224)5010enabled1050
GR00T N1.7BF161(B, 2, 3, 256, 256)404enabled830
MiniCPM-RobotBF161(B, 2, 3, 224, 224)304enabled1050
Cosmos3-Nano-PolicyBF161(B, 3, 33, 480, 832)324disabled25

Q4: Implementation details, code availability, and reproducibility

We have fully open-sourced our code at https://anonymous.4open.science/r/phyai-8602/ . Please take a look. We invested considerable time in building and thoroughly testing this repository, and we would sincerely appreciate it if you could review it carefully.

Q5: Control Rate Roofline assumptions, RTC implementation, and overlap validation

I respectfully disagree with your claim that "But in practice, it does not" (with respect to RTC overlap). The studies listed below all employ RTC, and they demonstrate that RTC can indeed achieve perfect overlap of inference and execution.

The asynchronous overlap illustrated in Figure 5 has been implemented on physical robots. GR00T 2 reports a real-robot evaluation with RTC enabled at 20 Hz and describes requesting the next action chunk while the controller executes the current one [1]. The original RTC paper evaluates π0.5\pi_{0.5} on six real-world manipulation tasks [2], training-time RTC is evaluated with π0.6\pi_{0.6} on box building and espresso making [3], and π0.7\pi_{0.7} uses training-time RTC in its robot deployment [4]. RynnBrain-VLA, described in the RynnBrain 1.1 report, triggers inference every five steps while the current 32-step action chunk is still executing [5]. DynamicWAM and TBD-VLA also evaluate RTC on physical robots and report improved task success in comparisons with RTC disabled [6,7].

RTC generates the next action chunk in the background while the robot executes the current one. The original algorithm conditions on previously committed actions through inference-time inpainting to maintain continuity across chunk boundaries [2]. Request latency can be fully hidden when the next chunk is ready before the active action buffer is exhausted.

Related methods further develop asynchronous action generation: Legato learns chunk continuation during training [8], Soft RTC uses action-prior denoising to improve continuity [9], and DiscreteRTC uses native discrete-diffusion inpainting for asynchronous chunk generation [10].

References

  1. NVIDIA. GR00T 2: Scaling Robot Foundation Models for Generalization, Dexterity, and Adaptation. 2026.

  2. Kevin Black, Manuel Y. Galliker, and Sergey Levine. Real-Time Execution of Action Chunking Flow Policies. 2025.

  3. Kevin Black, Allen Z. Ren, Michael Equi, and Sergey Levine. Training-Time Action Conditioning for Efficient Real-Time Chunking. 2025.

  4. Physical Intelligence et al. π0.7\pi_{0.7}: a Steerable Generalist Robotic Foundation Model with Emergent Capabilities. 2026.

  5. Kehan Li et al. RynnBrain 1.1: Towards More Capable and Generalizable Embodied Foundation Model. 2026.

  6. Yunfan Lou et al. DynamicWAM: Dual-Path Motion Conditioning for World-Action Models in Dynamic Manipulation. 2026.

  7. Sung-Wook Lee, Xuhui Kang, and Yen-Ling Kuo. TBD-VLA: Temporal Block Diffusion Vision Language Action Model. 2026.

  8. Yufeng Liu et al. Learning Native Continuation for Action Chunking Flow Policies. 2026.

  9. Dongyang Liu et al. Action-Prior Denoising for Smooth Real-Time Chunking. 2026.

  10. Pengcheng Wang et al. DiscreteRTC: Discrete Diffusion Policies are Natural Asynchronous Executors. 2026.

We also test real-time chunking (RTC) for reviewers in REAL Robot!

Our primary platform is Tianji Marvin Pro with an onboard Jetson Thor and a fine-tuned pi0.5 policy. Four quantities define its schedule:

  • Action execution rate: the number of predicted actions applied per second. We use 50 Hz to match the demonstrations, or 20 ms per action.
  • Action horizon: the number of actions predicted per chunk (50).
  • Execution horizon: the number of actions executed before requesting the next chunk (20).
  • Prefix length: the number of already committed action targets supplied to the next prediction (10).

With synchronous scheduling, each cycle takes 20×20 ms+Lreq=400 ms+Lreq20 \times 20\,\mathrm{ms} + L_{\mathrm{req}} = 400\,\mathrm{ms} + L_{\mathrm{req}}. With RTC, the next request overlaps execution of the committed prefix. Its 10 targets span 180 ms from the first target, due immediately, to the last. Across 1,000 onboard requests measured without RTC, snapshot-to-action latency averaged 136.2 ms, including 119.7 ms of model inference. Even the slowest request (152.6 ms) fit within the RTC window, leaving 27.4 ms of margin! PhyAI’s π₀.₅ outputs also closely matched the official implementation (cosine similarity 0.99998731).

Reviewer_Yu11OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PhyAI is a unified inference runtime for VLA and WAM models, combining rule-based kernel selection, DP/TP/CFG parallelism, CUDA Graph replay, kernel fusion, and cache precomputation to support edge and cloud deployment. The paper reports 1.40–4.65× inference speedups over official implementations across 11 configurations, alongside task-success evaluations, multi-GPU scaling, and RL workflow results. The supplementary video reports approximately 1.86× faster real-robot task completion. The paper also introduces a Control Rate Roofline model to analyze performance limits under ideal overlap between inference and execution.

Critical strengths
  1. Broad system coverage and clear practical value. The runtime supports VLA/WAM models across edge and server hardware, with multi-GPU execution and inference speedups across configurations. This integration could help reduce fragmentation in robot deployment software.
  2. Task-level and end-to-end validation. Matched success rates for π₀.₅ and GR00T are close to those of the official implementations. In the supplementary video, both implementations complete all three demonstrated stacking trials, with reported cumulative task times of 66.7 versus 123.9 seconds, excluding reset segments. The complete RL step also achieves a 1.17× speedup.
Critical weaknesses
  1. Comparison settings and contribution attribution remain unclear. Figure 2 includes specialized runtime baselines, but precision, denoising steps, and compilation/graph-capture settings are not fully specified. The existing ablations do not sufficiently characterize the benefits and overheads of the unified runtime. Selecting compatible kernels by predefined priorities also does not, by itself, substantiate the claim of automatically selecting optimal kernels.
  2. The control and serving claims lack direct validation. The real-robot video supports faster completion of a particular task, but does not specify the model, hardware, or synchronous/RTC execution settings. The empirical points in Figure 5 come from sequential execution and omit communication and queueing overheads, so they do not validate the roofline under ideal overlap. Static-batch tests also do not establish real-time performance under dynamic multi-robot workloads.
Questions for authors
  1. Could you provide the matched settings for the main baselines, together with representative ablations of individual optimizations and unified scheduling overhead? Which configurations are selected automatically?
  2. Which model, hardware, and scheduling mode are used in the real-robot video? What are the timing boundaries and success criteria? How do these results relate to the complete request latency and execution window in the roofline model?
  3. Are matched-input output-error or task-quality evaluations available for MiniCPM and Cosmos3, particularly for the TP/CFG execution path?
  4. The RL results average only four measured steps. Could you report variability and explain the additional 40.83 seconds saved outside the prediction component?
4 comments

Dear Reviewer Yu11,

Thank you for your careful reading and constructive feedback. We address your comments below.

W1/Q1: Baseline settings, optimization ablations, runtime overhead, and automatic selection

Baseline settings and timing. The following table summarizes the model configurations and measurement protocols for Table 2 and Figure 2.

ModelPrecisionBatch sizeInput tensor sizesAction chunk lengthDenoising stepsCUDA GraphWarm-up iterationsTimed iterations
pi0BF16 (FP8 for FlashRT, which did not support BF16 for pi0 at the time of benchmarking)1(B, 2, 3, 224, 224)5010enabled10100
pi0.5BF161(B, 3, 3, 224, 224)5010enabled1050
GR00T N1.7BF161(B, 2, 3, 256, 256)404enabled830
MiniCPM-RobotBF161(B, 2, 3, 224, 224)304enabled1050
Cosmos3-Nano-PolicyBF161(B, 3, 33, 480, 832)324disabled25

Representative ablations and overhead. Section 3.4 reports the following incremental optimizations for MiniCPM-RobotManip on H20:

MiniCPM-RobotManip configurationInference throughput (Hz)
Without CUDA Graph replay10.12
With CUDA Graph replay33.28
CUDA Graph replay + kernel fusion36.77

Graph replay provides a 3.29× throughput speedup. Adding kernel fusion in BF16 increases throughput by a further 10.5%. These measurements quantify the incremental benefits of the two optimizations.

Automatic kernel selection. In the mechanism described in the paper, the dispatcher filters kernel implementations by device, tensor shape, data type, and execution mode, then selects a backend according to the execution policy and priorities. CUDA Graph mode also excludes implementations that violate capture constraints. The selected implementation is cached for reuse with the same query signature.

W2/Q2: Real-robot setup, timing boundaries, and success criteria

The supplementary video compares PhyAI with native PyTorch using π0.5\pi_{0.5} on an Agilex Cobot with an RTX 4090. It illustrates how faster inference affects execution speed.

Both implementations use synchronous inference, so each control cycle consists of action execution plus request latency: Tseq=Tenv+LreqT_{\mathrm{seq}}=T_{\mathrm{env}}+L_{\mathrm{req}}. Both completed the three stacking trials shown in the video. Excluding reset segments, their cumulative completion times were 66.7 seconds for PhyAI and 123.9 seconds for PyTorch, corresponding to approximately a 1.86× speedup.

We report these as demonstration completion times, not as a systematic evaluation of policy success rates or generalization.

Our evaluation focuses on inference speed and numerical fidelity. We separately compared the outputs of π0.5\pi_{0.5} with the model provider's reference implementation:

ModelCosine similarity
π0.5\pi_{0.5}0.999941

W2/Q2: Control Rate Roofline assumptions and validation scope

The measured points in Figure 5 use sequential execution and account only for model inference and environment execution. The ideal-overlap curve gives an upper bound on control rate, with a corresponding period of Tideal=max⁡(Lreq,Tenv)T_{\mathrm{ideal}}=\max(L_{\mathrm{req}},T_{\mathrm{env}}). Here, LreqL_{\mathrm{req}} denotes the complete request latency, including observation processing, communication, queueing, and action handoff. Hiding latency requires the complete request to finish before the action handoff.

We collected additional request timings on a separate platform, Tianji Marvin Pro with a Jetson Thor, using a fine-tuned π0.5\pi_{0.5} policy. The action execution rate is 50 Hz, with 50 actions predicted per chunk. The RTC scheduling configuration triggers the next request after 20 actions and supplies 10 already committed action targets as a prefix. With the first target due immediately, the 10 target timestamps run from 0 ms to 180 ms, giving a prefix window of 180 ms.

Across 1,000 onboard requests measured without RTC, the mean latency from observation snapshot to available actions was 136.2 ms, including 119.7 ms of model inference. The maximum request latency was 152.6 ms, leaving 27.4 ms of margin within this window.

W2: Static batching and dynamic multi-robot serving

In the fixed robot deployments and batched simulation rollouts considered here, the number of robots or environments and the task configuration are typically set in advance. Requests mainly follow the control loop or rollout schedule, resulting in relatively stable workloads. PhyAI currently uses static batching, with batch size configured by the user according to hardware resources and the inference latency budget. It does not dynamically form batches from online requests or provide a policy for choosing how long to wait for a batch to fill.

Q3: Output fidelity of MiniCPM and Cosmos3, including TP/CFG execution

We compared the outputs of MiniCPM-Robot and Cosmos3-Nano against the reference inference implementations released by their model providers:

ModelOutput agreement
MiniCPM-RobotBitwise identical
Cosmos3-NanoCosine similarity 0.999991

We have not yet evaluated downstream task success rates for MiniCPM-Robot or Cosmos3-Nano.

MiniCPM-Robot is small enough to run on a single GPU without TP. Its inference procedure does not use classifier-free guidance, so CFG parallelism does not apply to this configuration. For Cosmos3, we also evaluated TP=4 with CFG=2 and obtained a cosine similarity of 0.999986 against the reference implementation.

Q4: RL timing variability and the 40.83-second reduction in non-prediction time

We conducted two independent repeats in addition to the original experiment, giving three comparisons between HF and PhyAI. Each comparison reports the mean over RL steps 2 to 5. Prediction speedups on the critical path ranged from 2.03× to 2.13×, while full-step speedups ranged from 1.15× to 1.17×. Within each run, step-to-step variation in prediction time was approximately 1 to 2 seconds.

The first table breaks down the full RL step. Times are in seconds, with each stage's share of the full step shown in parentheses.

RunFull stepWeight syncPredictionRest of rolloutLog-prob recomputationActor training
Paper HF772.450.58 (0.1%)143.13 (18.5%)431.94 (55.9%)87.95 (11.4%)108.83 (14.1%)
Paper PhyAI657.791.27 (0.2%)69.27 (10.5%)401.22 (61.0%)83.99 (12.8%)102.02 (15.5%)
Repeat 1 HF777.810.81 (0.1%)146.06 (18.8%)434.32 (55.8%)88.15 (11.3%)108.45 (13.9%)
Repeat 1 PhyAI663.321.23 (0.2%)68.69 (10.4%)405.93 (61.2%)84.44 (12.7%)103.01 (15.5%)
Repeat 2 HF779.860.79 (0.1%)146.44 (18.8%)438.52 (56.2%)88.13 (11.3%)105.95 (13.6%)
Repeat 2 PhyAI678.081.27 (0.2%)72.04 (10.6%)415.87 (61.3%)84.50 (12.5%)104.38 (15.4%)

The second table separates the rollout-worker and environment sides. Values are mean times across four ranks, in seconds.

RunPredictionRollout waiting for env observationsEnv resetSimulator steppingTrajectory packing & sendingEnv waiting for policy outputTail
Paper HF142.05386.78195.32164.4110.25168.5335.96
Paper PhyAI67.83375.28190.76164.344.6587.4922.72
Repeat 1 HF145.44389.32196.58166.1110.39171.4935.20
Repeat 1 PhyAI67.52381.74195.19165.964.6787.5620.67
Repeat 2 HF144.93394.19201.23166.799.23170.4636.60
Repeat 2 PhyAI70.04388.16200.42166.706.5590.4823.14

The three comparisons show non-prediction time reductions of 40.80, 37.12, and 27.38 seconds, respectively.

We also optimized trajectory processing. The HF backend stores deep copies of raw observations, while PhyAI stores preprocessed tensors that can be used directly. This reduces work in trajectory packing and transmission, as well as repeated preprocessing on the actor side.

Note

We have fully open-sourced our code at https://anonymous.4open.science/r/phyai-8602/ . Please take a look. We invested considerable time in building and thoroughly testing this repository, and we would sincerely appreciate it if you could review it carefully.

Thank you for the detailed clarifications and additional results. Your response has substantially alleviated my main concerns regarding the comparison settings, numerical fidelity, and consistency of the RL performance gains. I will therefore raise my rating.

Thank you for taking the time to review our paper and responses. We sincerely appreciate your recognition of our work, your thoughtful feedback and effort that helped us improve the paper!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.