- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
VLA-HWBench characterizes inference latency for 18 open VLA models, comprising 19 configurations, on two edge hardware platforms. It decomposes inference into seven stages, compares compilation and specialized backends, and analyzes bottlenecks using kernel profiling and roofline estimates. Measurements use a shared real LIBERO observation, supplemented by frame variation experiments and optimization checks. The study finds that sequential action generation often dominates latency, specialized backends substantially accelerate two supported models, and optimization exposes CPU preprocessing and other remaining bottlenecks.
Critical strengths
-
Broad and detailed characterization of heterogeneous VLA implementations. The study covers four action-generation forms across two platforms and includes preprocessing and postprocessing in its timing boundary. Tables 7–9 show substantial differences in inference latency and stage-level costs that parameter count alone cannot explain. This makes the benchmark a useful resource for identifying where optimization effort is likely to matter.
-
Concrete diagnosis beyond reporting speedups. Section 5.3 and Appendix G connect latency to identifiable implementation choices, including repeated context projections, inefficient patch embedding, and CPU worker-thread overhead. The thread-count experiment in Figure 6b provides particularly useful evidence that host configuration can materially affect inference latency.
Critical weaknesses
-
The breadth of measurements is not matched by the depth or generality of the insights. Insight 1 substantially overlaps with Finding #1 of Zhou et al., leaving its additional contribution unclear. Insight 2 compares all models without optional optimizations in Figure 4, but derives its optimization recommendations primarily from two FlashRT-supported models in Figure 5. Figure 4 should use compiled configurations as the baseline for all models where supported. This would provide a stronger basis for Insight 3 to analyze how architectural differences affect the benefits of kernel fusion and the resulting stage-level bottlenecks, rather than further diagnosing only the same two models. The CPU preprocessing analysis is also limited to comparing 1 and 14 threads on Jetson, without a broader thread-count sweep across both platforms. These limitations weaken the paper’s broader model, software, and hardware design recommendations.
-
Task performance after optimization is insufficiently validated. Table 14 measures action agreement on only 11 frames, despite reported gripper-command flips and larger deviations under NVFP4. Without task-success measurements, it remains unclear whether the headline speedups preserve policy effectiveness.
Questions for authors
-
Does a broader thread-count sweep on both platforms support the CPU preprocessing conclusions, and what thread configuration was used for AMD in Figure 6b?
-
Given that Insight 1 identifies sequential action generation as a major bottleneck, why does the main optimization comparison focus on compilation, specialized backends, and precision rather than also evaluating algorithmic methods that reduce action-generation iterations? Could including such methods at comparable task success change the optimization recommendations in Insight 2?
-
Since both platforms share physical memory between the CPU and GPU, what accounts for the reported host-to-device latency: explicit buffer copies, synchronization, or other runtime overhead? Would a shared-buffer or zero-copy implementation reduce this cost on other platforms?
6 comments
Response to Reviewer rSRP (Part 2)
W2. Task performance after optimization.
We now report LIBERO-10 success for the compiled, FlashRT FP16, FP8 and NVFP4 levels of Section 5.2, with 50 episodes per level.
| Thor | Eager | Compiled | FlashRT FP16 | FP8 | NVFP4 |
|---|---|---|---|---|---|
| π0.5 success | 44/50 | 42/50 | 45/50 | 40/50 | 43/50 |
| π0.5 latency | 317 ms | 124 ms | 90 ms | 61 ms | 55 ms |
| GR00T N1.7 success | 49/50 | 46/50 | 43/50 | 46/50 | 45/50 |
| GR00T N1.7 latency | 113 ms | 96 ms | 81 ms | 57 ms | 38 ms |
In larger runs with 200 episodes per configuration, the headline speedups stay within 3 points of native success. In the same runs, NVFP4 is 3.6 times faster than the native GR00T N1.7 runtime and two denoising steps shorten its call by 19%. We also ran this policy in real time, with the simulator advancing at 20 Hz while the policy runs.
| GR00T N1.7 on Thor | Paused, of 200 | Real time, 400 ms replanning, of 100 | Real time, 100 ms replanning, of 100 |
|---|---|---|---|
| Native, 4 steps | 183 | 89 | 87 |
| Two denoising steps | 177 | 86 | 88 |
| NVFP4 | 177 | 93 | 87 |
Compiled π0.5, whose actions can flip a gripper command, succeeds in 169 of 200 episodes on AMD against 170 natively at 1.8 times lower latency.
Q2. Methods that reduce iterations.
Reducing iterations is an algorithmic change to the model, while Section 5.2 studies system optimizations that keep the weights and the algorithm fixed. This is why Insight 1 addresses it as guidance for model design. To show how the two levels relate, we also tested the simplest algorithmic change, fewer denoising steps on the released checkpoints. For GR00T N1.7, W2 shows that two steps stay close to native success in both settings but shorten the call by only 19%. Going from 10 to 5 steps cuts the π0.5 call on AMD from 634 to 533 ms with 168 against 170 successes. Fewer steps shorten only the denoising loop, so at the same success of 177 episodes NVFP4 is still 2.9 times faster than two-step sampling. The two levels are therefore complementary, and the recommendations of Insight 2 hold. Algorithmic reduction pays off most where generation dominates the call, as in VLA-0 and the reasoning models with 91 to 99%.
Q3. Host-to-device time on shared memory.
This stage is small. On AMD it takes 0.79 ms of the 299 ms GR00T N1.7 call, or 0.3%. It consists of 12 explicit HIP copies issued by the PyTorch pipeline, which keeps GPU tensors in a separate memory pool, together with dtype and layout conversion and the synchronization at the stage boundary. Reusing preallocated buffers lowers it to 0.73 ms, and eliminating the stage would save at most 0.79 ms per call in this configuration. The larger host-side cost is CPU preprocessing, discussed in W1 and Q1.
References
[1] K. Zhou et al. Characterizing Vision-Language-Action Models across XPUs: Constraints and Acceleration for On-Robot Deployment. ICML 2026. https://proceedings.mlr.press/v306/zhou26e.html
Thank you for the additional experiments and clarifications. Most of my concerns have been addressed. I have two remaining concerns:
- Insight #1 reflects an established execution-cost analysis as novelty, but the relationship between repeated weight accesses and latency may depend on subsequent optimizations, including FlashRT’s CUDA Graphs, buffer reuse, and kernel fusion. Would these optimizations affect the conclusions drawn from Insight 1? If so, could build a more explicit cost model. For example, offline operator profiling combined with an estimate of the model’s theoretical performance limit?
- The simulation-frequency setup resembles the RTC experiments, but the difference for the native implementation is small: success decreases from 91.5% with paused simulation to 89% without paused. This is similar to the differences observed with compilation and does not demonstrate that more frequent replanning improves success. Indeed, success rates with 100 ms replanning is even lower than that of NVFP4 with paused simulation. Does this suggest that quantization is more effective than more frequent replanning in this setting?
Response to Reviewer rSRP (Part 1)
We thank the reviewer for the detailed and constructive feedback, and for recognizing the breadth of our characterization and the concrete diagnoses in Section 5.3 and Appendix G. We ran new experiments for each point and answer them below. Code and data are available at https://anonymous.4open.science/r/VLA_HWBench-4B77/.
W1. Relation of Insight 1 to Zhou et al.
Finding #1 of Zhou et al. [1] states that model size does not predict latency. Insight 1 goes further and identifies what does. Over 11 configurations spanning flow matching, token-by-token decoding and reasoning, iterations × active weight bytes explains 64% and 88% of the log-latency variance on Thor and AMD, and predicts held-out checkpoints with a Q² of 0.50 and 0.84, against 0.04 and 0.22 for parameter count. Insight 1 also connects latency to control. Only 10 of the 19 configurations keep up with the 20 Hz control rate of LIBERO on Thor, and the slower forms of action generation do not report higher success. We will cite Finding #1 and state this difference in the paper.
W1. Compiled baselines in Figure 4 and the effect of architecture.
We added compiled baselines for six models on both platforms, with 120 interleaved eager and compiled calls per model. The table gives the speedup of the median latency over eager execution, with the same weights and matmul precision in both runs.
The gain depends on how much of the call is compiled. Compiling the full model call of π0.5 and SmolVLA places vision, prefill and every denoising step in one graph and speeds them up 1.6 to 2.8 times. Compiling only the language model gains 1.03 to 1.17 times, close to the 1.02 to 1.15 times of compiling only the action head of GR00T N1.7 in Section 5.2. The bottleneck left after optimization also depends on the architecture. With 574 to 712 prefix tokens, prefill grows from 14% to 34% of the π0.5 call, while GR00T N1.7 prefills only 156 tokens and is left with CPU preprocessing as its largest stage at 39%. We will add the compiled points to Figure 4.
W1 and Q1. Thread sweep on both platforms.
The broader sweep supports our conclusions. We measured the processor and collator time of GR00T N1.7, its CPU preprocessing path, with 1, 2, 4 and 8 intra-op threads and the PyTorch default on both platforms, with 200 calls per setting. The AMD bar in Figure 6b uses the default of 16 threads.
On Thor, waking the idle worker threads doubles this time at the default setting, while the AMD host is insensitive to the thread count and remains 3 times faster even with one thread. The best setting also depends on how often the model is called. With NVFP4 on Thor, 4 threads give 37.5 ms per call against 45 to 47 ms with 14 threads when calls run back to back, while both settings give 44 ms with 50 ms of idle time after each call.