- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies inconsistencies in VLA inference caused by changes in batch size, request order, and batch composition. Stage assigns independent fixed execution widths to vision encoding, language encoding, and action generation, and uses request-specific noise to preserve action outputs within a fixed deployment. Experiments on four VLAs evaluate byte-level consistency and throughput, with additional closed-loop evaluation on π0.5.
Critical strengths
The main value of this work is the connection between numerical reproducibility and robot task outcomes. Section 2 shows that batch-dependent numerical differences can accumulate during closed-loop execution and change whether a task succeeds. This provides a convincing motivation for the problem. The method is clearly described. Independent stage widths account for the different computational characteristics of vision, language, and action generation, and the ablation in Section 4.2 supports this choice. The four models cover autoregressive, parallel-regression, and iterative generation. The paper explicitly discusses the assumptions behind determinism, possible output differences between configurations, and performance losses in dynamic serving.
Critical weaknesses
Performance under a varying workload with one fixed configuration remains insufficiently evaluated. The near-Native results in Section 4.3 use different configurations selected for B=1 and B=8, whereas deployment requires the configuration to remain fixed. Section 4.4 further reports 0.83–0.89× Native throughput and 1.32–1.50× Native P99 latency under burst arrivals. An analysis that selects and evaluates a single configuration for a workload distribution would make the practical efficiency claim more convincing. In particular, the compromise configuration identified in Section 4.2 deserves comparison with those selected for specific batch sizes. The LLM-42 comparison needs more information about tuning. Appendix C.2 provides three candidates evaluated on π0.5, but does not sufficiently explain how these choices transfer to the other models. Since execution width substantially affects throughput, whether the baseline was tuned per model matters when interpreting the speedups. The evaluation also has a limited scope. Four-model consistency tests use episode first frames, closed-loop and serving experiments concentrate on π0.5, and all experiments use one RTX 4090. The results establish consistency for the tested configurations, but provide less evidence about other models in closed-loop operation or sensitivity to the inference environment.
Questions for authors
- The paper states that execution widths should be selected for the target workload. When batch sizes vary over time, how should the workload distribution and throughput/latency requirements guide the choice of one fixed configuration? How does the compromise configuration from Section 4.2 perform under the arrival patterns in Section 4.4?
- Beyond the two LLM-42 configurations selected on π0.5 in Appendix C.2, were independent searches conducted for the other models? If so, please report the search ranges and results. Otherwise, can the authors assess whether per-model tuning changes the performance gap to Stage?
3 comments
Thank you to the authors for their point-by-point response to the review comments. The supplementary experiments and clarifications are reasonably thorough and have addressed most of my concerns. However, some issues remain only partially resolved, so I will maintain my score.
Update on Weakness: throughput on Jetson Thor
We repeated the forward-throughput experiment of Section 4.3 for π0.5 on the NVIDIA Jetson AGX Thor with the same 500 LIBERO-10 first frames, and evaluated all candidate configurations of each method on Thor.
- Byte identity. Every Invariant Kernels, LLM-42, and Stage configuration stays byte-identical to its own B = 1 reference at every batch size.
- Throughput. Stage (1,1,1) reaches 0.98× the throughput of Native at B = 1. At B = 8, the fastest Stage configuration on Thor is (8,1,8), which reaches 1.08× the throughput of Native, 1.50× the throughput of the best LLM-42, and 2.13× the throughput of Invariant Kernels. On the RTX 4090 the fastest is (8,4,8), so the stage widths are selected for each device.
Forward throughput of π0.5 in samples/s on 500 LIBERO-10 episode first frames on one Jetson AGX Thor. Labels give Stage (V,P,C) and LLM-42 (K,W,G).
| Method | B = 1 | B = 2 | B = 4 | B = 8 |
|---|---|---|---|---|
| Native | 5.91 | 7.16 | 7.39 | 8.04 |
| Invariant Kernels | 3.64 | 4.06 | 4.08 | 4.06 |
| LLM-42 (1,2,1) | 3.54 | 5.79 | 4.13 | 4.22 |
| LLM-42 (2,4,2) | 1.63 | 2.16 | 2.47 | 5.78 |
| LLM-42 (4,8,2) | 0.97 | 1.32 | 1.52 | 1.68 |
| Stage (1,1,1) | 5.77 | 5.86 | 5.93 | 5.94 |
| Stage (8,4,8) | 1.43 | 2.83 | 5.67 | 7.70 |
| Stage (8,1,8) | 2.23 | 3.87 | 6.13 | 8.66 |
| Stage (8,8,8) | 1.01 | 1.99 | 4.01 | 7.99 |
| Stage (1,8,8) | 1.11 | 2.18 | 4.22 | 8.00 |
| Stage (4,8,8) | 1.06 | 2.12 | 4.24 | 8.02 |
We hope these results, together with our earlier response, address your concerns. If they do, we would be grateful if you could consider raising your score. If any concern remains, please let us know, and we will address it before the discussion period ends.
Response to Reviewer TdQM
We thank the reviewer for the detailed review and for recognizing the link between numerical reproducibility and task outcomes, the stage-width ablation in Section 4.2, and the coverage of three generator types. We address each concern below.
Weakness 1 and Question 1: One fixed configuration under varying workloads, and the compromise configuration
A fixed configuration should be chosen from the tested batch-size distribution. We evaluated all 27 fixed configurations, including the compromise (1,1,4), under the loads of Section 4.4.
Weakness 2 and Question 2: Tuning LLM-42 for each model
We now tune LLM-42 separately for each model, and Stage keeps its speedups. We ran the three candidates of Appendix C.2, (K, W, G) ∈ {(1,2,1), (2,4,2), (4,8,2)}, on every model at B ∈ {1, 2, 4, 8}, and compared Stage with the best candidate for each model and batch size.
Weakness 3: Evaluation scope (first frames, closed loop, and hardware)
We added a second closed-loop model and a second hardware platform.