- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies the effect of inference latency on vision-language agents and proposes measuring deployment latency, fitting a stochastic timing profile, and replaying that profile during training and evaluation. Across 12 environments and three StarVLA architecture variants, the paper reports substantial degradation under hardware-timed execution and improvements in all 36 task–architecture pairs after profile-based training.
Critical strengths
- Relevant evaluation problem. Allowing environments to advance during inference exposes failures that world-pausing evaluation can conceal.
- Substantial empirical coverage. The study includes games and embodied-control environments, multiple architecture variants, and measured hardware timing.
- Transparent limitations. The appendices document many implementation details and explicitly acknowledge several experimental confounds.
Critical weaknesses
- The central training comparison is missing. Table 1 compares profile-trained policies against policies trained without latency. This establishes that accounting for deployment delay can help, but does not establish the value of fitting and preserving the measured temporal profile. This is the most significant gap because it concerns the paper’s principal contribution.
- The five-deployment calibration experiment uses one OpenVLA-style policy on Flappy Bird. Temporal achieves a lower average return gap than IID, 10.8 versus 18.1, but IID is closer on three of the five deployments. As Appendix D acknowledges, the comparison changes both the delay model and inference-capacity modeling. Consequently, the large improvement over Fixed does not isolate the importance of temporal correlation. Broader validation across policies, tasks, and held-out deployment sessions is needed, with capacity modeling held constant.
- The incremental contribution over existing delay-aware control is insufficiently demonstrated. the delay aware control has been established in papers like RTC and ARLI [1]. The paper appropriately uses a standard adaptation recipe, making its contribution primarily empirical and methodological. Such a contribution can be valuable, but requires convincing evidence that deployment-fitted profiles provide information or performance unavailable from simpler delay models.
- The deployment conclusions need tighter scope. Six tasks use adjusted environment or control rates to accommodate VLA inference, including Flappy Bird at 10 Hz and MuJoCo tasks at architecture-dependent rates of 5–10 Hz. These settings support controlled comparisons but do not establish effectiveness at native task rates.
[1] Learning to Act While Waiting: RL Finetuning of Generalist Robot Policies Under Inference Latency, Zhu et al.
Questions for authors
- Does Temporal-profile training outperform fixed-mean, IID, and randomized-delay training under matched VLA training budgets?
- Does the calibration advantage persist when every profile uses identical worker-capacity modeling and is evaluated on held-out sessions?
- Can the transfer experiments be repeated with matched inputs, training budgets, evaluation episodes, and multiple training seeds?
Response to Reviewer 8ryG
We thank the reviewer for noting the "substantial empirical coverage," the relevance of the evaluation problem, and the "transparent limitations" in the appendices.
[W1 / Q1] Central training comparison
Under a matched budget, both of our measured profiles score higher than Fixed and RTC on all three rows of Table R1. On L40S, Fixed reaches 3.95, IID 255.98 and Temporal 270.41. IID draws each latency at random from the measured distribution, so it also serves as the randomized-delay condition; we did not run VLA training with uniform or normal delays, and App. F has uniform-delay training for small policies (3 seeds). AR(1) and ARMA(1,1) generate slow requests at less than 1% of the measured share (Table R2). Training on our measured profiles improves over a fixed delay, and keeping the measured temporal structure improves replay fidelity.
[W2 / Q2] Calibration breadth, capacity modeling and held-out sessions
Over 30 held-out cases, our measured profiles roughly halve the replay error of a fixed delay: 3.81 (Temporal) and 4.58 (IID) percentage points, against 6.55 for Normal and 8.93 for Fixed (Table R8). The 30 cases cover three tasks, two policies and five GPU types; each profile is fitted on 25 hardware runs and tested on 25 held-out runs. IID and Temporal use the same inference-worker model and the same joint sampling of action latency and busy time (App. D), and differ only in whether slow requests occur consecutively; Temporal has the lower error.
Table R8: Normalized mean absolute error between simulated and hardware scores over 30 cases (percentage points, relative to the zero-latency return).
[W3] Contribution relative to RTC and ARLI
RTC and ARLI are both in Table R1. On Flappy Bird with the RTX 3090 profile, RTC + ARLI reaches 438.00, against 414.52 for Temporal. On L40S, training on our measured profiles alone reaches 255.98–270.41, against 58.46 for RTC + ARLI, and ARLI changes a Temporal checkpoint by −4.30% to +4.23%.
[W4] Environment rates
A deployment cannot run its control loop faster than the policy produces actions, and running it slower gives up control the policy could provide. For Flappy Bird, AirRaid and the four MuJoCo tasks, whose native rates exceed VLA inference speed, we therefore set each architecture's control rate to its RTX 3090 inference rate, at which its mean latency spans about one decision step. All three architectures then face the same delay in decision steps, so the comparison reflects how a policy acts under delay rather than which model runs fastest. The other six tasks run at their standard benchmark rate, and profile training improves all 18 of their pairs (§4.2). Delays of many decision steps are covered by the fixed-delay sweeps (§4.4).
[Q3] Transfer with matched inputs, budgets and seeds
The transfer pattern holds across three tasks. In Figure 6, Demon Attack, Deadly Corridor and MIKASA all transfer more reliably from longer training latencies to shorter evaluation latencies (§4.4), so the conclusion does not rest on a single cell. The training comparison of Table R1 averages 3 seeds per condition.