acceptodds
Withdrawn from ICLR 2027

Are Vision-Language Agents Latency-Aware? A Systematic Evaluation Under Action Latency

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Foundation-model agents, such as vision-language-action (VLA) policies, suffer from inference latency: the environment keeps changing while the policy computes. Faster inference reduces this delay but does not eliminate it, and training with a delay chosen in advance may not match the target deployment. Current benchmarks hide the problem by pausing the environment during inference; evaluated in real time on an RTX 3090 GPU, VLA policies on 12 environments retain a median of only 6.6% of their zero-latency return. We study how to estimate the residual latency of a deployment, adapt a policy to it, and determine when the adaptation transfers. We introduce a **latency profiling and replay framework** that fits the distribution and temporal dependence of measured action latencies and replays new delay sequences in simulation. (1) Across five GPU deployments, a Temporal profile reproduces hardware-timed return with an average gap of 10.8, versus 87.4 for a fixed mean delay. (2) Training under the target profile improves hardware-timed return in all 36 task–architecture pairs, by 16% to 86% of zero-latency return, while transfer across delays is asymmetric and task-dependent. (3) An explicit latency value can be essential when training across delays, whereas the benefit of visual history depends on the task. Residual deployment latency can be modeled and incorporated into policy training, although adaptation does not generalize uniformly across latency regimes.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Latency, Vision-Language-Action, VLA, Evaluation, Latency-Aware Agents

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_8ryGOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies the effect of inference latency on vision-language agents and proposes measuring deployment latency, fitting a stochastic timing profile, and replaying that profile during training and evaluation. Across 12 environments and three StarVLA architecture variants, the paper reports substantial degradation under hardware-timed execution and improvements in all 36 task–architecture pairs after profile-based training.

Critical strengths
  • Relevant evaluation problem. Allowing environments to advance during inference exposes failures that world-pausing evaluation can conceal.
  • Substantial empirical coverage. The study includes games and embodied-control environments, multiple architecture variants, and measured hardware timing.
  • Transparent limitations. The appendices document many implementation details and explicitly acknowledge several experimental confounds.
Critical weaknesses
  1. The central training comparison is missing. Table 1 compares profile-trained policies against policies trained without latency. This establishes that accounting for deployment delay can help, but does not establish the value of fitting and preserving the measured temporal profile. This is the most significant gap because it concerns the paper’s principal contribution.
  2. The five-deployment calibration experiment uses one OpenVLA-style policy on Flappy Bird. Temporal achieves a lower average return gap than IID, 10.8 versus 18.1, but IID is closer on three of the five deployments. As Appendix D acknowledges, the comparison changes both the delay model and inference-capacity modeling. Consequently, the large improvement over Fixed does not isolate the importance of temporal correlation. Broader validation across policies, tasks, and held-out deployment sessions is needed, with capacity modeling held constant.
  3. The incremental contribution over existing delay-aware control is insufficiently demonstrated. the delay aware control has been established in papers like RTC and ARLI [1]. The paper appropriately uses a standard adaptation recipe, making its contribution primarily empirical and methodological. Such a contribution can be valuable, but requires convincing evidence that deployment-fitted profiles provide information or performance unavailable from simpler delay models.
  4. The deployment conclusions need tighter scope. Six tasks use adjusted environment or control rates to accommodate VLA inference, including Flappy Bird at 10 Hz and MuJoCo tasks at architecture-dependent rates of 5–10 Hz. These settings support controlled comparisons but do not establish effectiveness at native task rates.

[1] Learning to Act While Waiting: RL Finetuning of Generalist Robot Policies Under Inference Latency, Zhu et al.

Questions for authors
  1. Does Temporal-profile training outperform fixed-mean, IID, and randomized-delay training under matched VLA training budgets?
  2. Does the calibration advantage persist when every profile uses identical worker-capacity modeling and is evaluated on held-out sessions?
  3. Can the transfer experiments be repeated with matched inputs, training budgets, evaluation episodes, and multiple training seeds?
1 comment
AuthorsOpenReview ↗

Response to Reviewer 8ryG

We thank the reviewer for noting the "substantial empirical coverage," the relevance of the evaluation problem, and the "transparent limitations" in the appendices.

[W1 / Q1] Central training comparison

Under a matched budget, both of our measured profiles score higher than Fixed and RTC on all three rows of Table R1. On L40S, Fixed reaches 3.95, IID 255.98 and Temporal 270.41. IID draws each latency at random from the measured distribution, so it also serves as the randomized-delay condition; we did not run VLA training with uniform or normal delays, and App. F has uniform-delay training for small policies (3 seeds). AR(1) and ARMA(1,1) generate slow requests at less than 1% of the measured share (Table R2). Training on our measured profiles improves over a fixed delay, and keeping the measured temporal structure improves replay fidelity.

[W2 / Q2] Calibration breadth, capacity modeling and held-out sessions

Over 30 held-out cases, our measured profiles roughly halve the replay error of a fixed delay: 3.81 (Temporal) and 4.58 (IID) percentage points, against 6.55 for Normal and 8.93 for Fixed (Table R8). The 30 cases cover three tasks, two policies and five GPU types; each profile is fitted on 25 hardware runs and tested on 25 held-out runs. IID and Temporal use the same inference-worker model and the same joint sampling of action latency and busy time (App. D), and differ only in whether slow requests occur consecutively; Temporal has the lower error.

Table R8: Normalized mean absolute error between simulated and hardware scores over 30 cases (percentage points, relative to the zero-latency return).

FixedNormalIIDTemporal
8.936.554.583.81

[W3] Contribution relative to RTC and ARLI

RTC and ARLI are both in Table R1. On Flappy Bird with the RTX 3090 profile, RTC + ARLI reaches 438.00, against 414.52 for Temporal. On L40S, training on our measured profiles alone reaches 255.98–270.41, against 58.46 for RTC + ARLI, and ARLI changes a Temporal checkpoint by −4.30% to +4.23%.

[W4] Environment rates

A deployment cannot run its control loop faster than the policy produces actions, and running it slower gives up control the policy could provide. For Flappy Bird, AirRaid and the four MuJoCo tasks, whose native rates exceed VLA inference speed, we therefore set each architecture's control rate to its RTX 3090 inference rate, at which its mean latency spans about one decision step. All three architectures then face the same delay in decision steps, so the comparison reflects how a policy acts under delay rather than which model runs fastest. The other six tasks run at their standard benchmark rate, and profile training improves all 18 of their pairs (§4.2). Delays of many decision steps are covered by the fixed-delay sweeps (§4.4).

[Q3] Transfer with matched inputs, budgets and seeds

The transfer pattern holds across three tasks. In Figure 6, Demon Attack, Deadly Corridor and MIKASA all transfer more reliably from longer training latencies to shorter evaluation latencies (§4.4), so the conclusion does not rest on a single cell. The training comparison of Table R1 averages 3 seeds per condition.

Reviewer_q7YnOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper shows that VLA evaluation lacks latency awareness: most existing benchmarks pause the simulation while the policy computes each step. It introduces a framework to profile and replay the latency characteristics of a given VLA on given hardware. Using these profiles, the authors train latency-aware models that account for their own inference delays when predicting the “correct” set of action chunks. They evaluate a few different profile models and show that the temporal profile model transfers best to a few different hardware platforms on a variety of robotic tasks.

Critical strengths

(a) This is a critical issue in many existing robotics benchmarks and is a well known limitation. Any progress in measurement, benchmarking and formulation of this important robotics problem is appreciated.

(b) Extensive experimentation showing the world pausing issue on many robotics benchmarks. The work is carefully executed.

Critical weaknesses

(a) A more general way to formulate this problem would be a producer–consumer queueing pattern, letting the simulation state evolve asynchronously with the prediction state. That gives a more natural way to reason about latency and performance. The simulator does model workers and capacity, but its execution rules (ordered application, drop-on-full, hold-last-action) are fixed and never studied, and no asynchronous execution baselines (e.g., RTC) are compared. The formulation, while reasonable, is limited.

(b) Beyond the GPU model, The framework does not systematically control for any of the characteristics that affect latency. There are many such considerations explored in prior systems, vision and robotics literature. Some of them are software and hardware contention (e.g., other perception models or safety systems running on the robot), memory pressure, intrinsic semantics of the task at hand (variable compute for perception/planning/control), network delays, robot's thermal or power constraints. The interesting study in my opinion is to model the effect of these considerations while simultaneously modelling the latency distribution, which would be multimodal in these difficult cases. In their case, the latency distribution is concentrated around a single dominant mode with occasional slow bursts, a regime where simple scheduling mechanisms are generally performant. For related work on scheduling under latency in the unimodal case, the authors may find the streaming perception literature useful (e.g., Li et al., 2020, “Towards Streaming Perception”).

How would their framework handle these effects, and how much does performance degrade when a policy trained on one profile is deployed under a different one, even one within the range of conditions seen in training? For example, a policy trained on two profiles, (i) RTX 3090 with co-located contention processes (e.g., a safety harness) and (ii) A100 with no contention, could be evaluated on A100 with contention processes. Each factor is in distribution, but their combination is not.

(c) Is the identification of “world pausing” novel? Many prior papers describe this problem (e.g. RTC, RTC during Training), and the tools used (delay models, training under delay) are well established. This is a systems benchmarking paper, which is well and good, but the novel contribution is unclear. One factor in accepting such a paper would be new insights uncovered by the systematic study. However, most of the results are incremental. For example, the main training result compares only against delay-unaware policies (which is unsurprising in my view). No experiment shows that world-paused evaluation changes ranking of the models themselves, and a fair comparison would substantially strengthen the paper’s significance.

Questions for authors

(a) What are the novel insights that their systematic investigation uncovers? The paper reads like a tech report to me, and I invite the authors to highlight their novel approach or novel insights, whichever is applicable.

(b) The considered factors affecting latency are not controlled for and are due to transient reasons. A more interesting study in my opinion would have controlled for the factors that affect latency in the first place, and shown the improvements of their latency profiling function while modelling multimodal latency distributions. Any reason to not model the latency source at all?

3 comments
AuthorsOpenReview ↗

Response to Reviewer q7Yn (1/3)

We thank the reviewer for the detailed reading, and for describing the work as "carefully executed" on "a critical issue in many existing robotics benchmarks."

[W(a)] Asynchronous formulation and execution rules

We keep three runtime rules fixed by design: actions are applied in order, observations are dropped when the queue is full, and the last action is held while a new one is pending. They are the rules of the runtime, and hardware-timed execution and replay must share them for calibration to be meaningful. The simulator already follows a producer–consumer pattern: the environment advances while requests are pending, and workers accept or drop observations according to capacity (App. B.2). Table R1 adds RTC as the requested asynchronous baseline: RTC is 6.99% below Temporal on Flappy Bird with the RTX 3090 profile and 85.65% below it with the L40S profile.

[W(b) / Q(b)] Other sources of latency

We model the measured end-to-end latency by design. A policy experiences only the resulting latency, and a deployment can measure it without instrumenting each source. Three results support this choice:

  1. When a background program alternates on the GPU, the latency is bimodal (about 76 ms and 128 ms), and our Temporal policy completes every 3,600-frame episode, while IID reaches 201.3, Fixed 94.2 and the original policy 16.4 (Tables R3 and R4).
  2. When the VLA and the simulator run on two different machines, the end-to-end profile scores 41.10 against 3.92 for the model-forward profile on Flappy Bird, and 549.75 against 104.75 on Demon Attack (Tables R5 and R6).
  3. When a background program occupies the GPU for short (0.2–0.8 s) and long (2–5 s) periods, slow requests make up 5.14% of the Temporal sequences against 4.29% measured, and fewer than 0.03% of the AR(1) and ARMA(1,1) sequences (Table R2).

The review describes "a single dominant mode with occasional slow bursts" as a regime where simple scheduling performs well, and streaming perception (Li et al., 2020) is a useful reference for that unimodal case. Slow bursts, which are runs of slow requests, occur most often in the L40S profile, and there training on our measured profiles reaches 255.98–270.41, against 3.95 for Fixed and 38.81 for RTC (Table R1). A single delay or scheduling rule therefore does not cover this regime. The end-to-end latency we measure includes inter-process communication, preprocessing, inference, action decoding and result delivery (App. C), so GPU contention and network transfer enter the profile directly.

Under a competing GPU workload, our Temporal policy completes every episode, including under a schedule it never saw in training. While Flappy Bird runs with OpenVLA on an RTX 3090, a background program on the same GPU alternates 5 s off and 5 s on during profiling and training, so the slow periods are known. One decision step lasts 100 ms.

Table R3: Latency from observation to action with and without the background program (p95: 95% of requests are faster than this value).

ConditionMean (ms)p95 (ms)
No background program70.7477.88
Program alternating, off phase75.6682.61
Program alternating, on phase127.67134.82

After a slow request (over 100 ms), the next request is also slow with probability 0.943 in the measured data; Temporal sampling gives 0.833 and IID 0.321 (20 sampling seeds).

Table R4: Return (mean ± SD over 3 seeds). The random schedule draws off and on durations from U(0, 10) s, with the same 5 s mean and slow periods of unpredictable length, and was not seen in training. 444.60 is the return of an episode that completes all 3,600 frames.

PolicyAlternating schedule (seen in training)Random schedule (not seen)No background program
Original policy (latency-unaware)16.4 ± 3.813.1 ± 4.220.7 ± 5.1
Fixed94.2 ± 17.678.5 ± 15.3214.8 ± 26.4
IID201.3 ± 21.9167.8 ± 18.7308.5 ± 29.3
Temporal444.60 ± 0.0444.60 ± 0.0444.60 ± 0.0

Our Temporal policy completes every episode in all three conditions. The episode length matches the other Flappy Bird experiments, so the differences above IID and Fixed are lower bounds. Without the background program, latency (about 76 ms) is below one decision step, yet each action still lands on a later frame than the one observed, and the original policy reaches only 20.7.

AuthorsOpenReview ↗

Response to Reviewer q7Yn (2/3)

Our end-to-end profile gives 5.2 to 10.5 times the real-time return of a model-forward profile when the two latencies differ. We run the VLA on one machine (an RTX PRO 6000) and the simulator on another (an RTX 3090), so every action also waits for the transfer between the two machines, and the two latencies differ by about 430 ms. End-to-end latency counts everything between the observation and the action; model-forward latency counts only the model's forward pass, without input preprocessing, action transfer to the CPU and decoding. Both comparisons use OpenVLA trained for 5,000 episodes, and each condition averages 3 seeds.

Table R5: Latency when the VLA and the simulator run on two different machines.

TaskMean model-forward latency (ms)Mean end-to-end latency (ms)Mean gap (ms)
Flappy Bird25.51455.48429.97
Demon Attack30.30460.70430.40

Table R6: Real-time return (mean ± SD) of policies trained under each profile.

TaskModel-forward profileEnd-to-end profile
Flappy Bird3.92 ± 0.0941.10 ± 12.35
Demon Attack104.75 ± 50.48549.75 ± 273.14

On the same machine, where the two latencies are close, the two profiles give returns within one standard deviation of each other (Flappy Bird 433.79 ± 47.14 vs. 422.47 ± 62.22; Demon Attack 1508.00 ± 354.20 vs. 1680.00 ± 263.74). Our end-to-end profile matches the model-forward profile when the two latencies are close, and when they differ it records all the latency a policy experiences, with 5.2 to 10.5 times the real-time return.

[W(b)] Deploying under conditions not seen in training

Our framework measures each target deployment, so a policy is trained on the profile of the deployment it will run on. The reviewer's combined example, training on an RTX 3090 with a background program and on an A100 without one, then testing on an A100 with a background program, combines conditions that were not measured together. This is a good direction that we have not yet run. Our Temporal policy already completes every episode under a background schedule it never saw in training (Table R4).

AuthorsOpenReview ↗

Response to Reviewer q7Yn (3/3)

[W(c) / Q(a)] Novel insights

World-pausing has been described by prior work, including RTC. Our contribution is a measured-profile framework for hardware-faithful real-time evaluation and training: a profile measured on a deployment is replayed on a controlled simulation clock, so the host's own speed does not enter the simulated delay, and the replay reproduces hardware-timed return within 3.81–4.58 percentage points (Table R8). The framework produces five findings:

  1. VLA policies keep a median of only 6.6% of their zero-latency return in real time, and zero-latency evaluation selects a model that is not the best one in deployment (Table R7).
  2. Transfer across latency is asymmetric: tested at 8 frames, a policy trained at 16 frames reaches 85% of the score of a policy trained at 8 frames; tested at 16 frames, a policy trained at 8 frames reaches only 27% of a policy trained at 16 frames (§4.4).
  3. A policy trained across latencies must be told the current latency (latency aliasing). On Flappy Bird at zero latency, return is 433.8 with the latency value and 18.8 without it, and at four frames 267.6 and 4.5 (Table 4).
  4. End-to-end latency is the quantity to measure: when the VLA and the simulator run on two machines, the end-to-end profile scores 5.2 to 10.5 times the return of the model-forward profile (Table R6).
  5. Training at a fixed delay is not enough: on the L40S profile, Fixed reaches 3.95, against 255.98–270.41 for our measured profiles (Table R1).

The comparison of rankings that the review asks for is Table R7: on all three game tasks, GR00T has the highest score at zero latency, and a different architecture has the highest score in real time. All checkpoints in Table R7 were trained for 5,000 updates, so the columns compare architectures under the same budget.

Table R7: Best architecture under zero-latency and real-time evaluation (Table 1).

TaskBest at zero latencyBest in real time
AirRaidGR00T (2.10k; π0.5 1.94k)π0.5 (1.39k; GR00T 733.8)
AtlantisGR00T (39.66k; range 39.56k–39.66k)OpenVLA (18.59k; range 4.37k–18.59k)
Demon AttackGR00T (2.37k; range 2.35k–2.37k)OpenVLA (584.4; range 112.3–584.4)

A new evaluation of profile-trained policies (100 episodes per checkpoint) shows the same effect. On Flappy Bird, the ranking changes from OpenVLA > π0.5 > GR00T in a paused world to GR00T > OpenVLA > π0.5 at the target latency. Evaluating in a paused world would select OpenVLA, and GR00T is the best model in deployment.

Table R1 adds latency-aware conditions (Fixed, IID, RTC, ARLI) to the comparison with latency-unaware policies in Table 1, and Table R4 adds Fixed and IID under a competing GPU workload. SFT and DAgger are standard (§4.1) and serve as probes of what latency training can recover.

Reviewer_cjuUOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how inference latency affects vision-language-action (VLA) policies when environments continue evolving during computation. It introduces a deployment-specific latency profiling and replay framework, including a Temporal model that captures consecutive slow requests, and trains VLA policies through RL-teacher distillation. Experiments across 12 tasks and three architectures demonstrate severe performance degradation under hardware-timed execution and improvements after profile-based training. Additional analyses examine latency-model fidelity, cross-delay transfer, explicit latency signals, and visual history.

Critical strengths
  1. The paper addresses an important practical problem: conventional world-pausing evaluation overlooks inference latency during real-time control. Experiments across 12 environments and three VLA architectures demonstrate substantial performance degradation, highlighting the importance of latency-aware evaluation and training.

  2. The proposed latency profiling and replay framework provides a systematic approach to modeling deployment-specific delays. Evaluations across five GPU deployments, together with extensive latency adaptation and transfer analyses, offer valuable insights into how inference latency affects VLA performance.

Critical weaknesses
  1. Although Temporal profiles reproduce hardware-timed performance more accurately, the paper does not establish whether this improved fidelity benefits policy learning compared with Fixed- or IID-profile training. The contribution of modeling consecutive slow requests to the reported performance gains is therefore not demonstrated. 
  2. The adaptation framework relies on task-specific RL teachers trained with simulator rewards under deployment-specific latency profiles. This dependence on task-specific simulation and reward functions limits its applicability to general-purpose VLA deployments where such training resources are unavailable. 
  3. The main results report improvements across all 36 task–architecture pairs without confidence intervals or multiple independent training runs. This limits the statistical reliability of the reported improvements, particularly for tasks exhibiting relatively small performance gains.
Questions for authors
  1. Does Temporal-profile training outperform Fixed- or IID-profile training under matched training budgets and hardware-timed evaluation?

  2. How could the proposed adaptation framework be applied to VLA tasks without task-specific RL teachers, simulators, or reward functions?

  3. ow consistent are the reported improvements across independent training runs, particularly for tasks with relatively small gains?

1 comment
AuthorsOpenReview ↗

Response to Reviewer cjuU

We thank the reviewer for the careful reading, and for recognizing that the paper addresses "an important practical problem" and that the framework offers "a systematic approach to modeling deployment-specific delays."

[W1 / Q1] Temporal, IID and Fixed training under matched budgets

Both of our measured profiles train better than Fixed and RTC, and our Temporal profile gives the most faithful replay. Under a matched 5,000-step budget, IID and Temporal score higher than Fixed and RTC on all three rows of Table R1; on L40S, Fixed reaches 3.95, against 255.98 for IID and 270.41 for Temporal. IID and Temporal reach similar training returns in these settings. In replay, our measured profiles reach 3.81 (Temporal) and 4.58 (IID) percentage points, against 8.93 for Fixed (Table R8), and Temporal also reproduces the runs of slow requests (Tables R2 and R3). When slow requests come in long runs, as under a competing GPU workload, our Temporal policy completes every 3,600-frame episode, while IID reaches 201.3 (Table R4). We therefore keep Temporal as the default profile, with IID as the second measured profile.

[W2 / Q2] RL teachers and task-specific simulation

We use task-specific RL teachers by design, because a standard recipe keeps the latency condition as the variable of interest (§4.1). An RL teacher is a policy trained with RL in the task simulator, and the VLA learns to copy its actions at the target latency. Three of the four steps in our framework need no RL teacher: measuring latency, replaying it in the simulator and evaluating a policy. Only training the VLA uses one.

RL refinement methods such as ARLI also need a simulator and a reward. We trained a policy across latencies only on Demon Attack, using its teacher. At a four-frame latency it also scores higher on Atlantis and AirRaid than a policy trained at zero latency (5,010 vs. 2,000 and 178 vs. 15, §4.5, Figure 7), and these two games were not used for training. On Asterix, the third companion game, the order reverses at four frames (178 vs. 212). Adapting on Demon Attack alone raises the scores on Atlantis and AirRaid, which have no teacher of their own.

[W3 / Q3] Consistency across independent runs

All 36 improvements in Table 1 are at least 39%, while the seed-to-seed SD of our trained policy is 2% to 7% of its mean (Table R1). Table 1 compares Target/Real with Zero/Real on the same evaluation environment seeds. The smallest improvement among the 36 pairs is OpenVLA on Intercept Grab (Fast), from 18.6 to 25.8 (+39%), and every other pair improves by at least 54%. To measure the variation between training runs, we repeat three settings with 3 seeds each (Table R1): the SD of Temporal is 8.9 on 414.52, 15.3 on 270.41 and 1.82 on 25.80. Training all 36 pairs with several seeds would multiply the compute of the whole study, so we repeat these three settings.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response (1/2): Summary and Training Comparison

Dear Reviewers,

We thank the reviewers for describing the problem as "important" with "valuable insights" (cjuU), as "a critical issue in many existing robotics benchmarks" with work that is "carefully executed" (q7Yn), and for noting the "substantial empirical coverage" and "transparent limitations" (8ryG). The reviewers' suggestions are all valuable, and we ran the experiments they asked for.

Summary

We propose latency profiles: we measure the action latency of a target deployment, fit a profile of which delays occur and whether slow requests arrive in runs, and replay new delay sequences from it in simulation for both evaluation and training. Our two measured profiles keep the trace itself: IID draws each latency from the measured distribution, and Temporal also keeps runs of slow requests. Fixed (one constant latency) and Normal (mean and variance) are summary-statistic baselines. The new experiments compare our profiles with RTC, ARLI, AR(1) and ARMA(1,1), and add a background program that competes for the GPU, a two-machine deployment and model rankings. The results are:

  1. Calibration: our measured profiles roughly halve the replay error of a fixed delay. Over 30 held-out cases, the average gap to hardware is 3.81 (Temporal) and 4.58 (IID) percentage points, against 8.93 for Fixed (Table R8, response to 8ryG).
  2. Evaluation: zero-latency evaluation selects a different best architecture than real-time evaluation on all three game tasks of Table 1 (Table R7, response to q7Yn).
  3. Training: training on our measured profiles scores higher than Fixed and RTC in all three settings. On L40S, Fixed reaches 3.95 and RTC 38.81, against 255.98 (IID) and 270.41 (Temporal) (Table R1).
  4. Training on our measured profiles alone also scores higher than RTC + ARLI on L40S, which reaches 58.46 (Table R1).
  5. Our end-to-end profile gives 5.2 to 10.5 times the real-time return of a model-forward profile when the VLA and the simulator run on different machines (Table R6, response to q7Yn).
  6. Under a competing GPU workload, our Temporal policy completes every 3,600-frame episode, including under a schedule not seen in training (Table R4, response to q7Yn).

Additional Experiments

Training on our measured profiles compared with Fixed, RTC and ARLI [cjuU W1, Q1; q7Yn W(a), W(c); 8ryG W1, Q1, W3]. We train GR00T under the Temporal, IID and Fixed profiles with a matched 5,000-step budget and add training-time real-time chunking (RTC) as an asynchronous baseline. ARLI freezes the base model and trains a noise policy with RL to adapt to latency; we apply it to the Temporal and RTC checkpoints. MIKASA uses H8 policies and is evaluated on a real GPU with hardware timing. Flappy Bird returns average 100 episodes and MIKASA scores average 200 episodes. Each condition averages 3 seeds.

Table R1: Return, mean ± SD over 3 seeds, under each training condition. The last two columns apply ARLI to the Temporal and RTC checkpoints. MIKASA is Intercept Grab (Fast).

TaskTemporalIIDRTCFixedTemporal + ARLIRTC + ARLI
Flappy Bird, RTX 3090414.52 ± 8.9388.23 ± 14.8385.55 ± 20.2329.01 ± 22.8420.60 ± 11.6438.00 ± 9.0
Flappy Bird, L40S270.41 ± 15.3255.98 ± 29.938.81 ± 10.93.95 ± 2.3258.78 ± 20.158.46 ± 17.6
MIKASA, RTX 309025.80 ± 1.8224.71 ± 0.8721.80 ± 1.6724.35 ± 1.5626.89 ± 1.4224.16 ± 0.58

Fixed and RTC have a lower mean return than both measured profiles in all three settings. The gap is largest on the L40S profile, which contains the most runs of slow requests: Fixed reaches 1.5% of the Temporal return and RTC 14.4%. Training on our measured profiles gives a higher mean return than Fixed and RTC in every setting. IID and Temporal reach similar training returns, and Temporal's additional advantage is replay fidelity (Tables R2 and R8).

ARLI refines a frozen policy with RL and changes a Temporal checkpoint by −4.30% to +4.23%. On Flappy Bird with the RTX 3090 profile, RTC + ARLI reaches 438.00. On L40S, training on our measured profiles alone reaches 255.98–270.41, against 58.46 for RTC + ARLI.

AuthorsOpenReview ↗

General Response (2/2): Classical Time-Series Models and Where to Find the Other Results

Our Temporal profile reproduces runs of slow requests that AR(1) and ARMA(1,1) miss [cjuU W1, Q1; 8ryG W1, Q1; q7Yn W(b), Q(b)]. We call a request slow when its latency exceeds 118.99 ms, the slowest 5% of the measured requests. The classical models are AR(1), x_t = μ + φ(x_{t−1} − μ) + ε_t, and ARMA(1,1), which adds the term θε_{t−1}. While OpenVLA plays Flappy Bird, a background program keeps the GPU busy for short intervals (0.2–0.8 s) and long intervals (2–5 s), each drawn uniformly. Each model is fitted to the same training sessions and generates 20 sequences, which we compare with two held-out measured sessions.

Table R2: Slow requests (latency above 118.99 ms, the slowest 5% of the measured requests) in measured and generated sequences. Run length averages the sequences that contain slow requests.

SourceShare of slow requestsMean number of slow requests in a row
Measured, held-out sessions4.29%6.12
Temporal5.14%3.64
AR(1)0.027%1.73
ARMA(1,1)0.023%1.39

Slow requests make up 4.29% of the measured sequences and 5.14% of the Temporal sequences, while AR(1) and ARMA(1,1) produce less than 1% of the measured share. Temporal also produces the longest runs of slow requests among the generated sequences, 3.64 against 1.73 for AR(1) and 1.39 for ARMA(1,1). When a background program alternates on the GPU, a slow request is followed by another slow request with probability 0.943 in the measured data, 0.833 under Temporal and 0.321 under IID (Table R3, response to q7Yn). IID matches the measured share of slow requests by construction and samples each request independently. Models with a single mean level, such as AR(1) and ARMA(1,1), cannot represent a slow state, which Temporal represents explicitly. Our Temporal profile is the closest to the measured data on both the share of slow requests and their runs, so we use it as the default profile when traces contain slow periods.

Other new results. The 30-case calibration is in Table R8 (response to 8ryG). GPU competition, two-machine latency and model rankings are in Tables R3 to R7 (response to q7Yn).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.