acceptodds
Under review as a conference paper at ICLR 2027

TRACE: Delay-Adaptive Temporal Residuals for Asynchronous Robotic Control

Abstract

Asynchronous inference supports real-time action execution and improves control responsiveness by allowing the robot to continue acting while the next action sequence is generated. However, overlapping inference with execution cannot eliminate computational latency: the robot and environment may evolve before predicted actions take effect, creating an observation-to-execution mismatch that requires correction. We introduce TRACE (Temporal Residual Adaptation for Asynchronous Control with Frozen Experts), a temporal residual injection framework for frozen flow-based action generators. Its central mechanism compresses causal observation–command history and committed actions into delay-conditioned context, then injects this context into internal action representations through gated cross-attention. TRACE preserves the backbone's original modules and pretrained parameters while learning an additional temporal residual pathway. Its injection mechanism is orthogonal in design to delay-aware policy training and sampling-time control, enabling complementary asynchronous enhancements around a frozen backbone. Experiments on Kinetix and LIBERO show aggregate success gains over naive asynchronous execution for the initial adapted configurations. An independent Kinetix evaluation further shows improved aggregate success with the calibrated deployment configuration.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Asynchronous robotic control, Vision-Language-Action Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_2S75OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the inference latency of robot policies by proposing a new model that compensates for this latency. It inserts a latency-gated cross-attention adapter into a frozen flow-based policy. The adapter encodes the state–instruction history, visual changes, and the committed action prefix, injects them into the model's intermediate representations, and is supervised only on the executable action suffix.

Critical strengths
  1. This paper carefully explores the latency issue faced by current VLA policies and analyzes the strengths and weaknesses of both synchronous and asynchronous approaches. This is a direction worthy of in-depth research.
  2. The proposed adapter is plug-and-play, making it more flexible and convenient than methods that require retraining the base model.
  3. The figures are clear, the writing is precise, and the claims are appropriately measured.
Critical weaknesses
  1. The latency setting is overly idealized: the latency at inference time is assumed to be identical to the default latency used during training.
  2. All experiments are conducted in idealized simulation environments, with no real-robot evaluation. Latency is an even more critical issue in real-world deployment.
  3. The performance gains in the experimental results are limited and do not fully demonstrate the advantages of the proposed model. Moreover, the experiments are insufficient to support the effectiveness of the method.
  4. The experiments lack a direct comparison with REMAC, even though the paper repeatedly positions it as a reference point. In addition, the RTC baseline is missing on the LIBERO tasks.
  5. The additional inference overhead introduced by the adapter is not taken into account. This is particularly important for a latency-focused method, as the adapter itself may add to the very delay it aims to compensate for. The authors should report the adapter’s runtime cost.
Questions for authors
  1. The authors could validate the practical effectiveness of the method through real-robot experiments in which the VLA is deployed remotely rather than on-device.
  2. What is the additional inference overhead introduced by the adapter (e.g., latency in ms, FLOPs, and parameter count)?
Reviewer_bsRPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TRACE addresses the observation-to-execution mismatch caused by inference latency in asynchronous robotic control. It introduces a delay-conditioned temporal adapter for frozen flow-based action generators. The adapter combines observation–command history, visual feature differences, and committed actions into compact context tokens, which are injected into internal action representations through gated cross-attention. Training supervises the executable action suffix, with domain-specific regularization. Experiments on Kinetix and LIBERO evaluate performance under simulated delays. The results show improvements over RTC on Kinetix and a small average improvement over naive asynchronous execution on LIBERO.

Critical strengths

The paper addresses an important problem in robotic deployment: maintaining continuous action execution does not eliminate the mismatch between launch-time observations and execution-time states. I find the proposed temporal adaptation approach interesting and practically motivated, with the following strengths.

  1. Temporal compensation within a frozen action generator. TRACE introduces a trainable residual pathway into the generator’s internal action representations. Cross-attention allows the correction to depend on the evolving action features throughout flow-based generation, while keeping the pretrained backbone parameters fixed. This provides a concrete mechanism for adapting an existing policy to delayed execution without retraining the full model.
  2. Joint use of historical responses and committed actions. The temporal context combines complementary information: observation–command history describes how the system has responded to recent commands, while the committed prefix specifies the commands scheduled before takeover. Visual feature differences further capture recent scene changes. Compressing these signals into delay-conditioned context is a sensible way to inform compensation without requiring an explicit dynamics rollout.
  3. Encouraging evidence on Kinetix with relevant controls. In the independent confirmation evaluation, calibrated TRACE achieves 77.69% average success over positive delays, compared with 74.59% for RTC. The initial evaluation also includes a guidance-matched comparison with and without the adapter, providing useful evidence of an additional benefit from temporal adaptation. Separating deployment calibration from subsequent evaluation strengthens this empirical contribution.
Critical weaknesses

Despite the reasonable motivation and encouraging Kinetix results, the current evidence does not yet sufficiently establish the method’s practical benefits for visual robotic control. My main concerns involve deployment validation, experimental attribution, and reproducibility.

  1. Insufficient validation under actual inference latency. All experiments use simulated delays, and no real-robot evaluation is reported. These delays are specified independently of measured inference time, although TRACE introduces additional temporal encoding and repeated cross-attention operations. Consequently, the experiments do not establish whether the compensation benefits outweigh the additional computational latency in deployment. End-to-end timing measurements and closed-loop evaluation under latency variation would materially strengthen the paper.
  2. Limited evidence of effective delay compensation in visual manipulation. On LIBERO, average success improves only from 82.47% to 82.85%. More importantly, the naive asynchronous baseline shows little degradation over the reported delay range relative to its zero-delay performance. The current setting therefore provides a limited stress test of the problem the method is intended to solve. Comparisons against relevant delay-aware methods, such as RTC [1] or training-time RTC (TT-RTC) [2], are also missing from the LIBERO evaluation. More challenging delay conditions and uncertainty estimates are needed to assess the practical significance of the improvement.
  3. The contribution of the core temporal mechanism is not sufficiently isolated. TRACE combines several components, including observation–command history, visual differences, prefix encoding, delay gating, and auxiliary displacement supervision. However, the main ablations focus on sampling guidance and residual attenuation, while the history-content controls remain incomplete. The current evidence therefore does not clearly distinguish the benefit of temporal information from that of additional adaptation capacity or other design choices. Matched comparisons with reduced temporal inputs and a generic adapter would help establish the specific contribution of the proposed mechanism.
  4. Experimental reliability and reporting completeness require clarification. The policy experiments use a single training seed per configuration. In LIBERO, initial states 0 through 4 were used for configuration selection and are also included in the final evaluation over states 0 through 49. Reporting results separately on states 5 through 49 would provide more independent evidence. Furthermore, Appendix D states that evaluation was completed for delays from zero to four control steps, whereas the manuscript reports only delays from zero to three control steps. The results for a delay of four control steps should be provided, together with an explanation of why this condition was omitted from the reported results.
  5. The supplementary code does not provide a complete reproduction path for the reported closed-loop results. The package provides the SmolVLA model and training implementation, but omits the full asynchronous controller, LIBERO rollout pipeline, and Kinetix evaluation code. There is also a configuration ambiguity: the sampling interface defaults to clamp_prefix=True, whereas the reported LIBERO setup requires clamp_prefix=False. The appropriate implementation exists, but the missing evaluation entry point prevents verification of the exact configuration used. This is a reproducibility concern rather than evidence that the reported experiments used an incorrect sampler.

References

[1] Kevin Black, Manuel Y. Galliker, and Sergey Levine. Real-Time Execution of Action Chunking Flow Policies. Advances in Neural Information Processing Systems, 38, 2025.

[2] Kevin Black, Allen Z. Ren, Michael Equi, and Sergey Levine. Training-Time Action Conditioning for Efficient Real-Time Chunking. arXiv preprint arXiv:2512.05964, 2025.

Questions for authors
  1. Could you add real-robot experiments that account for the inherent and potentially different delays of the sensors used, including image acquisition, data transmission, and proprioceptive feedback? Please demonstrate whether TRACE remains effective under these sensor delays and timing variations, and report end-to-end latency including the adapter’s computational overhead. I consider this validation important for substantiating the claimed benefits for practical robotic deployment.
  2. Can you provide matched ablations that remove observation–command history, visual differences, or displacement supervision while retaining comparable adaptation capacity? These comparisons would help determine whether the gains specifically depend on the proposed temporal information.
  3. Do the LIBERO improvements persist across independent training runs and under delays that meaningfully degrade the baseline? Comparisons with RTC [1] or training-time RTC (TT-RTC) [2], together with uncertainty estimates for the paired success-rate differences, would materially strengthen the assessment.
  4. Appendix D states that LIBERO evaluation included a delay of four control steps. What were the results, and why was this condition omitted from the reported tables? Can you also report results separately on initial states 5 through 49, which were not used for configuration selection?
  5. Please confirm whether the reported LIBERO experiments disabled prefix clamping, as described in the manuscript, despite the supplementary sampling interface enabling it by default. Can you provide the exact evaluation configuration and asynchronous controller implementation needed to reproduce these results?

References

[1] Kevin Black, Manuel Y. Galliker, and Sergey Levine. Real-Time Execution of Action Chunking Flow Policies. Advances in Neural Information Processing Systems, 38, 2025.

[2] Kevin Black, Allen Z. Ren, Michael Equi, and Sergey Levine. Training-Time Action Conditioning for Efficient Real-Time Chunking. arXiv preprint arXiv:2512.05964, 2025.

Reviewer_XiWvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TRACE (Temporal Residual Adaptation for Asynchronous Control with Frozen Experts), an adapter for frozen flow-based robotic action generators. The problem is the mismatch between the state observed at query launch and the state at which a newly generated action chunk begins execution. During inference, the controller executes a committed prefix Pt=(at,…,at+d−1)P_t=(a_t,\ldots,a_{t+d-1}) over a planned delay δ=dΔ\delta=d\Delta.

TRACE encodes (i) recent observation–command history Hts\mathcal{H}_t^s, (ii) visual feature differences Htv\mathcal{H}_t^v when images are available, and (iii) the committed prefix PtP_t into delay-conditioned context tokens TtT_t. These tokens condition cross-attention residuals inserted into selected frozen action-generator layers:

= h_{\ell} + g(\delta) r_{\ell}, \qquad g(\delta) = 1 - \exp\left(-\frac{\delta}{\delta_{\mathrm{ref}}}\right).$$ At zero delay, the adapter is bypassed. Training uses flow-matching supervision only on the executable action suffix ($i\ge d$), with a velocity-deviation regularizer on Kinetix and an auxiliary end-effector displacement objective on LIBERO. On 12 Kinetix levels, the initially evaluated binary-guided TRACE achieves 77.29% mean success over $d=1,\ldots,4$, compared with 74.52% for the guidance-matched, unadapted hard-masking control and 63.49% for Naive Async. After deployment calibration, TRACE reaches 77.69% on a separate evaluation cohort, compared with 74.59% for RTC. On 40 LIBERO tasks, however, TRACE improves positive-delay success only from 82.47% to 82.85% over $d=1,2,3$.
Critical strengths
  1. TRACE preserves the pretrained policy weights and provides an explicit zero-delay bypass. More importantly, the initial Kinetix comparison includes an unadapted binary-prefix-guidance control: TRACE (binary) achieves 77.29% success, compared with 74.52% for Hard masking, a gain of 2.77 percentage points within the same guidance category. The independent evaluation cohort also reports 77.69% for calibrated TRACE versus 74.59% for RTC. These results provide evidence that the complete adapted system can improve delay robustness on Kinetix, although they do not identify which input branch causes the gain.

  2. The authors match episode identities, initial observations, and sampling noise within evaluation cohorts. They clearly separate Kinetix development calibration from subsequent confirmation and report several unsuccessful or exploratory modifications rather than presenting all variants as conclusive ablations. The explicit discussion of simulated versus real inference delay is also welcome.

Critical weaknesses
  1. Table 1 reports Naive Async success of 82.60% at d=0d=0, followed by 82.35%, 82.20%, and 82.85% at d=1,2,3d=1,2,3. Thus, baseline success is essentially flat rather than showing a systematic decline as delay increases. TRACE averages 82.85% versus 82.47% for Naive Async over the positive delays, a difference of only 0.38 percentage points (23 additional successful evaluations over 6,000 task–state–delay instances). Such a small aggregate difference does not by itself establish a reliable improvement. Paired uncertainty estimates and per-task analyses are needed, particularly because the same tasks and initial states are reused across delays. Appendix D also records a two-success difference between the two nominally equivalent zero-delay runs (1,652 versus 1,650 out of 2,000), reinforcing the need to quantify run-level variability rather than to interpret small differences without uncertainty quantification.

There is a specific reporting concern: Appendix D states that the completed 20,000-episode LIBERO evaluation covered d=0,…,4d=0,\ldots,4, yet the main result tables report only d=0,…,3d=0,\ldots,3. The d=4d=4 full-cohort results should be provided, with an explanation for their omission. Appendix C's five-state d=4d=4 sensitivity results are not a substitute for the matched 50-state comparison. The model is trained with delays as large as d=10d=10, but the principal evaluation covers a much narrower range. Under the paper's single-worker schedule, d≤R=5d\le R=5; d=5d=5 could therefore be tested at the existing stride, whereas larger delays would require a changed stride and appropriately matched baselines. Finally, LIBERO compares only against Naive Async, not an RTC-style or other appropriate delay-aware VLA baseline. This makes the method's advantage over existing asynchronous compensation strategies unclear.

  1. A central motivation is that PtP_t specifies intended future commands but not their realized effects, so history Hts\mathcal{H}_t^s and Htv\mathcal{H}_t^v should provide complementary information. Yet no reported closed-loop comparison independently removes history, removes the committed prefix, or tests their combination. Appendix C.1 states that the Kinetix controls do not remove history or isolate the encoder architecture, and Appendix C.2 confirms that the LIBERO history-content controls were paused before training. As a result, the Kinetix gain over Hard masking could arise from learned prefix conditioning, historical observations, the combined adapter capacity, or their interaction. The separate contributions of the Kinetix residual regularizer Lres\mathcal{L}_{\mathrm{res}} and LIBERO displacement loss Ldisp\mathcal{L}_{\mathrm{disp}} are likewise untested in matched closed-loop ablations. The offline visual-feature comparison in Table 14 is informative but does not demonstrate improved closed-loop success.

  2. The paper correctly states that the reported delays are simulated action-availability offsets, independent of measured inference time. This separation is useful experimentally, but it leaves the real-time deployment claim incomplete. On LIBERO, TRACE adds approximately 16.88M trainable parameters, including multiple temporal encoders and six residual-injection layers used during ten Euler steps. On Kinetix, guided sampling also uses a VJP through the adapted endpoint predictor. The paper does not provide wall-clock chunk-generation times, additional encoding and injection costs, GPU utilization, or deadline-miss rates relative to the corresponding baselines. If the added computation changes the achievable inference-to-takeover delay, comparing both methods at the same simulated dd may overstate the realized advantage. A latency-aware comparison should report success at the respective measured delays under the same hardware, batching, and queueing conditions.

  3. Both adapter instantiations are trained using a single reported training seed. Kinetix uses a distinct evaluation episode-seed cohort after calibration, which supports confirmation of the selected configuration but does not measure retraining variability. On LIBERO, the reference recipe was selected using initial states 0–4, which are then included in the 0–49 main evaluation; this does not invalidate the results, but it means the full cohort is not strictly untouched by model selection. Moreover, the released frozen backbones had already been trained on the source demonstration corpora, so the adapter-held-out episodes are not necessarily unseen by the complete system. These facts warrant a more precise statement about the scope of generalization and, ideally, validation across adapter training seeds and an untouched evaluation set. They do not by themselves establish data leakage or explain the increase in late-training MSE.

Questions for authors
  1. What are the exact per-suite and overall Naive Async/TRACE success rates at d=4d=4 in the completed 50-state evaluation described in Appendix D, and why were they omitted from Tables 1 and 9? Could the authors also evaluate d=5d=5 with the current R=5R=5 schedule, or a larger-delay setting with an appropriately increased takeover stride and matched baselines? Please report paired uncertainty estimates and clarify when the unadapted policy begins to degrade.

  2. Could the authors provide a matched closed-loop ablation with (a) prefix-only context PtP_t, (b) observation–command/visual history only (Hts,Htv)(\mathcal{H}_t^s,\mathcal{H}_t^v), and (c) full TRACE? A no-adapter control and separate removal of Lres\mathcal{L}_{\mathrm{res}} or Ldisp\mathcal{L}_{\mathrm{disp}}, as applicable, would further clarify the mechanism. How much of the improvement remains when the historical evidence is removed?

  3. What are the measured milliseconds per chunk for Naive Async, RTC where applicable, and TRACE on the same hardware? Can the overhead be decomposed into one-time context encoding, per-flow-step residual injection, and VJP guidance? Do these costs increase the effective delay or cause missed takeover deadlines at the reported control period Δ\Delta?

  4. What do paired task-level confidence intervals or paired bootstrap analyses show for the +0.38-percentage-point LIBERO gain and for the Kinetix improvement over Hard masking? How stable are results across independently trained adapter seeds? Can the authors evaluate the selected LIBERO configuration on initial states not previously used for candidate selection and distinguish adapter-held-out evaluation from evaluation on demonstrations unseen by the frozen backbone?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.