acceptodds
Under review as a conference paper at ICLR 2027

DEFLECT: Temporal Counterfactual Preference Learning for Delay-Robust Asynchronous VLAs

16 views

Abstract

Vision-Language-Action (VLA) policies increasingly rely on asynchronous inference to hide model latency behind ongoing robot motion. During inference, the robot continues executing its current action chunk while the next chunk is generated from the observation available at inference start. By the time the new chunk is executed, the robot or scene may have changed, so an action that fits the prediction-time state can be misaligned with the execution-time state. Training-time methods modify the policy input or supervised target based on the expected execution delay, but do not directly supervise how actions should change as the scene evolves between prediction and execution. Runtime repair methods improve chunk continuity or modify actions during deployment, but can still leave actions aligned with the prediction-time state rather than the execution-time state. Our key insight is that execution-time observations are unavailable when the online prediction is made but are available in recorded trajectories during offline training, making the prediction–execution mismatch observable during training. We propose DEFLECT, an offline post-training framework that turns this temporal contrast into preference supervision for a policy acting from stale inputs. Across Kinetix, LIBERO, and three real-robot tasks, DEFLECT improves delay robustness across the evaluated VLA policies and settings. At the longest supported delay, it improves success by 4.2 percentage points over VLASH on Kinetix and by 22.7 points over naive asynchronous execution with GR00T N1.7 on LIBERO.

Primary area:
applications to robotics, autonomy, planning
Keywords:
robot learning, vision-language-action models, asynchronous control, preference learning, delay robustness

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_W3VROpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper aims to address the performance degradation of large VLA policies caused by inference delays at test time. The proposed method, DEFLECT, fine-tunes a policy using preference learning. It uses a frozen reference policy to predict action chunks from observations at time tt and at the shifted time t+δt+\delta. The optimization objective augments the original SFT loss with a DPO loss that favors the action chunk generated from the observation at t+δt+\delta over the one generated from the observation at tt. The experiments evaluate a flow-matching policy on Kinetix and different VLA backbones on LIBERO, as well as three real-world tasks, with evaluations under different inference delays.

Critical strengths

1.The paper explains its central idea clearly, and inference delay is a critical problem for large models.

2.The paper evaluates the proposed method across different VLA backbones and in both simulation and real-world settings.

Critical weaknesses
  1. The motivation for preference learning is unclear. The current objective combines an SFT loss on the unshifted action chunk, weighted by 1.0, with a DPO loss favoring the shifted action chunk, weighted by 0.02. The contribution of the DPO term appears small compared with that of the SFT term. The paper also evaluates Future SFT, which replaces the DPO term with a regression loss. It is unclear to me why preference optimization would perform better than regression. The paper should provide further analysis and explanation of this difference. Furthermore, Future SFT is evaluated only for GR00T on LIBERO, without a real-world comparison. Since this is the closest baseline and the performance gap on LIBERO is small, comparisons across additional backbones and potentially real-world tasks would provide stronger evidence for the proposed method.

  2. The evaluation provides limited coverage of dynamic manipulation scenarios that require reactive control. The two simulation environments only partially reflect real-world scenarios involving dynamic scenes that require reactive actions. Kinetix is a 2D dynamic environment that evaluates a non-VLA policy, and its action space is simpler than that of typical manipulation tasks. LIBERO is a common VLA manipulation benchmark, but most tasks involve picking and placing static objects and may not place high demands on reactivity. Additional benchmarks featuring dynamic tasks would therefore better assess reactivity and robustness to inference delays—for example, RoboDojo[1], which includes a conveyor task.

[1]Chen, Tianxing, et al. "RoboDojo: A unified sim-and-real benchmark for comprehensive evaluation of generalist robot manipulation policies." arXiv preprint arXiv:2607.04434 (2026).

Questions for authors
  1. What is the weight on the future-target term in Future SFT? Did you similarly ablate that term to choose the best one as you did for your method?
  2. Can you list more detailed breakdown results in the appendix instead of only the average success results?
  3. Also it might be more straightforward to the readers if you can provide metrics beyond success rate to demonstrate the robustness to delay. In simulation, you can potentially compute the prediction error to oracle to show the robustness or visualize the trajectories to show how the proposed method works better than the other baselines. More qualitative analysis or failure mode analysis of baselines can also help readers better understand the proposed method.
1 comment

We thank the reviewer for the detailed feedback and for recognizing the importance of inference delay, the clarity of our central idea, and our evaluation across multiple VLA backbones and real-world settings. Changes in the revised manuscript are highlighted in blue. We hope our responses and revisions address your concerns

Comment/Question:

The motivation for preference learning is unclear ... The contribution of the DPO term appears small ... why preference optimization would perform better than regression.

Answer: Preference learning provides greater flexibility by teaching the policy which actions are better, rather than forcing it to imitate a single future action as the only correct answer. Since future actions cannot be uniquely determined from a stale observation, directly regressing to one future-generated target can impose an overly restrictive constraint. DEFLECT instead learns to prefer future-aligned actions over stale alternatives while allowing multiple valid behaviors. The original SFT loss preserves basic action skills, and the preference update weakens once a sufficient margin is reached (Section 3.2 and Appendix D, Eq. 11). This advantage is reflected in the results. On GR00T at d=4, DEFLECT achieves 73.10% success versus 65.75% for Future SFT (Table 1a), with smoother action transitions (Appendix M). Appendix I (Fig. 10) further shows that the DPO term contributes substantially despite its small coefficient, with weighted gradient-norm ratios of 0.33 on Kinetix and 0.40 on GR00T.

Comment/Question:

Future SFT is evaluated only for GR00T on LIBERO ... comparisons across additional backbones and potentially real-world tasks …

Answer: We have extended the evaluation to SmolVLA, Kinetix, and three real-robot tasks. On SmolVLA, DEFLECT achieves 60.26% average success, compared with 55.48% for Future SFT. On Kinetix, we swept four values of λ₊ for Future SFT. Averaged over d = 1–4, DEFLECT achieves 88.6%, compared with 87.53% for the best Future SFT configuration (λ₊ = 0.1). The real-robot results further support this advantage. On Conveyor-I and Conveyor-II, DEFLECT achieves 96.7% and 90.0% full-task success, compared with 93.3% and 86.7% for Future SFT. On Whack-a-Mole, DEFLECT strikes 13.6 targets per 30-second trial, compared with 11.6 for Future SFT.

Comment/Question:

The evaluation provides limited coverage of dynamic manipulation scenarios ... for example, RoboDojo …

Answer: We have added an evaluation on RoboDojo's dynamic conveyor manipulation task (Section 4.1.3, Fig. 6, and Appendix O). Compared with the Naive Async baseline using the original π0.5 checkpoints, DEFLECT improves success across all tested delays, with gains of 4.0 percentage points at d = 6 (15.33% vs. 11.33%) and 7.3 points at d = 8 (14.67% vs. 7.33%), despite constructing preference pairs using only 1–4-step delays. These results further support DEFLECT's robustness to observation delays in dynamic VLA manipulation.

Comment/Question:

What is the weight on the future-target term in Future SFT? Did you similarly ablate that term ...?

Answer: Future SFT uses λ₊ = 0.1, and we have added a weight sweep for both methods (Appendix H, Fig. 9 and Tables 7–8). We evaluate weights of 0.01, 0.1, 0.25, and 1. The best Future SFT configuration achieves 82.91% average success, compared with 85.76% for DEFLECT at its default weight. DEFLECT also outperforms Future SFT at every tested weight.

Comment/Question:

Can you list more detailed breakdown results in the appendix instead of only the average success results?

Answer: We have added per-suite and per-delay results in Appendix N (Table 13) for GR00T N1.7, π0.5, and SmolVLA, providing the detailed breakdown behind the averages in Table 1.

Comment/Question:

... provide metrics beyond success rate ... prediction error to oracle ... visualize the trajectories ... failure mode analysis …

Answer: We have added quantitative behavioral metrics and representative trajectories beyond success rates (Appendix M, Figs. 12–15). First, we evaluate action prediction errors against future expert actions. At d = 4, DEFLECT achieves the lowest scaled RMSE of 0.471, compared with 0.528 for Naive Async and 0.495 for Future SFT, indicating better alignment with the future expert reference. We also analyze action discontinuities, completion steps, and representative failure cases under matched conditions (Figs. 12–14). Specifically, we measure the replanning action jump ratio, which quantifies action changes at chunk boundaries relative to those within a chunk. Compared with Naive Async, DEFLECT reduces this ratio by 0.9–1.5, versus 0.3–0.4 for Future SFT, indicating smoother transitions between action chunks. The trajectory visualizations further illustrate differences in execution behavior and failure patterns.

Reviewer_sutiOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DEFLECT, an offline post-training framework for addressing the mismatch between prediction-time observations and execution-time states in asynchronous vision-language-action policies. A frozen reference policy generates paired action chunks from earlier and later observations in expert trajectories, with the later-observation chunks labeled as preferred. The policy is then optimized under its original deployment input using a flow-matching DPO surrogate combined with the original supervised objective. The method preserves the deployment interface and inference budget. Experiments on Kinetix, LIBERO, and three real-robot tasks report improved performance across multiple policy backbones. Comparisons with Continued SFT and Future SFT, together with preference-label ablations, assess the contribution of temporal preference supervision.

Critical strengths
  1. A practical advantage of DEFLECT is that it introduces no additional inference-time computation. The method preserves the base policy’s deployment inputs, asynchronous controller, and inference budget, making it a practical post-training extension to existing asynchronous VLA systems.
  2. The use of future observations from recorded demonstrations to construct temporal preferences is simple and insightful. It converts information available only during offline training into useful supervision, without requiring additional human preference annotations or online robot interaction.
Critical weaknesses
  1. Modest gains over the strong delay-aware baseline. On π0.5, DEFLECT improves VLASH from 95.54% to 96.28%, with gains of 0.74 percentage points on average and 1.40 at the largest delay. Without a matched continued-SFT control, the contribution of preference learning beyond additional supervised training remains unclear.
  2. Missing matched delay-aware SFT baselines. SmolVLA and GR00T lack matched comparisons with delay-aware expert supervision. GR00T’s Future SFT uses reference-generated targets, leaving the advantage over directly learning future expert actions unclear.
Questions for authors
  1. Could independently sampling the contrast horizon and context delay create conflicting supervision within the same training example?
  2. See weakness
2 comments

We thank the reviewer for the thoughtful feedback and for recognizing the practical value of DEFLECT, particularly its unchanged inference cost and simple offline preference construction. Changes in the revised manuscript are highlighted in blue. We hope the explanations and revisions provided below address your questions and concerns.

Comment/Question:

On π0.5, DEFLECT improves VLASH from 95.54% to 96.28% ... Without a matched continued-SFT control, the contribution of preference learning beyond additional supervised training remains unclear.

Answer: We have added a matched Continued SFT baseline, which shows that the improvement is not simply due to additional training. Under the same training recipe, Continued SFT achieves 94.70%, compared with 95.54% for the original VLASH policy and 96.28% for DEFLECT. This comparison is now reported in Section 4.1.2.

Comment/Question:

SmolVLA and GR00T lack matched comparisons with delay-aware expert supervision. GR00T's Future SFT uses reference-generated targets, leaving the advantage over directly learning future expert actions unclear.

Answer: We additionally evaluated VLASH-style delay-aware SFT baselines for both SmolVLA and GR00T N1.7, following the original VLASH method's use of future expert actions as supervised targets. In contrast, our Future SFT baseline uses reference-generated targets to compare direct regression with preference optimization. Since VLASH did not release its SmolVLA checkpoint or provide a GR00T N1.7 adaptation, we implemented VLASH-style training for both backbones and evaluated DEFLECT under matched settings. On SmolVLA, DEFLECT improves average success from 70.98% to 71.91%. On GR00T N1.7, it improves success from 90.60% to 91.49%, averaged over d=1–4. Although the reproduced SmolVLA baseline differs in absolute performance from the published result, the comparison consistently shows additional gains from DEFLECT under the same implementation and evaluation protocol.

Comment/Question:

Could independently sampling the contrast horizon and context delay create conflicting supervision within the same training example?

Answer: There is no conflict in supervision because the context delay determines the SFT target, while δ only defines a relative preference comparison. As clarified in Section 3.2 and Appendix D (Eq. 11), DEFLECT does not introduce A⁺ as another absolute regression target. Instead, the preference loss optimizes the relative contrast between A⁺ and A⁻, with an update that weakens as the preference margin grows. The original SFT objective remains unchanged, allowing temporal preference learning to complement expert supervision without requiring exact imitation of a potentially mismatched future action.

Thank you for the additional experiments and clarifications. The matched Continued SFT and delay-aware SFT baselines largely address my main concerns, although the gains remain modest. The explanation of relative preference optimization is helpful, but does not fully rule out potential conflicts with expert supervision under independent temporal sampling. Overall, the new results strengthen the empirical support, and I will maintain my weak-accept rating.

Reviewer_91N1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes DEFLECT, which constructs temporal preferences between action chunks generated by a frozen policy from earlier and later observations of the same expert trajectory. It combines a flow-matching preference objective with the original SFT objective and improves asynchronous execution. Experiments show performance improvements on Kinetix, LIBERO, and three real-robot tasks.

Critical strengths
  1. The preference pair construction is easy to implement and compute during training. The preference pairs turn future observations into supervision without additional environment interaction for preference labeling or access to future observations at deployment.
  2. The paper includes a variety of baselines in its experiments (BID, RTC, VLASH, REMAC, etc). These results support the value of temporal preferences beyond additional SFT and supervision from future-generated actions.
Critical weaknesses
  1. DEFLECT uses actions generated from later observations in demonstrations as preferred candidates. When training and testing follow similar task sequences, the policy may execute familiar next steps to compensate for delay. However, grasp errors, object movement, or goal changes can lead to states not covered by the demonstrations, where those next steps may no longer be appropriate. The paper does not directly evaluate whether preference training still improves delay robustness when execution departs from these familiar trajectories.
  2. Eq. (2) assigns preferences by timestamp, but A+A^+ may require a task step that the robot has not yet completed. Since δ\delta is sampled independently of the context delay, this mismatch can occur even in familiar tasks. For example, A+A^+ may begin lifting before the deployed robot has grasped the object, potentially encouraging premature motions.
  3. The success rates and ablations demonstrate the effectiveness of DEFLECT, but provide limited insight into how the learned behavior changes. In particular, Future SFT uses the same future-generated candidates, yet it remains unclear what behavioral differences explain DEFLECT’s advantage. Comparing action trajectories and execution-time target positions across the base policy, Future SFT, and DEFLECT under matched conditions, together with representative success and failure cases, would help clarify which errors the preference objective actually corrects.
Questions for authors

Could you compare A+A^+ and A−A^- from actual rollout states, especially after failed grasps or object displacement? This would clarify how often the temporal preference remains valid when the robot has not reached the expert’s assumed state.

1 comment

We thank the reviewer for the thoughtful feedback and for recognizing the simplicity of our preference-pair construction and the breadth of our baseline comparisons. Changes in the revised manuscript are highlighted in blue. We hope our responses and the revisions will satisfactorily address all concerns.

Comment/Question:

DEFLECT uses actions generated from later observations in demonstrations as preferred candidates ... The paper does not directly evaluate whether preference training still improves delay robustness when execution departs from these familiar trajectories.

Answer: Our new experiments show that temporal preferences remain useful even when execution deviates from expert demonstrations. Following the reviewer's suggestion, we added paired-rollout experiments on LIBERO-Goal and LIBERO-Object, including failed grasps and object displacements (Appendix L, Fig. 11). Starting from identical rollout states, we compare planning with fresh and stale observations after these deviations. Fresh-observation planning achieves 6–15 percentage points higher success rates under the tested perturbations, with positive gains also observed after naturally occurring failed grasps. These results support the temporal preference assumption beyond familiar expert trajectories.

Comment/Question:

Eq. (2) assigns preferences by timestamp, but A⁺ may require a task step that the robot has not yet completed. Since δ is sampled independently of the context delay ... potentially encouraging premature motions.

Answer: Independent sampling does not require the policy to execute future-generated actions prematurely. The original SFT objective continues to supervise the expert action corresponding to the sampled deployment context, rather than replacing it with A⁺. The contrast horizon δ determines which temporal candidates are compared, while the context delay determines the expert target retained by the SFT objective. As clarified in Section 3.2 and Appendix D (Eq. 11), DEFLECT uses A⁺ and A⁻ only for relative preference optimization. Its update acts on the contrast between the candidates and weakens as the preference margin grows, rather than forcing the policy to reproduce A⁺. Together with the original SFT anchor, this provides temporal correction while retaining supervision for the appropriate task stage.

Comment/Question:

The success rates and ablations demonstrate the effectiveness of DEFLECT, but provide limited insight into how the learned behavior changes ... Comparing action trajectories and execution-time target positions ... together with representative success and failure cases ...

Answer: We added matched behavioral comparisons across all four LIBERO suites (Appendix M, Figs. 12–15), showing smoother replanning, lower prediction errors, and fewer early failures with DEFLECT. Compared with Future SFT, DEFLECT reduces the action jump ratio by 0.9–1.5 versus 0.3–0.4, and achieves a lower prediction error at d = 4 (0.471 vs. 0.495). On LIBERO-10, early failures decrease from 21 to 5. Representative success and failure trajectories are also provided. We further clarify the mechanism in Section 3.2 and Appendix D (Eq. 11). Unlike Future SFT, which directly regresses toward A⁺, DEFLECT optimizes the relative preference between A⁺ and A⁻, with a margin-adaptive gradient that does not require exact reconstruction of A⁺. This distinction is consistent with the observed behavioral improvements.

Comment/Question:

Could you compare A⁺ and A⁻ from actual rollout states, especially after failed grasps or object displacement? ...

Answer: We added paired rollout comparison in Appendix L (Fig. 11). As discussed in our first answer, the figure also reports paired win/loss counts for each condition, showing how often fresh-observation planning succeeds when stale-observation planning fails, and vice versa.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.