acceptodds
Withdrawn from ICLR 2027

Forward Dynamics Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

17 views

Abstract

Vision-language-action (VLA) models and world action models (WAMs) have advanced robotic policy learning by building on pretrained vision-language models (VLMs) and video generative models (VGMs), respectively. Yet VLAs learn without explicit supervision from future world states, while WAMs that condition actions on predicted futures incur extra inference cost and let visual prediction errors propagate into actions. We present Forward Dynamics Models (FDM), a family of robotic action models that jointly predicts actions and future world states through an action-to-future paradigm. Future prediction is conditioned on action representations, while action prediction remains independent of predicted futures. End-to-end training lets ground-truth future states supervise the policy through these action representations. At inference, FDM skips future generation and runs as efficiently as a standard VLA. The paradigm is backbone-agnostic, and we instantiate it with both VLM and VGM backbones. Experiments in simulation and on real robots demonstrate the effectiveness of FDM across both backbone families, achieving 99.4% average success on LIBERO and gains of up to 13 percentage points over VLA and WAM methods on real-world tasks.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Manipulation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_cLJcOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Vision-language-action (VLA) models learn without supervision from future states, while world action models (WAMs) that condition actions on predicted futures add inference cost and let prediction errors propagate into actions. This paper introduces Forward Dynamics Models (FDM), which jointly predict actions and future observations under an asymmetric attention mask: future prediction reads the action expert's hidden states HaH_a, action prediction never reads future tokens, and the future loss backpropagates into HaH_a, so the deployed policy runs action-only. FDM is instantiated on π0.5\pi_{0.5}, with an added generation expert that regresses one future-image latent, and on Wan2.2-5B, with an added action expert trained jointly with future-video flow matching. The paper evaluates both on LIBERO, RoboTwin 2.0 and five real-robot tasks, and reports an action-to-future ablation, a future-perturbation analysis, gradient alignment and efficiency measurements.

Critical strengths
  • The ablation design is clean. Four variants (action-only, context-only future, stop-gradient, full FDM) separate future supervision, forward conditioning on HaH_a, and backward supervision through HaH_a. The stop-gradient control keeps the shared-context gradient paths and detaches only the action interface (App. C.3), so D vs. C isolates the gradient path. The ordering A << B << C << D holds in both backbone families (Table 4).

  • Action-only deployment with real-robot gains. FDM improves over π0.5\pi_{0.5} by 13 points and over an architecture-matched WAM by 5 points on five real-robot tasks (Fig. 3). Its inference latency matches π0.5\pi_{0.5} (294 vs. 298 ms/chunk), and FDM-VGM is 1.7×1.7\times faster than the matched WAM (Table 5).

Critical weaknesses
  • Training a policy with an action-conditioned future-prediction objective and dropping that objective at inference is an established design. DriveVLA-W0 (ICLR 2026) trains an action-conditioned world model jointly with the policy and bypasses image generation at inference. UWM (RSS 2025) obtains its policy by setting the future-observation timestep to its maximum. VideoVLA (NeurIPS 2025) already shows that letting video tokens attend to action tokens improves success. What remains new is the specific asymmetric mask over action hidden states (Table 6).
  • The closest baseline is missing and reports a higher RoboTwin score. Fast-WAM corresponds to FDM's context-only variant B. With the same Wan2.2-5B backbone, no embodied pretraining and the same RoboTwin protocol, it reports 91.8% (91.88 clean / 91.78 randomized), above FDM-VGM's 89.23% (Table 3). The paper positions FDM against Fast-WAM (Sec. 2) but omits it from Table 3. As a result, the claim that action-to-future supervision beats context-only future supervision is supported only by the real-robot ablation (Table 4).
  • LIBERO is saturated and the RoboTwin gains are small. Every method in Table 2 scores between 94.1% and 99.4% on LIBERO. On RoboTwin, FDM-VLM improves over π0.5\pi_{0.5} by only 1.45 points and is lower in the clean setting (81.54 vs. 82.74). FDM-VGM is 2.97 points below LingBot-VA (Table 3).
  • The "action-grounded supervision" interpretation is not established. The generation expert reads hidden states computed from the noised demonstrated action chunk (Eq. 2; App. A.2), and these hidden states also carry context features. Table 4 shows that supervising this interface helps, but not that the benefit comes from modelling action consequences. An action-swap test or a numerical-action bottleneck would be needed to show this.
  • The robustness advantage holds by construction. FDM's zero sensitivity in Fig. 4 follows directly from the attention mask. The WAM's action error rises by only 0.22–5.52% at α=1.0\alpha = 1.0, and no closed-loop evidence ties WAM failures to prediction errors.
  • Minor:
    • Table 2 and Sec. 4.1 state "without robot policy pretraining", yet FDM-VLM is initialized from pi05_base (App. B.1).
    • Step time ×\times throughput in Table 5 implies a batch of 128 for every row, while App. C states 64; please state the timing configuration.
Questions for authors
  • Can you add Fast-WAM to Table 3 and explain why FDM-VGM scores below it with the same backbone and protocol? Does your variant B, trained with your recipe, reach Fast-WAM's reported 91.8%?
  • With the observation and instruction fixed, does the predicted future change appropriately when only the action chunk is swapped?
  • Which weights are loaded from pi05_base, and do the real-world comparisons use the same initialization?
Reviewer_tbGjOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

Summary

VLA models lack a world-modeling objective, World Action Models (WAMs) suffer from high inference costs and error propagation from visual predictions into actions. To address these limitations, Forward Dynamics Models (FDMs) is proposed to jointly predict actions and future world states without letting prediction errors compromise action execution.

Critical strengths

Strengths

  1. Method to drop the future-prediction head/module, hence non compromising on inference latency and reactivity.
  2. Able to achieve very good scores LIBERO and ROBO-TWIN benchmarks.
  3. Well written text and comprehensive effort to showcase FDM working in both Vision-Language-Model and Video-Generative-Model paradigm.
Critical weaknesses

Weakness

  1. Tab.4 all the rows without backprop mean only the generation-expert parameters are trainable with L-world loss. The VLM/VGM + action expert parameters would not see the gradients from the L-world loss, and therefore should have no effect on the actions. Unless this is not the case, the success rate should be the same (83 vs 85 vs 87).
    The ablation 'Future perturbations of WAM vs FDM' is expected and also highlights the drawbacks of the other ablations in Tab4.
  2. ‘Context-only’ ablation is missing details. “Context-only future adds the future objective but prevents generation-branch tokens from reading action representations ". Does that mean that only the generation-expert parameters are trainable with the L_world loss ? If yes, then actions shouldn’t be effected with the L_world ?
  3. Lack of large scale dataset training and deployment is still not verified, e.g. DROID, BRIDGE, ABC datasets. On the real robot, only fine-tuning paradigm is explored. The FDM Numbers on LIBERO are not very indicative given how saturated the benchmark is, ( still impressive to be able to solve the benchmark).
Questions for authors

Questions:

  1. In motivation it is very similar to FAST-WAM paper and missing comparison both in method and experiments. Clarifications here would be useful to differentiate.
  2. Tab4. Action-only is the baseline method, and aligns with pi05 number. What does context-only future branch entail ? 'Context-only future adds the future objective but prevents generation-branch tokens from reading action representations' means that the generation-expert branch of the model is completely detached from the action-branch of the model. Need clarification why this experiment is needed ? and why are the eval numbers different from baseline ? Can the difference between action-only and context-only success rate be attributed to noise ?
  3. Tab.4 the rows without backprop, indicate only the generation-expert parameters are trainable with L-world loss ? And has no effect on the actions-branch? I don't understand the reason of carrying this ablation, when the generation expert is eventually discarded during inference.
  4. For real world tasks, the results are based only on finetuning, with the VLM-FDM initialized from pi05_base. Very little information about the if and how the baselines were fine-tuned and how efforts were made to make sure that the fine-tuning is not biased. Analysis about the failure mode of the baseline would be useful. I expect to be able to finetune the baseline to higher success rate ? Perhaps FDM enables better efficiency during finetuning/adaption ?

Small Clarifications: Eqn. 3 is a trivial back-propagation equation, doesnt add value i think. L_future and L_world in the method figure v/s in the text.

Reviewer_s1wtOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Forward Dynamics Models (FDM) for robot manipulation.

Existing WAMs predict future states first and then get actions from them, while FDM bases future state prediction on internal action features, while the action expert itself gets no input from future tokens. Real future states guide action features through end-to-end backpropagation during training. At test time, the model runs action-only inference and drops the future prediction branch, avoiding future video generation cost and prediction error build-up. The authors build FDM on both pi0.5 and Wan2.2, and test on LIBERO, RoboTwin 2.0, and five real-world dual-arm tasks.

Critical strengths
  1. The proposed action-to-future design effectively protects policy execution from visual prediction errors. Because the action generation path never reads predicted future tokens, the noise experiments in Figure 4 show that adding noise into future states causes zero change in action prediction error.

  2. Action-only inference keeps low latency via skipping future generation at test time. FDM-VLM matches standard VLA latency, i.e., 294.1 ms for FDM-VLM and 298.0 ms for pi0.5, while FDM-VGM is 1.7x faster than the WAM baseline, i.e., 420.3 ms v.s. 715.4 ms.

  3. The framework shows strong performance across different backbones without robot policy pretraining. It reaches 99.4% average success on LIBERO and beats other baselines across five real-world manipulation tasks.

  4. The paper is well-written and easy to follow.

Critical weaknesses
  1. The choices of future prediction horizons are empirical. FDM-VLM fixes future targets to a 15-frame offset and FDM-VGM uses a stride of 4, but the paper lacks sensitivity analysis on whether these choices generalize across tasks with different dynamic speeds.

  2. The training overhead increases noticeably for the VLM instantiation. Adding the generation expert increases the per-step training time from 0.444s to 0.631s, and adds around 5GiB/GPU peak memory, which could become a bottleneck for large-scale training.

  3. Dropping future generation at deployment removes test-time visual verification. Conventional world action models can use future video rollouts for trajectory selection or recovery, but FDM cannot leverage future visual predictions during execution.

Questions for authors

Please kindly see the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.