acceptodds
Desk rejected at ICLR 2027

Vid2WAM: Distilling Video Diffusion Priors into World Action Models

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

World Action Models (WAMs) improve robot policy learning by jointly modeling future visual dynamics and actions. However, their future prediction targets typically come from recorded robot trajectories, limiting supervision when target-task demonstrations are scarce. Therefore, we ask whether future supervision for WAMs must originate from target-task expert trajectories. In this paper, we propose Vid2WAM, an offline distillation framework that transfers visual diffusion priors from a large video foundation model into a compact WAM student. Given an observation and language instruction, Vid2WAM distills supervision through two complementary channels: task-conditioned future rollouts directly supervise the student's future prediction branch, while an inverse dynamics model recovers embodiment-specific pseudo-actions for action learning. To integrate synthetic and real supervision, we introduce source-aware residual action adaptation that learns source-specific corrections around a shared action backbone and mitigates interference from noisy pseudo-actions. During inference, both the video teacher and inverse dynamics model are discarded, leaving only the WAM student for efficient deployment. Simulation and real-world experiments show gains under limited expert demonstrations and in target-conditioned adaptation to novel tasks without target-task action trajectories, while preserving low-latency inference.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Action Models

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_TBunOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Vid2WAM, a framework for distilling future prediction priors from a large video diffusion model into a World Action Model. Given an initial observation and language instruction, an adapted video teacher generates task-conditioned future rollouts, which are used to directly supervise the student’s future-prediction branch. In parallel, an inverse dynamics model predicts pseudo-actions from the generated videos to provide additional action supervision. To combine real demonstrations and noisy pseudo-data, the authors introduce source-aware residual action adapters around a shared action backbone. The proposed method is efficient, as only a student WAM is used for action generation.

Critical strengths
  1. The paper is easy to follow and well-motivated. Moreover, the approach is inference-efficient while learning WAM better.
  2. Extensive experiments and ablations demonstrate the effectiveness of the proposed method.
Critical weaknesses
  1. The performance gain from the proposed method is somewhat marginal, e.g., uncertain whether 3% gains on RoboTwin 2.0 is statistically meaningful or not. In particular, the contribution of the source-aware residual adapters seems modest, making it unclear how essential the proposed adapter design is relative to simpler ways of separating real and pseudo supervision.
  2. The conceptual novelty is somewhat limited. Prior work such as DreamGen already use video-model generated rollouts together with inverse dynamics to construct synthetic robot supervision, while prior WAMs, e.g., FastWAM, leverage future visual prediction as an auxiliary signal for policy training.
Questions for authors

Please answer the weaknesses part.

Reviewer_zNkMOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Vid2WAM transfers an embodiment-adapted video teacher into a WAM student through two offline supervision channels: generated future-video latents and IDM-derived pseudo-actions. Source-specific residual action adapters provide separate corrections for real and synthetic samples, while both backbones remain shared. The teacher, IDM, auxiliary future head, and pseudo-source adapters are removed at deployment. Experiments cover low-demonstration learning and target-conditioned adaptation without target-task action trajectories on RoboTwin, LIBERO, LIBERO-Plus, and nine physical tasks. The low-data LIBERO ablation supports a benefit from generated-future supervision beyond pseudo-actions alone. RoboTwin held-out-task success improves over Fast-WAM from 45.0%/42.8% to 54.7%/55.3% in clean/randomized settings. LIBERO held-out success is much less consistent across task splits than the full-benchmark averages suggest. The contribution is offline dual-channel transfer with source-aware training.

Critical strengths
  1. Controlled evidence for complementary supervision. Pseudo Action Only retains real-video co-training and the same pseudo-action supervision, removing only the pseudo-video loss; success drops from 89.7% to 86.4%. Future Latent Only reaches 88.2% (Table 5; Appendix A.4). These matched-buffer results support a contribution from generated futures beyond additional pseudo-actions. The residual-adapter design also reaches 89.7%, compared with 88.5% without adapters and 89.1% with separate action DiTs, while adding approximately 1M training parameters rather than approximately 1B for the second action DiT.

  2. Useful offline transfer with little extra deployment cost. On the 15 held-out RoboTwin tasks, Vid2WAM improves over Fast-WAM by 9.7 and 12.5 percentage points under clean and randomized evaluation. The three physical held-out tasks reach 30%, 15%, and 30% without their action-labeled demonstrations, although absolute reliability remains limited. Student inference takes 212.5 ms per chunk, close to Fast-WAM's 209.1 ms and much lower than the online teacher pipeline's 4894.5 ms (Tables 1, 4, and 6). These results support transferring expensive offline supervision into a directly deployable student.

Critical weaknesses
  1. The contribution of the proposed training design to novel-task gains remains unclear. Comparing methods under the same real-demonstration budget is reasonable. However, Vid2WAM additionally learns from target-task synthetic data, while the baselines rely on direct transfer. Table 5 helps separate the effects of the training components on low-data LIBERO, but does not establish whether these benefits extend to novel-task adaptation.

  2. IDM reliability on generated videos remains uncertain. Table S4 measures prediction accuracy on recorded trajectories using an absolute-error tolerance; it does not establish label quality on teacher-generated videos. Figure S2 usefully shows a successful replay and two failure cases, and the eight-second cutoff excludes the later visual anomalies. However, three qualitative cases do not characterize how often the pseudo-actions in the retained training data reproduce the intended motion or task progress. Because these labels directly supervise the student, uncertainty about their reliability limits our understanding of a central training signal.

  3. The useful behavioral coverage added by synthetic data is unclear. Trajectory counts and appearance variation do not by themselves establish diverse, nonredundant behaviors or coverage of relevant task stages. Simulation rollouts are truncated to eight seconds to reduce generation errors, but the paper does not establish how much later-stage behavior remains represented. The low-data experiments add one generated trajectory per real trajectory without characterizing how useful coverage or performance changes with a larger synthetic buffer. The results support augmentation at the tested ratio; their implications for scaling varied, reliable experience, especially for longer tasks, are less clear.

  4. Limited methodological novelty. The video-generation and pseudo-action-labeling pipeline has precedents such as DreamGen. The main additions are direct generated-future supervision for WAMs and source-specific residual adapters; Table 5 provides useful evidence for these additions, but the broader pipeline is established.

Questions for authors
  1. Beyond the low-data LIBERO ablations, what evidence separates the proposed training design from access to additional synthetic supervision? In particular, would the novel-task advantage remain against a baseline such as Fast-WAM trained on the same target-task pseudo data?

  2. How reliable are the IDM-derived actions on the generated videos actually retained for training? What evidence beyond Figure S2 characterizes this reliability, and is any quality screening applied beyond the duration cutoff?

  3. What evidence shows that generated trajectories add diverse behaviors and cover relevant task stages? How does the eight-second simulation cutoff affect later-stage coverage, and how does the benefit depend on synthetic-data quantity at a fixed real-demonstration budget?

Reviewer_gb2aOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a method to distill knowledge from a video foundation model to a world action model for efficient transfer to novel tasks. The idea is to use a video diffusion model (finetuned on embodiment data) and an inverse dynamics model as teachers to generate pseudo training data for a WAM student. Specifically, distillation happens via two streams: (1) the video diffusion model generates future frames from initial states, which are then encoded by the student WAM's VAE to use as supervision; (2) a separate inverse dynamics model generates pseudo actions from the synthetic videos, which are then fed through a residual adapter to be combined with real actions for distillation. The WAM is finetuned jointly on the synthetic frames and the pseudo actions. The main benefit is efficient transfer: for a new task, they only need to collect a few initial frames and query the teacher models to generate data for distillation (together with any real expert demonstrations). Experiments on RoboTwin, LIBERO, LIBERO-Plus, and a real-world bimanual platform show that a LVP teacher + a FastWAM student achieves higher success rates transferring to novel tasks in the low-data regime, compared to FastWAM without teacher distillation and other baselines. Ablation experiments validate the contribution of each design choice, e.g., dual action DiT design and each individual stream of supervision.

Critical strengths
  1. The proposed method enables quick adaptation to new tasks. Instead of collecting a large number of expert demonstrations, they only need to collect initial states and query the teacher for dense supervision. This makes it more practical to finetune WAMs to new tasks.
  2. The method is evaluated on a comprehensive set of benchmarks covering three sim benchmarks (RoboTwin, LIBERO, LIBERO-Plus) and a real-world bimanual platform. The method outperforms baselines in the low-data and novel regimes across these benchmarks (Figure 4, Table 1, 2, 3,4).
  3. The design choices are supported by ablation experiments (Table 5, 6)
Critical weaknesses
  1. The method can feel a bit incremental. They take the DreamGen recipe (video diffusion model + inverse dynamics model) but instead of directly rolling out the actions, distill it to a small WAM model. Compared to the original DreamGen recipe (with finetuning on target tasks), this seems mostly a training and inference speed gain, which could be addressed by optimization (e.g. quantization).
  2. The assumption used to motivate the paper, "future supervision must be observed in taskspecific expert demonstrations" (line 43), is too strong: WAM's future supervision doesn't necessarily require expert demonstrations. They can be cotrained on human videos (e.g. DreamZero, Motus).
  3. The experiments lack comparison to a few immediate baselines. See questions below.
  4. Simulation experiments seem to report one seed only.
  5. The overall presentation lacks clarify. Some details are swept under the rug.
Questions for authors
  1. The teacher video model is mid-trained on embodiment-specific expert data (line 319: "further finetune it on our training set"). Can you disclose the mid-training dataset for each domain? This information seems to be scatter around the paper.
  2. What happens if you finetune the student WAM on the embodiment-specific expert data (used to mid-train the teacher model), and then finetune on the task-specific expert data?
  3. In the task-specific finetuning stage, what if you spend equal amount of compute to finetune the teacher model + IDM on the handful of expert demonstrations? In other words, no student model, but finetune the DreamGen recipe. The video prior is directly retained through the weights. Yes, inference would be slower, but inference speed can be optimized by a lot of tricks.
  4. The comparison to Pi0.5 and Motus are uninformative since these baselines have no teacher distillation. What happens if you distill a VLA or another WAM, e.g. Motus? Does the same improvement hold? Table 5's action-stream-only ablation is not exactly equivalent to distilling a VLA becuase there are architectural differences.
  5. According to Table 5, the action adapter doesn't contribute much to the overall performance. Is it a sim-only artifact?
  6. Do the held out initial states (used to query the teacher model) overlap with evaluation initial states?
Reviewer_JcZtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Summary. The paper proposes a simple idea: train a diffusion-based world-action model p(v(1:T), a(1:T) | v(0), lang) using a mix of real (v,a) trajectories (where a(1:T) is from an expert), and syntehtic v trajectories coming from a large video generative model (WAN-2.1) and corresponding synthetic a trajectories coming from an IDM model (ResNet) that estimates a(1:T) = IDM(v(1:T)). This is a simple form of data augmentation. They show modest gains on manipulation benchmarks, both simulated and real.

Critical strengths

Sensible idea for synthetic data augmentation to tackle lack of (video,action) paired data for novel tasks.

Critical weaknesses

Idea is simple and similar to prior work, and empirical gains seem fairly small. No error bars are reported.

Questions for authors

none

Comments on the paper

1 comment
Area_Chair_iXLqOpenReview ↗

Discussion with the authors

Dear reviewers,

the authors have provided responses for your reviews. Reviewer zNkM, thank you for letting the authors know that your concerns have been addressed. Can the others respond as well, please?

Thank you, Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.