acceptodds
Withdrawn from ICLR 2027

Self-distilling Future Video Representations for World Action Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

12 views

Abstract

World Action Models (WAMs) have emerged as a promising approach to learning generalist robot policies using rich spatiotemporal priors from pre-trained video diffusion models. A recent line of WAMs improves inference efficiency by conditioning a separate action decoder on intermediate future video representations extracted from a single denoising step, avoiding costly iterative denoising process required for video generation. This design makes action prediction depend directly on the quality of these future representations. In this paper, we introduce Future REpresentation learning via self-Distillation (FRED), a framework that trains WAMs to improve the quality of future video representations from noise in a single denoising step. Specifically, FRED uses self-distillation to predict future representations from noise, using representations extracted from ground-truth future observations as targets. We further condition the video model on historical observations, providing temporal context for predicting future representations better. Across diverse real-world manipulation scenarios, FRED consistently outperforms competing WAMs. It achieves particularly large gains on long-horizon and out-of-domain tasks under zero-shot environment generalization, and substantially improves human-to-robot skill transfer, reaching 50.0% task success on tasks absent from the robot training data and demonstrated only in human videos, compared with 19.4% for the best competing WAM.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robot Learning, World Action Models, Video Generation Models, Representation Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_P9FMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

FRED trains a video-based world action model to predict useful future representations in one denoising pass. An EMA teacher extracts targets from noisy future observations, while the student predicts them from noise and observation history; a separate action decoder uses these features. Experiments cover real-world environment generalization, human-video skill transfer, and three simulation benchmarks, with ablations of self-distillation, history, and training choices.

Critical strengths
  1. The objective targets the features actually consumed by the action decoder. This is a clear motivation for future-representation learning beyond improving generated video quality.

  2. The real-world transfer results are promising. Human-video co-training raises the reported transfer score to 50.0%, versus 19.4% for DiT4DiT, and the 14B model shows large gains on the evaluated unseen-environment tasks.

  3. The analysis covers several relevant alternatives and design choices. Table 3 separates history and self-distillation in simulation, and Tables 4–7 examine representation objectives, another video backbone, and inference cost.

Critical weaknesses
  1. The real-world metric is ambiguous. Tables 1–2 use 'success rate', but Appendices B.1–B.2 allow partial credit even beyond long-horizon tasks. Without the scoring rubric and full-completion counts, the headline transfer and generalization numbers are difficult to interpret.

  2. The strongest claims concern real-world transfer, but the history/self-distillation ablation is only on RoboCasa. The real-world comparisons do not isolate how much of the gain comes from self-distillation rather than additional observation history or other training differences.

  3. Each physical task has only 12 trials, with no reported uncertainty or independent training-seed variation. The out-of-domain set contains just two tasks, and human transfer three, so the breadth and reliability of the generalization claim remain uncertain.

Questions for authors
  1. Which entries include partial credit, and what are the exact scoring rules? Please report binary full-task completion alongside progress and instruction-following scores.

  2. On the human-transfer or unseen-environment tasks, how does a history-matched model perform when only the self-distillation loss is removed?

  3. Can you provide uncertainty for the physical results and clarify whether the improvements persist across training runs and task configurations?

2 comments
AuthorsOpenReview ↗

Response to Reviewer P9FM (1/2)

Dear reviewer P9FM,
We sincerely appreciate your efforts in reviewing our manuscript. We respond to each comment in the following content.


[W1/Q1-1] Clarifying the scoring rules for reported performance

[A] We clarify that the reported scores for the DROID zero-shot environment generalization benchmark and OpenArm in-domain tasks allow partial credit, whereas the OpenArm human-to-robot skill transfer results use strict binary scoring based on full task completion as follows:

  • In-domain Easy, Out-of-domain, and Object Grounding tasks on DROID experiments and In-domain Atomic tasks on OpenArm experiments: We give partial credit of 0.5 out of 1.0 when the robot successfully grasps the correct target object but fails to complete the task, as it means that the model identifies the target object correctly.

  • In-domain Long-horizon tasks on DROID experiments and In-domain Composite tasks on OpenArm experiments: We give partial credit of 1/N for each completed atomic task (e.g., in a task requiring the robot to pick and place 5 objects consecutively, we give 1/5 for each completed PnP).

  • Human-to-robot Skill Transfer tasks on OpenArm experiments: We do not give partial credit for partial progress even though these tasks are atomic, to evaluate whether the robot can execute target behavior rather than merely identify or grasp the correct object.


[Q1-2] Reporting strict binary success rates

[A] Following your suggestion, we report the strict binary success rates for DROID zero-shot environment generalization benchmark and OpenArm in-domain tasks. As shown in the tables below, FRED still outperforms competing WAMs on DROID out-of-domain and object grounding tasks and OpenARM in-domain tasks, while achieving comparable or higher binary success rates on DROID in-domain tasks.

Table. Results on DROID zero-shot environment generalization (binary success rates) MethodBackboneIn-domain EasyIn-domain Long-horizonOut-of-domainObject Groundingπ0.5−55.6±5.416.7±10.88.3±5.616.7±7.4GR00T N1.7−41.7±5.616.7±10.84.2±4.020.8±8.3Fast-WAMWan2.2-5B25.0±6.60.0±0.00.0±0.08.3±5.6DiT4DiTWan2.2-5B16.7±5.90.0±0.00.0±0.08.3±5.6AGRAWan2.2-5B41.7±4.20.0±0.08.3±5.633.3±9.5FRED(Ours)Wan2.2-5B41.7±6.616.7±10.829.2±8.937.5±9.9DreamZeroWan2.1-14B63.9±7.725.0±12.58.3±5.416.7±7.4FRED(Ours)Wan2.1-14B77.8±6.725.0±12.541.7±9.954.2±10.1\begin{array}{l|c|cccc}\hline\text{Method}&\text{Backbone}&\text{In-domain Easy}&\text{In-domain Long-horizon}&\text{Out-of-domain}&\text{Object Grounding} \\\\ \hline \pi_{0.5}&-&55.6\pm5.4&16.7\pm10.8&8.3\pm5.6&16.7\pm7.4\\\\ \text{GR00T N1.7}&-&41.7\pm5.6&16.7\pm10.8&4.2\pm4.0&20.8\pm8.3\\\\ \hline \text{Fast-WAM}&\text{Wan2.2-5B}&25.0\pm6.6&0.0\pm0.0&0.0\pm0.0&8.3\pm5.6\\\\ \text{DiT4DiT}&\text{Wan2.2-5B}&16.7\pm5.9&0.0\pm0.0&0.0\pm0.0&8.3\pm5.6\\\\ \text{AGRA}&\text{Wan2.2-5B}&\mathbf{41.7\pm4.2}&0.0\pm0.0&8.3\pm5.6&33.3\pm9.5\\\\ \mathbf{FRED\\ (Ours)}&\text{Wan2.2-5B}&\mathbf{41.7\pm6.6}&\mathbf{16.7\pm10.8}&\mathbf{29.2\pm8.9}&\mathbf{37.5\pm9.9}\\\\ \hline \text{DreamZero}&\text{Wan2.1-14B}&63.9\pm7.7&\mathbf{25.0\pm12.5}&8.3\pm5.4&16.7\pm 7.4\\\\ \mathbf{FRED\\ (Ours)}&\text{Wan2.1-14B}&\mathbf{77.8\pm6.7}&\mathbf{25.0\pm12.5}&\mathbf{41.7\pm9.9}&\mathbf{54.2\pm 10.1}\\\\ \hline \end{array}

Table. Results on OpenArm in-domain atomic tasks (binary success rates)

\hline \end{array}$$ --- **[W2/Q2] Contribution of self-distillation and historical observations in real-world experiments** **[A]** Following your constructive suggestions, we additionally evaluate two variants of FRED (i.e., FRED without history conditioning and FRED without the self-distillation objective) in real-world environments. Specifically, we evaluate these variants on (a) zero-shot environment generalization by training the models using the DROID dataset and testing them on the in-domain long-horizon task (we train models for 100K iterations for the ablation); and (b) human-to-robot skill transfer by training the models using the compositional dataset of OpenArm robot demonstrations and human videos and evaluating them on 3 human-to-robot skill transfer tasks (PnP Bottle on Basket, PnP Novel Objects, and Pour Coffee). As shown in the table below, both components provide complementary gains in real-world experiments (both zero-shot environment generalization and human-to-robot skill transfer), as removing either component reduces performance, while removing both leads to further degradation. $$\begin{array}{l|cc}\hline\text{Method}&\text{DROID (Long-horizon)}&\text{OpenArm (Human2Robot)}\\\\ \hline\mathbf{FRED\\ (Ours)}&\mathbf{43.3}&\mathbf{50.0(SR)\\ /\\ 97.2\\ (IF)}\\\\ \text{w/o Self-distillation}&21.7&25.0(\text{SR})\\ /\\ 77.8\\ (\text{IF})\\\\ \text{w/o History}&28.3&38.9(\text{SR})\\ /\\ 83.3\\ (\text{IF})\\\\ \hline \text{Base WAM (DiT4DiT)}&8.3&19.4(\text{SR})\\ /\\ 55.6\\ (\text{IF}) \\\\ \hline \end{array}$$
AuthorsOpenReview ↗

Response to Reviewer P9FM (2/2)

[W3/Q3] Uncertainty about the breadth and reliability of generalizability

[A] We believe our real-world experiments provide meaningful evidence of the effectiveness of FRED in both zero-shot environment generalization and human-to-robot skill transfer, because FRED shows substantial performance gains (e.g., 10.4%$\rightarrow$52.1% on DROID out-of-domain tasks and 19.4%$\rightarrow$50.0% on human skill transfer tasks) across tasks where the scale is comparable to prior works. For example, we conduct 8 tasks x 12 trials on DROID and 7 tasks x 12 trials on OpenArm per model, compared with 40 tasks x 2 trials for DreamZero on DROID and 7 tasks x 20 trials for DiT4DiT on Unitree G1.

Nevertheless, we agree that reporting uncertainty scores and expanding additional tasks will strengthen our evaluation. To address these concerns, we conduct the following additional experiments.

  • Reporting uncertainty scores: For the original results in Tables 1 and 2, we compute the mean and its standard deviation calculated using the rliable library [1] for each evaluation category. As shown in the tables below, FRED consistently achieves substantial improvements over baselines, e.g., improving out-of-domain performance on DROID from 10.4$\pm$5.5% to 52.1$\pm$8.2% at the 14B scale, as well as human-to-robot transfer task success on OpenARM from 19.4$\pm$6.2% to 50.0$\pm$7.8%.

Table. Zero-shot environment generalization (DROID)

MethodBackboneIn-domain EasyIn-domain Long-horizonOut-of-domainObject groundingFast-WAMWan2.2-5B26.4±6.726.7±4.32.1±2.010.4±5.9DiT4DiTWan2.2-5B16.7±5.916.7±4.60.0±0.08.3±5.6AGRAWan2.2-5B41.7±4.231.7±6.010.4±5.933.3±9.5FRED(Ours)Wan2.2-5B47.2±6.246.7±9.537.5±8.937.5±9.9DreamZeroWan2.1-14B63.9±7.761.7±9.010.4±5.539.6±6.6FRED(Ours)Wan2.1-14B79.2±6.581.7±4.452.1±8.254.2±10.1\begin{array}{l|l|cccc} \hline \text{Method}&\text{Backbone}&\text{In-domain Easy}&\text{In-domain Long-horizon}&\text{Out-of-domain}&\text{Object grounding} \\\\ \hline \text{Fast-WAM}&\text{Wan2.2-5B}&26.4 \pm 6.7&26.7 \pm 4.3&2.1 \pm 2.0&10.4 \pm 5.9 \\\\ \text{DiT4DiT}&\text{Wan2.2-5B}&16.7 \pm 5.9&16.7 \pm 4.6&0.0 \pm 0.0&8.3 \pm 5.6 \\\\ \text{AGRA}&\text{Wan2.2-5B}&41.7 \pm 4.2&31.7 \pm 6.0&10.4 \pm 5.9&33.3 \pm 9.5 \\\\ \mathbf{FRED\\ (Ours)}&\text{Wan2.2-5B}&\mathbf{47.2 \pm 6.2}&\mathbf{46.7 \pm 9.5}&\mathbf{37.5 \pm 8.9}&\mathbf{37.5 \pm 9.9} \\\\ \hline \text{DreamZero}&\text{Wan2.1-14B}&63.9 \pm 7.7&61.7 \pm 9.0&10.4 \pm 5.5&39.6 \pm 6.6 \\\\ \mathbf{FRED\\ (Ours)}&\text{Wan2.1-14B}&\mathbf{79.2 \pm 6.5}&\mathbf{81.7 \pm 4.4}&\mathbf{52.1 \pm 8.2}&\mathbf{54.2 \pm 10.1} \\\\ \hline \end{array}

Table. Human-to-robot skill transfer (OpenArm; w/ human)

MethodIn-domain AtomicIn-domain CompositeHuman Task successHuman Instr. followingDreamZero68.1±5.10.0±0.00.0±0.013.9±5.3Fast-WAM65.3±6.50.0±0.00.0±0.011.1±5.1DiT4DiT86.1±5.541.7±5.419.4±6.255.6±4.9AGRA75.0±6.612.5±6.30.0±0.00.0±0.0FRED(Ours)97.2±2.775.0±9.350.0±7.897.2±2.7\begin{array}{l|cccc} \hline \text{Method}&\text{In-domain Atomic}&\text{In-domain Composite}&\text{Human Task success}&\text{Human Instr. following} \\\\ \hline \text{DreamZero}&68.1 \pm 5.1&0.0 \pm 0.0&0.0 \pm 0.0&13.9 \pm 5.3\\\\ \text{Fast-WAM}&65.3 \pm 6.5&0.0 \pm 0.0&0.0 \pm 0.0&11.1 \pm 5.1 \\\\ \text{DiT4DiT}&86.1 \pm 5.5&41.7 \pm 5.4&19.4 \pm 6.2&55.6 \pm 4.9 \\\\ \text{AGRA}&75.0 \pm 6.6&12.5 \pm 6.3&0.0 \pm 0.0&0.0 \pm 0.0 \\\\ \mathbf{FRED\\ (Ours)}&\mathbf{97.2 \pm 2.7}&\mathbf{75.0 \pm 9.3}&\mathbf{50.0 \pm 7.8}&\mathbf{97.2 \pm 2.7} \\\\ \hline \end{array}

  • Reporting additional tasks: We further evaluate our models on newly designed tasks for both zero-shot environment generalization and human-to-robot skill transfer. For zero-shot environment generalization, we introduce an additional out-of-domain task (Perch Hat on Peg). For human-to-robot skill transfer, we introduce a novel placement task (PnP Box to Center), while increasing the number of trials per object category for the novel object tasks. As shown in the tables below, FRED still consistently outperforms the other WAM baselines, supporting its effectiveness across diverse objects and manipulation behaviors.

Table. Results on additional out-of-domain task (DROID). We report task success (%) with partial credit.

MethodBackbonePerch Hat on PegFast-WAMWan2.2-5B0.0DiT4DiTWan2.2-5B0.0AGRAWan2.2-5B8.3FRED(Ours)Wan2.2-5B41.7DreamZeroWan2.1-14B41.7FRED(Ours)Wan2.1-14B66.7\begin{array}{l|l|c} \hline \text{Method}&\text{Backbone}&\text{Perch Hat on Peg} \\\\ \hline \text{Fast-WAM}&\text{Wan2.2-5B}&0.0 \\\\ \text{DiT4DiT}&\text{Wan2.2-5B}&0.0 \\\\ \text{AGRA}&\text{Wan2.2-5B}&8.3 \\\\ \mathbf{FRED\\ (Ours)}&\text{Wan2.2-5B}&\mathbf{41.7} \\\\ \hline \text{DreamZero}&\text{Wan2.1-14B}&41.7 \\\\ \mathbf{FRED\\ (Ours)}&\text{Wan2.1-14B}&\mathbf{66.7} \\\\ \hline \end{array}

Table. Results on additional human skill transfer tasks (OpenArm; w/ human). We report task success (%).

MethodPnP Box to CenterPnP Novel BottlePnP Novel BoxPnP Novel DollPnP Novel BallDreamZero0.08.30.00.00.0Fast-WAM0.00.00.00.00.0DiT4DiT0.00.025.025.00.0AGRA0.00.00.00.00.0FRED(Ours)33.366.750.075.041.7\begin{array}{l|ccccc}\hline\text{Method}&\text{PnP Box to Center}&\text{PnP Novel Bottle}&\text{PnP Novel Box}&\text{PnP Novel Doll}&\text{PnP Novel Ball} \\\\ \hline\text{DreamZero}&0.0&8.3&0.0&0.0&0.0 \\\\ \text{Fast-WAM}&0.0&0.0&0.0&0.0&0.0 \\\\ \text{DiT4DiT}&0.0&0.0&25.0&25.0&0.0 \\\\ \text{AGRA}&0.0&0.0&0.0&0.0&0.0 \\\\ \mathbf{FRED\\ (Ours)}&\mathbf{33.3}&\mathbf{66.7}&\mathbf{50.0}&\mathbf{75.0}&\mathbf{41.7} \\\\ \hline \end{array}

[1] https://github.com/google-research/rliable

Reviewer_GQz8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FRED, which uses self-distillation to improve single-step future video representations for action prediction in world-action models. The student model predicts representations extracted from real future observations by a momentum teacher, starting from pure noise. It also uses historical observations to provide temporal context. A separate action decoder then generates robot actions from these representations. The authors evaluate the method on real-world generalization, skill transfer with human video data, and multiple simulation benchmarks. Component ablations and representation probes are also used to analyze the method.

Critical strengths
  1. The method addresses a clear training need.
    FRED directly supervises the single-step video representations used by the action decoder. The motivation is reasonable, and the method preserves the inference efficiency of a single-step video backbone. The simulation ablations provide some evidence that self-distillation has an independent contribution.

  2. The evaluation includes meaningful robot settings.
    The real-world generalization experiments and training with unlabeled human videos provide initial evidence for the applicability of the method in different settings.

Critical weaknesses
  1. The contribution of the core design to the main experimental improvements is not sufficiently isolated.
    The paper combines self-distillation, historical observations, and action gradient blocking. The simulation ablations support the effectiveness of these components, but the main improvements in real-world generalization and human skill transfer lack direct comparisons that isolate self-distillation while keeping other conditions fixed. Therefore, it remains unclear how much of the paper's strongest empirical results comes from the proposed representation learning objective.

  2. How knowledge from human videos transfers to executable robot actions remains insufficiently explained.
    Human videos update the video representations, while action supervision for the action decoder still comes from robot demonstrations. Self-distillation aligns the student and teacher within each video, but does not explicitly establish a representation correspondence between humans and robots performing the same task. Appendix F also acknowledges that this correspondence is learned implicitly. A shared backbone may support transfer, but the paper does not verify how human data improves future representations under robot observations, or how the action decoder uses these improvements to perform behaviors not present in robot demonstrations. The current results cannot clearly distinguish the transfer of new action knowledge from improvements in object recognition, target-state understanding, or recombination of existing actions. The correlation between action primitive probes and control performance is also insufficient to explain this cross-embodiment transfer.

The paper provides little detailed analysis of the cross-embodiment mechanism. Many prior works have discussed the gap between human and robot data, but this paper does not sufficiently address it. Simply stating that the correspondence emerges implicitly is not enough. This is a main reason for my rejection recommendation. More careful experiments are needed to explain how human or egocentric data contributes to robot action learning and to isolate the source of the reported improvements.

Questions for authors

Key Questions That May Affect My Evaluation

  1. Does self-distillation explain the main improvements on real robots?
    Could the authors directly compare models with and without self-distillation on representative generalization and human skill transfer tasks, while keeping historical inputs, the backbone, action decoder, and gradient settings unchanged?

  2. Is implicit cross-embodiment correspondence sufficient to support new manipulation skills?
    Please clarify whether the robot demonstrations already cover the motion primitives needed for pouring, such as grasping and wrist rotation. More importantly, please provide evidence of how action knowledge from human videos improves representations and execution under robot observations. A comparison that preserves object and goal information but removes temporal information from human videos would be useful. Cross-embodiment representation analysis at corresponding task stages could also help distinguish action knowledge transfer from improvements due to semantic information.

I find the lack of detailed cross-embodiment analysis a major weakness and a key reason for my rejection recommendation. Many prior works have discussed the gap between human and robot data, but this paper does not sufficiently address it. Simply stating that the correspondence is learned implicitly does not explain how this gap is bridged. I would like to see more carefully designed experiments and stronger evidence showing how human or egocentric data contributes to robot action learning. More experimental results and a clearer attribution analysis are needed to address this concern.

  1. Do the improvements really depend on future information?
    Could the authors compare distillation using ground-truth future observations with distillation using current observations as targets? A probing baseline that only uses historical representations would also help determine whether future prediction is the main source of improvement. Please also discuss whether incorrect future predictions can lead to cascading errors in action generation.

Separate Reporting Checks: Not Used in My Current Score, but Require Clarification

  1. Inconsistency between the main table and appendix results.
    In Table 12(b), the task scores and instruction-following scores of DiT4DiT average approximately 33.3% and 58.3%, respectively, while Table 2(b) reports 19.4% and 55.6%. Since each task is evaluated 12 times, standard weighting and rounding cannot explain these differences. Do the two tables come from different evaluation runs? Please clarify their relationship and verify the improvement claims in the abstract using consistent statistics.

  2. Task success rates and partial completion scores should be distinguished.
    Appendices B.1 and B.2 explicitly allow partial credit. Please clarify whether the 50.0% reported in the abstract refers to the full task success rate or the average task score. Please report strict binary success rates, task progress scores, and the rules for assigning partial credit separately.

  3. The simulation evaluation protocol needs more details.
    Please clarify the exact task set used in Table 4, the number of evaluation episodes per task, the random initialization settings, and how the results are aggregated.

One additional presentation concern, which also does not affect my score. The figures and data labels on page 8 are relatively crowded, while pages 16, 18, and 21 of the appendix have considerable unused space. The configurations, data, and results for the same experiments are also scattered across different sections. I suggest adjusting the figure layouts and grouping related information together to make the paper easier to read and verify.

5 comments
AuthorsOpenReview ↗

Response to Reviewer GQz8 (1/4)

Dear reviewer GQz8,
We sincerely appreciate your efforts in reviewing our manuscript. We respond to each comment in the following content.


[W1/Q1] Contribution of self-distillation and historical observations in real-world experiments

[A] Following your constructive suggestions, we additionally evaluate two variants of FRED (i.e., FRED without history conditioning and FRED without the self-distillation objective) in real-world environments. Specifically, we evaluate these variants on (a) zero-shot environment generalization by training the models using the DROID dataset and testing them on the in-domain long-horizon task (we train models for 100K iterations for the ablation experiment); and (b) human-to-robot skill transfer by training the models using the compositional dataset of OpenArm robot demonstrations and human videos and evaluating them on 3 human-to-robot skill transfer tasks (PnP Bottle on Basket, PnP Novel Objects, and Pour Coffee).

As shown in the table below, both components provide complementary gains in real-world experiments (both zero-shot environment generalization and human-to-robot skill transfer), as removing either component reduces performance, while removing both leads to further degradation.

\hline \text{Method} & \text{DROID (Long-horizon)}&\text{OpenArm (Human2Robot)} \\\\ \hline \mathbf{FRED\\ (Ours)}&\mathbf{43.3}&\mathbf{50.0\\ (SR)\\ /\\ 97.2\\ (IF)} \\\\ \text{w/o Self-distillation}&21.7&25.0\\ (\text{SR})\\ /\\ 77.8\\ (\text{IF}) \\\\ \text{w/o History}&28.3&38.9\\ (\text{SR})\\ /\\ 83.3\\ (\text{IF}) \\\\ \hline \text{Base WAM (DiT4DiT)}&8.3&19.4\\ (\text{SR})\\ /\\ 55.6\\ (\text{IF}) \\\\ \hline \end{array}$$ --- **[W2/Q2-2] How human videos improve robot actions? Please provide evidence** **[A]** Thank you for an important question. We hypothesize that human videos improve robot action generation by helping the VDM learn future representations that capture how to perform tasks following language instructions, rather than merely recognizing target objects or goal states. Specifically, human videos provide task-specific motion information about how arms, hands, and objects move to complete tasks, which VDM representations can capture [1,2], while robot demonstrations already cover many of the low-level actions needed for execution. FRED further improves how well future representations generated from noise capture such motion information (see Figure 3(c)), allowing the robot-trained action decoder to leverage these representations to execute tasks demonstrated only in human videos. To further investigate this hypothesis, following your suggestion, we compare FRED co-trained on robot demonstrations and human videos with a variant that removes temporal changes from future frames of human videos. In particular, we replace all future frames in each human-video training batch with the current observation, while preserving the observation history and task instructions (i.e., FRED w/o Human Future Dynamics). As shown in the table below, this variant achieves 0% task success and 0% instruction-following rates on human-to-robot skill transfer tasks, with the robot remaining stationary for over one minute. These results suggest that temporal information in human videos plays an important role in human-to-robot skill transfer. $$\begin{array}{l|ccc} \hline \text{Method} & \text{Task success (\\%)} & \text{Instruction following (\\%)} & \text{Arm movement (deg/s)} \\\\ \hline \mathbf{FRED\\ (Ours)} & \mathbf{50.0} & \mathbf{97.2} & 3.54 \pm 1.25 \\\\ \text{w/o Human Future Dynamics} & 0.0 & 0.0 & 0.43 \pm 0.18 \\\\ \hline \end{array}$$ [1] Vélez et al., From Image to Video: An Empirical Study of Diffusion Representations, ICCV 2025 \ [2] Nam et al., Emergent Temporal Correspondences from Video Diffusion Transformers, NeurIPS 2025 --- **[Q2-1] Does the robot demonstrations already cover the motion primitives for human-to-robot transfer tasks, e.g., grasping and wrist rotation or pouring task?** **[A]** The robot demonstrations cover low-level motion primitives required for pouring, such as grasping, lifting, and wrist rotation. For example, some robot demonstrations of the PnP Banana task include slight wrist rotation while grasping or placing the banana. However, we note that the complete pouring motions, i.e., tilting the bottle and then stopping motions while maintaining the tilt, are absent from our robot training data for the OpenArm human-to-robot transfer tasks.
AuthorsOpenReview ↗

Response to Reviewer GQz8 (2/4)

[Q3-1] Does action generation quality improvement really depend on future information?

[A] Since the action decoder directly conditions on intermediate VDM hidden states obtained from noise, policy performance should depend on how well these hidden states capture future information, particularly robot motions. Consistently, we observe a strong correlation between the quality of generated future representations and policy performance in Figure 3(c).

Nevertheless, to further address your concerns, we conduct two additional ablations to examine the effect of future information on action generation as follows:

  • Following your suggestion, we replace the self-distillation targets extracted from noisy ground-truth future observations with those extracted from clean historical observations (i.e., FRED w/ History as Distillation Targets). As shown in the table below, we find that FRED with historical distillation targets substantially underperforms the original FRED, suggesting that distilling representations extracted from future observations is crucial for improving action generation.

  • To evaluate whether future prediction contributes to action generation beyond historical information, we train and evaluate a variant that conditions the action decoder on historical features instead of predicted future representations (i.e., FRED w/ History for Action Conditioning). As shown in the table below, we observe a substantial performance drop, even below the Base WAM, supporting the importance of predicted future representations for action generation beyond historical information alone.

\hline \text{Method} & \text{RoboCasa Kitchen} \\\\ \hline \mathbf{FRED\\ (Ours)} & \mathbf{68.8} \\\\ \text{w/ History as Distillation Targets} & 58.8 \\\\ \text{w/ History for Action Conditioning} & 56.4 \\\\ \hline \text{Base WAM (DiT4DiT)} & 61.4 \\\\ \hline \end{array}$$ --- **[Q3-2] Do inaccurate future predictions lead to cascading errors in action generation?** **[A]** In our WAM setup, inaccurate future predictions by the VDM can lead to incorrect action generation, as the separate action decoder is conditioned on intermediate future video representations. To investigate this issue, we analyze failure cases where implausible video predictions accompany incorrect actions in our OpenArm experiments. For example, on the *PnP Apple/Mango* task (see Figure 10), VDM in DiT4DiT (Base WAM) often predicts that the object is grasped and transported to the air fryer, even though the robot hand is still too far from the object, which appears to warp into the hand. The corresponding action sequence closes the hand without grasping the object and then proceeds to the air fryer with an empty hand, suggesting that wrong video prediction can result in incorrect action generation. We note that this analysis uses a full VDM forward pass at a single denoising step to decode future videos, while action generation remains conditioned on the intermediate hidden states of VDM.
AuthorsOpenReview ↗

Response to Reviewer GQz8 (3/4)

[Q4] Clarify the discrepancy between Tables 2(b) and 12(b)

[A] We would like to clarify that the discrepancy between Tables 2(b) and 12(b) was caused by an incorrect reporting of the DiT4DiT results in Table 12(b). Specifically, while Tables 2(b) and 12(b) are based on exactly the same evaluation runs, the “Pour Coffee” results for DiT4DiT co-trained on robot and human data in Table 12(b) were incorrectly reported as 66.7% instead of 25.0% for task success and 100.0% instead of 91.7% for instruction following. Therefore, the results in Table 2(b) are correct, and the improvement claims in the abstract remain unchanged.


[Q5-1] Clarifying the scoring rules for reported performance, e.g., whether 50% success rates in the abstract refer to the binary task success rates or the partial completion scores.

[A] We clarify that the reported scores for the DROID zero-shot environment generalization benchmark and OpenArm in-domain tasks allow partial credit, whereas the OpenArm human-to-robot skill transfer results use strict binary scoring based on full task completion as follows:

  • In-domain Easy, Out-of-domain, and Object Grounding tasks on DROID experiments and In-domain Atomic tasks on OpenArm experiments: We give partial credit of 0.5 out of 1.0 when the robot successfully grasps the correct target object but fails to complete the task, as it means that the model identifies the target object correctly.

  • In-domain Long-horizon tasks on DROID experiments and In-domain Composite tasks on OpenArm experiments: We give partial credit of 1/N for each completed atomic task (e.g., in a task requiring the robot to pick and place 5 objects consecutively, we give 1/5 for each completed PnP).

  • Human-to-robot Skill Transfer tasks on OpenArm experiments: We do not give partial credit for partial progress even though these tasks are atomic, to evaluate whether the robot can execute target behavior rather than merely identify or grasp the correct object.

Therefore, the 50.0% success rate reported in the abstract for human-to-robot skill transfer tasks refers to the strict binary task success rate, without partial credit.


[Q5-2] Reporting strict binary success rates

[A] Following your suggestion, we report the strict binary success rates for DROID zero-shot environment generalization benchmark and OpenArm in-domain tasks. As shown in the tables below, FRED still outperforms competing WAMs on DROID out-of-domain and object grounding tasks and OpenARM in-domain tasks, while achieving comparable or higher binary success rates on DROID in-domain tasks.

Table. Results on DROID zero-shot environment generalization (binary success rates)

\hline \text{Method} & \text{Backbone} & \text{In-domain Easy} & \text{In-domain Long-horizon} & \text{Out-of-domain} & \text{Object Grounding} \\\\ \hline \pi_{0.5} & \text{-} & 55.6 \pm 5.4 & 16.7 \pm 10.8 & 8.3 \pm 5.6 & 16.7 \pm 7.4 \\\\ \text{GR00T N1.7} & \text{-} & 41.7 \pm 5.6 & 16.7 \pm 10.8 & 4.2 \pm 4.0 & 20.8 \pm 8.3 \\\\ \hline \text{Fast-WAM} & \text{Wan2.2-5B} & 25.0 \pm 6.6 & 0.0 \pm 0.0 & 0.0 \pm 0.0 & 8.3 \pm 5.6 \\\\ \text{DiT4DiT} & \text{Wan2.2-5B} & 16.7 \pm 5.9 & 0.0 \pm 0.0 & 0.0 \pm 0.0 & 8.3 \pm 5.6 \\\\ \text{AGRA} & \text{Wan2.2-5B} & \mathbf{41.7 \pm 4.2} & 0.0 \pm 0.0 & 8.3 \pm 5.6 & 33.3 \pm 9.5 \\\\ \mathbf{FRED\\ (Ours)} & \text{Wan2.2-5B} & \mathbf{41.7 \pm 6.6} & \mathbf{16.7 \pm 10.8} & \mathbf{29.2 \pm 8.9} & \mathbf{37.5 \pm 9.9} \\\\ \hline \text{DreamZero} & \text{Wan2.1-14B} & 63.9 \pm 7.7 & \mathbf{25.0 \pm 12.5} & 8.3 \pm 5.4 & 16.7 \pm 7.4 \\\\ \mathbf{FRED\\ (Ours)} & \text{Wan2.1-14B} & \mathbf{77.8 \pm 6.7} & \mathbf{25.0 \pm 12.5} & \mathbf{41.7 \pm 9.9} & \mathbf{54.2 \pm 10.1} \\\\ \hline \end{array}$$ **Table. Results on OpenArm in-domain atomic tasks (binary success rates)** $$\begin{array}{l|cc} \hline \text{Method} & \text{w/o Human} & \text{w/ Human} \\\\ \hline \pi_{0.5} & \text{-} & 63.9 \pm 6.2 \\\\ \text{GR00T N1.7} & \text{-} & 91.7 \pm 4.5 \\\\ \hline \text{DreamZero} & 61.1 \pm 6.6 & 58.3 \pm 5.3 \\\\ \text{Fast-WAM} & 55.6 \pm 7.4 & 63.9 \pm 6.5 \\\\ \text{DiT4DiT} & 77.8 \pm 6.0 & 86.1 \pm 5.5 \\\\ \text{AGRA} & 80.6 \pm 6.5 & 75.0 \pm 6.6 \\\\ \mathbf{FRED\\ (Ours)} & \mathbf{91.7 \pm 4.2} & \mathbf{97.2 \pm 2.7} \\\\ \hline \end{array}$$
Reviewer_Mp7sOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FRED, a framework for improving future video representations in World Action Models (WAMs). FRED uses self-distillation to align representations generated from noise with those extracted from ground-truth future observations. It also incorporates historical observations to provide richer temporal context. Experiments across simulation benchmarks and real-world robot platforms show improvements in generalization, long-horizon manipulation, and human-to-robot skill transfer.

Critical strengths
  1. The paper addresses an important problem in WAMs by explicitly improving the future representations used for action prediction. The proposed framework is simple and can be integrated into existing architectures without introducing additional inference-time denoising steps.
  2. The experimental results are impressive, particularly on out-of-domain manipulation and human-to-robot skill transfer. I also appreciate the extensive evaluation across multiple simulation benchmarks, real-world robot platforms, and different video backbones.
  3. The paper provides useful analysis beyond reporting task success rates. In particular, the representation probing experiments in Figure 3 offer interesting insights into how the quality of single-step future representations relates to downstream control performance.
Critical weaknesses
  1. The novelty of the proposed method seems relatively limited. The self-distillation objective largely follows established techniques such as DINO and iBOT, while conditioning on observation history is also a common practice. Although applying these ideas to WAMs is meaningful, the paper does not introduce a substantially new learning mechanism. The distinction from existing representation alignment methods, particularly SRA and AGRA, could be better justified.
  2. It is difficult to determine how much of the reported improvement comes specifically from future representation distillation. FRED introduces both self-distillation and historical observation conditioning. While Table 3 provides component ablations in simulation, the main real-world experiments lack similar controlled comparisons. Moreover, Table 7 indicates that FRED uses five observation frames while DiT4DiT uses only one. This makes it unclear whether the performance gains are primarily due to better representations or additional temporal information.
  3. I am not fully convinced that the experiments demonstrate the claimed ability to predict meaningful future representations. The analysis in Figure 3 relies on motion-primitive classification accuracy as a proxy for representation quality. However, good classification performance does not necessarily imply that the representations accurately capture future states or dynamics. The reported correlation is interesting, but it is based on only a few model variants and does not establish a causal relationship with control performance.
  4. The real-world evaluation is somewhat limited. Zero-shot generalization is tested in only one unseen environment, and the human-to-robot transfer experiments cover three tasks with 12 trials each. It is unclear whether the improvements would remain consistent across more diverse environments, objects, and manipulation behaviors.
Questions for authors
  1. Could you further clarify how FRED differs from existing self-distillation and representation alignment approaches, particularly SRA and AGRA?
  2. Could you provide additional real-world ablations that isolate the contributions of self-distillation and historical conditioning under matched observation settings?
  3. Could you provide more direct evidence that the learned representations capture meaningful future states or dynamics, beyond motion-primitive classification accuracy?
  4. Could you clarify the discrepancy between Tables 2(b) and 12(b) and explain whether the reported task success rates include partial completion scores?
3 comments
AuthorsOpenReview ↗

Response to Reviewer Mp7s (1/3)

Dear reviewer Mp7s,
We sincerely appreciate your efforts in reviewing our manuscript. We respond to each comment in the following content.


[W1-1] Limited methodological novelty

[A] We would like to emphasize that FRED’s key contribution lies in reframing the problem of improving action prediction as a representation learning problem through a self-distillation framework tailored to WAMs. To this end, FRED uses a representation learning objective, inspired by DINO and iBOT, to train representations generated from noise to predict those extracted from noisy ground-truth future observations, encouraging them to better capture future robot dynamics for action generation. We additionally condition the VDM on historical observations to provide temporal cues (e.g., recent robot motion) that support this prediction. We believe that our work highlights a promising direction for robot learning: learning video representations that encode richer future information can translate into better policy performance without directly optimizing the VDM within WAMs for action generation.


[W1-2/Q1] Clarify differences from existing representation alignment methods, e.g., SRA and AGRA

[A] A key difference between FRED and existing representation alignment methods for generative modeling, e.g., SRA [1], is that FRED improves future representations generated from noise for action conditioning in WAMs, whereas existing methods improve representations of noisy inputs to enhance generation quality. Specifically, FRED views the VDM as a future representation generator and uses self-distillation to align representations generated from Gaussian noise with those extracted by a momentum teacher from noisy ground-truth future observations at the same VDM layer used for action conditioning. In contrast, methods such as SRA often align higher-noise, earlier-layer representations with lower-noise, later-layer representations to improve generative training.

For AGRA [2], which applies representation alignment to WAMs, it uses a frozen visual encoder (e.g., DINOv2 [3]) to provide semantically structured features for the action decoder, while FRED instead learns to better capture future dynamics using representations extracted from ground-truth observed futures as targets, without alignment to an external representation space.

[1] Jiang et al., No Other Representation Component Is Needed: Diffusion Transformers Can Provide Representation Guidance by Themselves, ICLR 2025
[2] Qiu et al., Making Foresight Actionable: Repurposing Representation Alignment in World Action Models, arXiv 2026
[3] Oquab et al., DINOv2: Learning Robust Visual Features without Supervision, TMLR 2024

AuthorsOpenReview ↗

Response to Reviewer Mp7s (2/3)

[W2/Q2] Contribution of self-distillation and historical observations in real-world experiments

[A] Following your constructive suggestions, we additionally evaluate two variants of FRED (i.e., FRED without history conditioning and FRED without the self-distillation objective) in real-world environments. Specifically, we evaluate these variants on (a) zero-shot environment generalization by training the models using the DROID dataset and testing them on the in-domain long-horizon task (we train models for 100K iterations for the ablation experiment); and (b) human-to-robot skill transfer by training the models using the compositional dataset of OpenArm robot demonstrations and human videos and evaluating them on 3 human-to-robot skill transfer tasks (PnP Bottle on Basket, PnP Novel Objects, and Pour Coffee).

As shown in the table below, both components provide complementary gains in real-world experiments (both zero-shot environment generalization and human-to-robot skill transfer), as removing either component reduces performance, while removing both leads to further degradation.

\hline \text{Method} & \text{DROID (Long-horizon)}&\text{OpenArm (Human2Robot)} \\\\ \hline \mathbf{FRED\\ (Ours)}&\mathbf{43.3}&\mathbf{50.0\\ (SR)\\ /\\ 97.2\\ (IF)} \\\\ \text{w/o Self-distillation}&21.7&25.0\\ (\text{SR})\\ /\\ 77.8\\ (\text{IF}) \\\\ \text{w/o History}&28.3&38.9\\ (\text{SR})\\ /\\ 83.3\\ (\text{IF}) \\\\ \hline \text{Base WAM (DiT4DiT)}&8.3&19.4\\ (\text{SR})\\ /\\ 55.6\\ (\text{IF}) \\\\ \hline \end{array}$$ --- **[W3/Q3] Clarify the method’s effectiveness in learning meaningful future representations** **[A]** We would like to clarify that the probing analysis in Figure 3 assesses how well the generated representations encode future robot motion, and FRED’s improved probing accuracy suggests that it learns more informative future video representations. Concretely, motion-primitives are defined by end-effector movements and gripper state changes over time (see Appendix D.2) and cannot be determined from the initial state alone. To further support our claim, we use CKNNA [1] to measure the structural similarity between future representations generated from noise and those extracted from ground-truth future observations for the Base WAM and FRED in Figure 3(c). As shown in the table below, we find that FRED consistently improves the Base WAM’s CKNNA score across all training iterations from 20K to 100K, suggesting that our training scheme helps the generated future representations better capture future robot states and dynamics. **Table. Structural similarity to representations extracted from clean future observations, measured by CKNNA (higher is better)** $$\begin{array}{l|c|c|c|c|c} \hline \text{Method} & \text{20K iter} & \text{40K iter} & \text{60K iter} & \text{80K iter} & \text{100K iter} \\\\ \hline \text{Base WAM }(\tau=1.0) & 0.2473 & 0.2585 & 0.2657 & 0.2818 & 0.2942 \\\\ \textbf{FRED }(\tau=1.0) & 0.4152 & 0.4174 & 0.4248 & 0.4320 & 0.4372 \\\\ \hline \end{array}$$ [1] Huh et al., The Platonic Representation Hypothesis., ICML 2024
AuthorsOpenReview ↗

Response to Reviewer Mp7s (3/3)

[W4] Limited real-world experiments to support the method’s effectiveness

[A] We believe our real-world experiments provide meaningful evidence of the effectiveness of FRED in both zero-shot environment generalization and human-to-robot skill transfer, because FRED shows substantial performance gains (e.g., 10.4% →\rightarrow 52.1% on DROID out-of-domain tasks and 19.4% →\rightarrow 50.0% on human-to-robot skill transfer tasks) across diverse evaluation tasks where the scale is comparable to prior works. For example, we conduct 8 tasks x 12 trials on DROID and 7 tasks x 12 trials on OpenArm per model, compared with 40 tasks x 2 trials for DreamZero on DROID and 7 tasks x 20 trials for DiT4DiT on Unitree G1. Moreover, DreamZero on DROID and DiT4DiT on Unitree G1 also evaluate zero-shot generalization in a single environment, as we do.

Nevertheless, we agree that expanding evaluation to additional tasks will strengthen our evaluation. To address these concerns, we expand our real-world evaluations for both zero-shot environment generalization and human-to-robot skill transfer. Specifically, for zero-shot environment generalization, we first introduce an additional unseen environment with a different tabletop setting (e.g.,a modified surface) and evaluate models trained on the DROID dataset on our in-domain easy tasks (see Appendix B.1). We also introduce an additional out-of-domain task (Perch Hat on Peg) to evaluate generalization to more diverse manipulation behaviors. For human-to-robot skill transfer, we introduce a novel placement task (PnP Box to Center), while increasing the number of trials per object category for the novel object tasks. As shown in the tables below, FRED still consistently outperforms the other WAM baselines, supporting its effectiveness across diverse environments, objects and manipulation behaviors.

Table. Results on additional tasks for zero-shot environment generalization (DROID). We report task success rates (%) with partial credit.

\hline \text{Method} & \text{Backbone} & \text{New Environment (In-domain Easy)} & \text{Out-of-domain (Perch Hat on Peg)} \\\\ \hline \text{Fast-WAM} & \text{Wan2.2-5B} & 30.6 & 0.0 \\\\ \text{DiT4DiT} & \text{Wan2.2-5B} & 12.5 & 0.0 \\\\ \text{AGRA} & \text{Wan2.2-5B} & 47.2 & 8.3 \\\\ \mathbf{FRED\\ (Ours)} & \text{Wan2.2-5B} & \mathbf{55.6} & \mathbf{41.7} \\\\ \hline \text{DreamZero} & \text{Wan2.1-14B} & 70.8 & 41.7 \\\\ \mathbf{FRED\\ (Ours)} & \text{Wan2.1-14B} & \mathbf{73.6} & \mathbf{66.7} \\\\ \hline \end{array}$$ **Table. Results on additional human skill transfer tasks (OpenArm; w/ human).** We report task success rates (%). $$\begin{array}{l|ccccc}\hline\text{Method}&\text{PnP Box to Center}&\text{PnP Novel Bottle}&\text{PnP Novel Box}&\text{PnP Novel Doll}&\text{PnP Novel Ball} \\\\ \hline\text{DreamZero}&0.0&8.3&0.0&0.0&0.0 \\\\ \text{Fast-WAM}&0.0&0.0&0.0&0.0&0.0 \\\\ \text{DiT4DiT}&0.0&0.0&25.0&25.0&0.0 \\\\ \text{AGRA}&0.0&0.0&0.0&0.0&0.0 \\\\ \mathbf{FRED\\ (Ours)}&\mathbf{33.3}&\mathbf{66.7}&\mathbf{50.0}&\mathbf{75.0}&\mathbf{41.7} \\\\ \hline \end{array}$$ --- **[Q4] Clarify the reported real-world experimental results: (1) Discrepancy between Table 2(b) and 12(b), and (2) scoring rules for reported performance** **[A1]** We clarify that the discrepancy between Tables 2(b) and 12(b) was caused by an incorrect reporting of the DiT4DiT results in Table 12(b). Specifically, while Tables 2(b) and 12(b) are based on exactly the same evaluation runs, the “Pour Coffee” results for DiT4DiT co-trained on robot and human data in Table 12(b) were incorrectly reported as 66.7% instead of 25.0% for task success and 100.0% instead of 91.7% for instruction following. **[A2]** We clarify that the reported scores for the DROID zero-shot environment generalization benchmark and OpenArm in-domain tasks allow partial credit, whereas the OpenArm human-to-robot skill transfer results use strict binary scoring based on full task completion as follows: - *In-domain Easy*, *Out-of-domain*, and *Object Grounding* tasks on DROID experiments and *In-domain Atomic* tasks on OpenArm experiments: We give partial credit of 0.5 out of 1.0 when the robot successfully grasps the correct target object but fails to complete the task, as it means that the model identifies the target object correctly. - *In-domain Long-horizon* tasks on DROID experiments and *In-domain Composite* tasks on OpenArm experiments: We give partial credit of 1/N for each completed atomic task (e.g., in a task requiring the robot to pick and place 5 objects consecutively, we give 1/5 for each completed PnP). - *Human-to-robot Skill Transfer* tasks on OpenArm experiments: We do not give partial credit for partial progress even though these tasks are atomic, to evaluate whether the robot can execute target behavior rather than merely identify or grasp the correct object.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.