- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
FRED trains a video-based world action model to predict useful future representations in one denoising pass. An EMA teacher extracts targets from noisy future observations, while the student predicts them from noise and observation history; a separate action decoder uses these features. Experiments cover real-world environment generalization, human-video skill transfer, and three simulation benchmarks, with ablations of self-distillation, history, and training choices.
Critical strengths
-
The objective targets the features actually consumed by the action decoder. This is a clear motivation for future-representation learning beyond improving generated video quality.
-
The real-world transfer results are promising. Human-video co-training raises the reported transfer score to 50.0%, versus 19.4% for DiT4DiT, and the 14B model shows large gains on the evaluated unseen-environment tasks.
-
The analysis covers several relevant alternatives and design choices. Table 3 separates history and self-distillation in simulation, and Tables 4–7 examine representation objectives, another video backbone, and inference cost.
Critical weaknesses
-
The real-world metric is ambiguous. Tables 1–2 use 'success rate', but Appendices B.1–B.2 allow partial credit even beyond long-horizon tasks. Without the scoring rubric and full-completion counts, the headline transfer and generalization numbers are difficult to interpret.
-
The strongest claims concern real-world transfer, but the history/self-distillation ablation is only on RoboCasa. The real-world comparisons do not isolate how much of the gain comes from self-distillation rather than additional observation history or other training differences.
-
Each physical task has only 12 trials, with no reported uncertainty or independent training-seed variation. The out-of-domain set contains just two tasks, and human transfer three, so the breadth and reliability of the generalization claim remain uncertain.
Questions for authors
-
Which entries include partial credit, and what are the exact scoring rules? Please report binary full-task completion alongside progress and instruction-following scores.
-
On the human-transfer or unseen-environment tasks, how does a history-matched model perform when only the self-distillation loss is removed?
-
Can you provide uncertainty for the physical results and clarify whether the improvements persist across training runs and task configurations?
2 comments
Response to Reviewer P9FM (2/2)
[W3/Q3] Uncertainty about the breadth and reliability of generalizability
[A] We believe our real-world experiments provide meaningful evidence of the effectiveness of FRED in both zero-shot environment generalization and human-to-robot skill transfer, because FRED shows substantial performance gains (e.g., 10.4%$\rightarrow$52.1% on DROID out-of-domain tasks and 19.4%$\rightarrow$50.0% on human skill transfer tasks) across tasks where the scale is comparable to prior works. For example, we conduct 8 tasks x 12 trials on DROID and 7 tasks x 12 trials on OpenArm per model, compared with 40 tasks x 2 trials for DreamZero on DROID and 7 tasks x 20 trials for DiT4DiT on Unitree G1.
Nevertheless, we agree that reporting uncertainty scores and expanding additional tasks will strengthen our evaluation. To address these concerns, we conduct the following additional experiments.
- Reporting uncertainty scores: For the original results in Tables 1 and 2, we compute the mean and its standard deviation calculated using the rliable library [1] for each evaluation category. As shown in the tables below, FRED consistently achieves substantial improvements over baselines, e.g., improving out-of-domain performance on DROID from 10.4$\pm$5.5% to 52.1$\pm$8.2% at the 14B scale, as well as human-to-robot transfer task success on OpenARM from 19.4$\pm$6.2% to 50.0$\pm$7.8%.
Table. Zero-shot environment generalization (DROID)
Table. Human-to-robot skill transfer (OpenArm; w/ human)
- Reporting additional tasks: We further evaluate our models on newly designed tasks for both zero-shot environment generalization and human-to-robot skill transfer. For zero-shot environment generalization, we introduce an additional out-of-domain task (Perch Hat on Peg). For human-to-robot skill transfer, we introduce a novel placement task (PnP Box to Center), while increasing the number of trials per object category for the novel object tasks. As shown in the tables below, FRED still consistently outperforms the other WAM baselines, supporting its effectiveness across diverse objects and manipulation behaviors.
Table. Results on additional out-of-domain task (DROID). We report task success (%) with partial credit.
Table. Results on additional human skill transfer tasks (OpenArm; w/ human). We report task success (%).
Response to Reviewer P9FM (1/2)
Dear reviewer P9FM,
We sincerely appreciate your efforts in reviewing our manuscript. We respond to each comment in the following content.
[W1/Q1-1] Clarifying the scoring rules for reported performance
[A] We clarify that the reported scores for the DROID zero-shot environment generalization benchmark and OpenArm in-domain tasks allow partial credit, whereas the OpenArm human-to-robot skill transfer results use strict binary scoring based on full task completion as follows:
In-domain Easy, Out-of-domain, and Object Grounding tasks on DROID experiments and In-domain Atomic tasks on OpenArm experiments: We give partial credit of 0.5 out of 1.0 when the robot successfully grasps the correct target object but fails to complete the task, as it means that the model identifies the target object correctly.
In-domain Long-horizon tasks on DROID experiments and In-domain Composite tasks on OpenArm experiments: We give partial credit of 1/N for each completed atomic task (e.g., in a task requiring the robot to pick and place 5 objects consecutively, we give 1/5 for each completed PnP).
Human-to-robot Skill Transfer tasks on OpenArm experiments: We do not give partial credit for partial progress even though these tasks are atomic, to evaluate whether the robot can execute target behavior rather than merely identify or grasp the correct object.
[Q1-2] Reporting strict binary success rates
[A] Following your suggestion, we report the strict binary success rates for DROID zero-shot environment generalization benchmark and OpenArm in-domain tasks. As shown in the tables below, FRED still outperforms competing WAMs on DROID out-of-domain and object grounding tasks and OpenARM in-domain tasks, while achieving comparable or higher binary success rates on DROID in-domain tasks.
Table. Results on DROID zero-shot environment generalization (binary success rates) Methodπ0.5GR00T N1.7Fast-WAMDiT4DiTAGRAFRED(Ours)DreamZeroFRED(Ours)Backbone−−Wan2.2-5BWan2.2-5BWan2.2-5BWan2.2-5BWan2.1-14BWan2.1-14BIn-domain Easy55.6±5.441.7±5.625.0±6.616.7±5.941.7±4.241.7±6.663.9±7.777.8±6.7In-domain Long-horizon16.7±10.816.7±10.80.0±0.00.0±0.00.0±0.016.7±10.825.0±12.525.0±12.5Out-of-domain8.3±5.64.2±4.00.0±0.00.0±0.08.3±5.629.2±8.98.3±5.441.7±9.9Object Grounding16.7±7.420.8±8.38.3±5.68.3±5.633.3±9.537.5±9.916.7±7.454.2±10.1
Table. Results on OpenArm in-domain atomic tasks (binary success rates)
\hline \end{array}$$ --- **[W2/Q2] Contribution of self-distillation and historical observations in real-world experiments** **[A]** Following your constructive suggestions, we additionally evaluate two variants of FRED (i.e., FRED without history conditioning and FRED without the self-distillation objective) in real-world environments. Specifically, we evaluate these variants on (a) zero-shot environment generalization by training the models using the DROID dataset and testing them on the in-domain long-horizon task (we train models for 100K iterations for the ablation); and (b) human-to-robot skill transfer by training the models using the compositional dataset of OpenArm robot demonstrations and human videos and evaluating them on 3 human-to-robot skill transfer tasks (PnP Bottle on Basket, PnP Novel Objects, and Pour Coffee). As shown in the table below, both components provide complementary gains in real-world experiments (both zero-shot environment generalization and human-to-robot skill transfer), as removing either component reduces performance, while removing both leads to further degradation. $$\begin{array}{l|cc}\hline\text{Method}&\text{DROID (Long-horizon)}&\text{OpenArm (Human2Robot)}\\\\ \hline\mathbf{FRED\\ (Ours)}&\mathbf{43.3}&\mathbf{50.0(SR)\\ /\\ 97.2\\ (IF)}\\\\ \text{w/o Self-distillation}&21.7&25.0(\text{SR})\\ /\\ 77.8\\ (\text{IF})\\\\ \text{w/o History}&28.3&38.9(\text{SR})\\ /\\ 83.3\\ (\text{IF})\\\\ \hline \text{Base WAM (DiT4DiT)}&8.3&19.4(\text{SR})\\ /\\ 55.6\\ (\text{IF}) \\\\ \hline \end{array}$$