acceptodds
Under review as a conference paper at ICLR 2027

Future Representation Distillation for World Action Models

12 views

Abstract

World Action Models (WAM) use the predictive knowledge of pretrained video diffusion models for robot control through future video prediction. However, iterative video denoising is costly, and inaccurate visual predictions can compromise action prediction. Our key idea is to make this predictive knowledge available through visual representations computed from the current observation, enabling action prediction without generating future videos. We introduce DreamFree, a WAM that repurposes a pretrained video diffusion model as a predictive visual encoder through representation distillation. The visual encoder learns to predict a frozen video diffusion model's representation using only the current observation. An action module is then trained to predict actions by extracting action-relevant information from the visual encoder. Across real-world and simulated tasks, DreamFree achieves higher success rates than state-of-the-art VLAs and WAMs, while performing real-time closed-loop control at 11Hz with a 14B video diffusion backbone. Experiments show that DreamFree can translate stronger video priors and additional video data into better robot control and enable learning new robot tasks from video demonstrations, including videos from a different embodiment, without action annotation for the new tasks.

Primary area:
applications to robotics, autonomy, planning
Keywords:
world action model, video diffusion model, video action model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ezkbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes DreamFree, a World Action Model for robotic manipulation. It uses a pretrained video diffusion model as a teacher and applies future representation distillation to train a visual encoder that extracts predictive information from current observations, avoiding iterative future video generation at inference time. An action module is subsequently trained using the visual features and robot states. The evaluation covers LIBERO, RoboCasa-Kitchen, and real-world DROID tasks, along with additional studies on extra video data, cross-embodiment transfer, video-task transfer, inference latency, and visual representation quality. The results demonstrate competitive performance on several robotic manipulation tasks and substantially reduced inference latency. However, the distinction between the proposed representation distillation approach and existing predictive representation learning and feature distillation methods remains insufficiently clear.

Critical strengths

The main strength of this paper is its broad experimental coverage. Beyond reporting task success rates, the authors investigate the role of distillation through loss ablations, representation probing, and video-task transfer experiments. The paper reports substantial performance improvements on RoboCasa-Kitchen and competitive results on real-world robotic tasks. In addition, the inference latency evaluation shows that avoiding iterative future video generation can improve control efficiency. These results provide meaningful empirical evidence for the practical value of predictive visual representations in robotic manipulation.

Critical weaknesses
  • Limited methodological novelty. Equations (2) and (4) largely adopt standard flow-matching objectives, while Eq. (1) primarily transfers predictive information from a pretrained video model into visual representations of current observations through feature distillation. Although this design is reasonable, it does not introduce a clearly novel distillation mechanism or learning objective, limiting the originality of the core method. Furthermore, the paper does not adequately identify fundamental limitations of existing predictive representation learning methods or REPA, nor does it demonstrate through direct comparisons that the proposed distillation approach offers distinct advantages over these methods. Thus, the experiments support the effectiveness of the approach, but the evidence for its originality and relative advantages remains insufficient.

  • Insufficient differentiation from prior work and an underdeveloped motivation. The paper does not clearly identify fundamental limitations of existing predictive representation learning methods or REPA, nor does it demonstrate that DreamFree addresses problems that these methods cannot adequately handle. The current experiments mainly establish that distillation is effective within DreamFree, but do not sufficiently demonstrate its distinct advantages over existing predictive representation learning and feature distillation methods.

  • The performance gains require cautious interpretation. Although the paper covers multiple simulation benchmarks, real-world robotic tasks, and transfer settings, the magnitude of improvement varies across tasks, and the gains on some benchmarks are small. In addition, the number of trials per task in the real-world experiments is limited. The current results support the practical potential of the approach, but do not establish consistent advantages across a broad range of task settings.

Questions for authors
  1. Distinction from prior work: Could the authors clarify the technical differences between DreamFree and existing predictive representation learning methods, including REPA? Specifically, what problems does future representation distillation address that existing methods do not adequately resolve?

  2. Independent contribution of the proposed method: Beyond the ablation that removes the distillation loss, could the authors provide direct comparisons with alternative predictive representation learning or feature distillation methods to establish whether the proposed distillation design offers distinct advantages?

  3. Scope of the performance gains: Given that the magnitude of improvement varies across tasks, could the authors further analyze the task settings in which DreamFree is most effective and clarify whether its performance advantages are consistent across repeated trials?

5 comments
AuthorsOpenReview ↗

Response to Reviewer ezkb [1/2]

We sincerely thank you for your time and effort in reviewing our paper. We appreciate your positive comments on:

  • Our broad experimental coverage, including simulation, real-world evaluations, and transfer settings
  • Our investigation of distillation through loss ablations, representation probing, and video-task transfer experiments
  • Substantial performance improvements
  • Improved control efficiency

Please find our responses to your comments below.


W1: Methodological novelty and independent contribution

Our main contribution is future representation distillation for imagination-free robot control, which transfers a pretrained video model’s predictive knowledge into representations computed from the current observation and instruction.

The problem we address is how to make this predictive knowledge directly available in the visual features used for action prediction. The video generation objective does not explicitly align these features with future representations. DreamFree introduces this supervision by training current observation features to anticipate future frame representations from a pretrained video diffusion teacher.

Eq.(1) addresses this through two design choices:

  • Restricted student information: The encoder’s aligned features depend only on the current observation and instruction. A structured attention mask prevents these features from accessing future frames.
  • Future-informed prediction target: The teacher processes current and future frames, and a projector maps the encoder’s current observation features to the teacher’s future representations.

Eq.(1) therefore trains the encoder to anticipate features of future content unavailable to the encoder. Our contribution lies in this predictive distillation formulation and its integration into imagination-free control.

Note that Eqs. (2) and (4) use standard flow matching objectives for video and action modeling. They serve as supporting components of our method, and we do not claim methodological novelty in these objectives.


W2: Technical differences from existing predictive representation learning methods or REPA

Thank you for requesting this clarification. DreamFree shares the goal of learning useful visual representations with these methods. However, it addresses the specific problem of transferring a pretrained video diffusion model's predictive knowledge into an encoder for imagination-free control.

Compared with REPA, the information available to the student and the direction of knowledge transfer differ. REPA aligns a diffusion model’s features with clean-image representations from a pretrained visual encoder to improve generation. DreamFree instead uses the video diffusion model as the teacher. Its student must predict representations of future content from the current observation, under the information restriction described in W1. Applying same-content alignment alone does not impose this requirement.

Compared with V-JEPA 2, both methods learn to predict unobserved visual content, but their representation targets are constructed differently. V-JEPA 2 jointly learns an encoder and predictor using masked representations supplied by an exponential-moving-average target encoder. DreamFree instead uses intermediate representations from a frozen pretrained video diffusion teacher. Our objective is to transfer the teacher’s existing video-prediction knowledge into current observation features for action learning.

Compared with FLARE, the differences concern the source and location of supervision. FLARE aligns features inside the action denoising network with future-observation embeddings from a vision-language encoder. On the other hand, DreamFree distills future-informed features from a video diffusion teacher into a separate visual encoder before action learning. This separates predictive visual representation learning from action-policy optimization and places the transferred video knowledge directly in the encoder used for control.

We added this discussion in Appendix E.1 to clarify these technical distinctions and the specific problem DreamFree addresses.


(Response continued in the next comment.)

AuthorsOpenReview ↗

Response to Reviewer ezkb [2/2]

W3: Direct comparisons with alternative predictive representation learning or feature distillation methods

Thank you for suggesting these comparisons. Among the methods discussed, FLARE provides the closest comparison for robot control, as it also uses future representation alignment to improve action prediction. Our RoboCasa-Kitchen evaluation includes this comparison:

  • DreamFree achieves 71.7% success, compared with 66.4% success for FLARE.

REPA and V-JEPA 2 address different learning formulations and cannot be directly substituted into our representation-learning stage:

  • REPA aligns a diffusion model’s hidden features with pretrained visual representations of the corresponding clean image. Applying it to our setting would require specifying a current-observation-to-future prediction task, including the teacher representations and temporal alignment. This would evaluate an adaptation of REPA to predictive distillation rather than its original formulation.
  • V-JEPA 2 jointly learns an encoder and predictor using an evolving target encoder. Applying its learning procedure would require redesigning our representation-learning stage. Using a frozen V-JEPA 2 encoder as our teacher would evaluate the choice of teacher, rather than V-JEPA 2’s predictive representation-learning method itself.

A controlled comparison with adaptations of REPA or V-JEPA 2 would therefore require additional design choices beyond their published formulations.

We use FLARE as the closest existing method-level comparison for predictive representation learning in robot control, although differences in backbone and training setup prevent attributing the gap solely to the alignment design.


W4: Performance gains in simulation (LIBERO, RoboCasa-Kitchen)

We agree that the magnitude of improvement varies across benchmarks and tasks. On LIBERO, performance is nearly saturated, and we interpret DreamFree’s results as competitive performance on a standard benchmark and place greater emphasis on the more challenging RoboCasa-Kitchen evaluation.

On RoboCasa-Kitchen, we evaluate each model over 50 trials per task across 24 tasks. The overall average includes tasks where methods already achieve nearly saturated performance, such as CloseDrawer. Thus, the average gain across all tasks is less pronounced than the gain on challenging pick-and-place (PnP) tasks.

The table below shows success rates (%) for representative tasks.

Task / groupFast-WAM 5BFast-WAM 14BDreamFree 5BDreamFree 14BDreamFree 14B + extra videos
PnP: Counter → Stove2416346674
PnP: Counter → Sink2620547272
PnP: Sink → Counter4218546876
CloseDrawer9810010010098
TurnOffMicrowave981009410090
All 8 PnP tasks19.811.837.358.565.0
All tasks44.444.758.571.774.4

DreamFree 14B achieves substantial improvements over Fast-WAM 14B on all eight PnP tasks, with gains ranging from 20 to 68 percentage points. Across PnP tasks, DreamFree achieves 58.5% success compared to 11.8% for Fast-WAM. Additional videos further increase the average to 65.0%. These results identify pick-and-place as a setting with particularly strong gains over Fast-WAM.


W5: Real-world results and evaluation scope

We appreciate the reviewer’s concern regarding the limited number of real-world trials. Given the practical constraints of real-world evaluation, we prioritized task diversity, evaluating 10 tasks with five trials per task. We acknowledge that additional trials would help further assess the consistency of the observed performance gains.

To improve comparability across policies, we used image overlays to align initial scene configurations and reduce setup variation. All real-world rollouts, including successes and failures, are provided in the supplementary material.


W6: Task-dependent gains in real-world evaluation

DreamFree achieves higher success than π0.5\pi_{0.5} on nine of ten tasks, while gains over DreamZero vary across tasks. DreamFree achieves higher success on four tasks and the clearest improvements occur in towel placement (Cover strawberry with towel) and successive object rearrangement (Put bolts in bowl).

We hypothesize that directly supervising predictive features helps the encoder retain information about spatial relations and scene changes that is useful for these tasks. For example, covering the strawberry requires anticipating the towel’s placement and resulting coverage, while placing multiple bolts requires adapting actions as the arrangement changes after each placement. We interpret these results as evidence of practical effectiveness on the evaluated tasks, with particularly promising gains in these specific settings.


Please let us know if there are any remaining points we should clarify. We are happy to provide further details.

AuthorsOpenReview ↗

Follow-up on Our Response and Revisions for Reviewer ezkb

Thank you again for your time and valuable feedback on our paper.

In our response, we:

  • Clarified DreamFree’s contribution and technical distinctions from REPA, V-JEPA 2, and FLARE.
  • Discussed the comparison with FLARE and its limitations.
  • Provided task-level performance analyses, highlighting gains on challenging pick-and-place tasks while acknowledging the limited number of real-world trials.

As the author–reviewer discussion ends on November 18, we would appreciate hearing whether any concerns remain. We would be happy to provide further clarification before the deadline.

If our responses and revisions have satisfactorily addressed your concerns, we would be grateful if you would consider revising your score upward.

Yours sincerely, Authors

Reviewer_KM8rOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DreamFree, a method for leveraging pretrained video diffusion models for robot control without generating future videos at inference. The main idea is to distill future representations from a frozen video diffusion model into a visual encoder that only observes the current frame. The encoder is trained with both representation distillation and video flow matching, then frozen while a separate action diffusion transformer is trained on robot demonstrations. The authors evaluate on LIBERO, RoboCasa, and real-world DROID manipulation tasks, and demonstrate task transfer from videos without action annotations.

Critical strengths
  1. The video-only task transfer experiments are interesting. The authors show that additional videos of new composite tasks can improve execution without corresponding action labels, including demonstrations from a different robot embodiment. They also demonstrate adaptation of a trained policy by updating only the visual encoder while keeping the action module frozen.
  2. The paper demonstrates a practical implementation using a large pretrained video diffusion backbone, achieving inference latency comparable to a direct VLA policy without generating future videos at inference.
Critical weaknesses
  1. The novelty and motivation of the proposed representation distillation are not fully convincing. Fast-WAM [1] has already shown that pretrained video models can support action prediction without future video generation at inference, while FLARE [2] uses future representation alignment for action learning. Table 3 shows that distillation improves performance, but does not establish why the chosen teacher representations are particularly effective. The teacher operates at a fixed, relatively high noise level while also being conditioned on the current observation and instruction. It is unclear how much of the benefit comes from information about the demonstrated future versus the teacher's pretrained representations more generally. Comparisons with current observation teacher features without information about the demonstrated future (when td=1t_d=1), other teacher noise levels, or future representations from a predictive video encoder such as V-JEPA 2 [3] would help clarify the contribution.
  2. The benchmark evaluations could better establish the advantages over existing methods. Standard LIBERO is nearly saturated, making the small differences from strong baselines difficult to interpret. An evaluation on a more challenging benchmark such as LIBERO-Plus [10], which tests robustness under visual and environmental perturbations, would be more informative. On RoboCasa, the updated Cosmos Policy using Cosmos-Predict2.5 [4] reports 71.1% success with 50 demonstrations per task, close to DreamFree's 71.7%. Discussing this result would help contextualize the reported performance. Since the Fast-WAM RoboCasa results are reproduced by the authors, it would also be useful to clarify how the 14B variant was adapted and whether it received comparable video-backbone adaptation.
  3. The connection to latent action and motion-centric representation learning is underexplored. A substantial body of work, including LAPA [5], Moto [6], UniVLA [7], ViPRA [8], and Motus [9], investigates learning action relevant representations from videos without action annotations. These approaches emphasize motion or visual transitions rather than directly predicting future visual representations. While they are not equivalent to DreamFree, they provide relevant alternative perspectives on transferring video knowledge to robot control. The paper cites Motus but does not meaningfully discuss this connection or motivate its choice of future visual representations relative to motion-centric alternatives.
Questions for authors
  1. How sensitive is performance to the teacher noise level? Have the authors considered distillation with (td=1t_d=1) or alternative predictive video encoders to better isolate the benefit of future-conditioned diffusion features?
  2. Could the authors provide more details on the Fast-WAM reproduction and comment on how the updated Cosmos-Predict2.5 results compare under the respective evaluation settings?

References

[1] Yuan et al. Fast-WAM: Do World Action Models Need Test-time Future Imagination? arXiv:2603.16666, 2026.

[2] Zheng et al. FLARE: Robot Learning with Implicit World Modeling. CoRL, 2025.

[3] Assran et al. V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning. arXiv:2506.09985, 2025.

[4] Kim et al. Cosmos Policy: Fine-Tuning Video Models for Visuomotor Control and Planning. ICLR, 2026. Updated Cosmos-Predict2.5 results, NVIDIA Cosmos Cookbook, February 2026.

[5] Ye et al. Latent Action Pretraining from Videos. ICLR, 2025.

[6] Chen et al. Moto: Latent Motion Token as the Bridging Language for Learning Robot Manipulation from Videos. ICCV, 2025.

[7] Bu et al. Learning to Act Anywhere with Task-centric Latent Actions. RSS, 2025.

[8] Routray et al. ViPRA: Video Prediction for Robot Actions. ICLR, 2026.

[9] Bi et al. Motus: A Unified Latent Action World Model. CVPR, 2026.

[10] Fei et al. LIBERO-Plus: A Progressive Robustness Benchmark for Visual-Language-Action Models. CVPR, 2026.

7 comments
AuthorsOpenReview ↗

Response to Reviewer KM8r [1/2]

We sincerely thank you for your time and effort in reviewing our paper. We appreciate your positive comments on:

  • Video-only task transfer, including demonstrations from a different robot embodiment
  • Policy adaptation by updating only the visual encoder while keeping the action module frozen
  • Our practical use of a large pretrained video backbone to achieve VLA-comparable inference latency without generating future videos

Please find our responses to your comments below.


W1: Novelty of DreamFree

Thank you for highlighting closely related works, Fast-WAM and FLARE.

Compared with Fast-WAM, the key difference is explicit supervision of the visual representation. Fast-WAM uses joint video and action flow-matching objectives. DreamFree instead trains an encoder to predict a frozen video teacher’s representations of noisy future frames from the current observation and instruction alone. This encourages predictive features that support control without future frames at inference.

Our experiments on Robocasa-Kitchen support the importance of this representation supervision:

  • Removing distillation reduces DreamFree's success to 47.2% (Table 3).
  • Scaling the video backbone from 5B to 14B improves DreamFree's success from 58.5% to 71.7%, while Fast-WAM barely improves from 44.4% to 44.7% (Table 2).

These results support the effectiveness of our explicit supervision of the visual representation.

Compared with FLARE, the differences concern both the source of supervision and where it is applied. FLARE aligns features inside the action denoising network with future observation embeddings from a vision-language encoder. On the other hand, DreamFree distills future-informed features from a pretrained video diffusion teacher into a separate visual encoder. We learn these predictive visual representations before action learning, then train the action denoising network with the features from the visual encoder.

  • On RoboCasa-Kitchen experiments (Table 2), DreamFree achieves 71.7% success, compared with 66.4% for FLARE.

W2: Choice of teacher model

Thank you for raising this question. We use a pretrained video diffusion teacher because our goal is to transfer its learned predictive knowledge into features available for control from the current observation.

To examine the teacher choice, we evaluate a variant that distills future representations from a frozen V-JEPA 2 video encoder: we keep the student backbone (14B), training data, and action-learning procedure unchanged.

  • On RoboCasa-Kitchen, this variant achieves 70.0% success, compared with 71.7% using our video diffusion teacher.

These results show that V-JEPA 2 also provides effective supervision within DreamFree, with the video diffusion teacher achieving higher observed success.

We hypothesize that learning to generate future videos helps the diffusion teacher retain information about object positions and motion, making its features more useful for manipulation than those from V-JEPA 2.

We added a discussion of this experiment to Appendix D.4.


W3: Choice of teacher layer for the distillation target

Within the video diffusion teacher, we select the teacher layer jj based on the action probing performance: We evaluate how accurately actions can be predicted from each layer representation and choose a layer with strong performance. The distillation target therefore contains information whose relevance to action prediction has been empirically assessed. This encourages the encoder to capture future information that is useful for control.

  • Our representation analysis (Fig. 3) further shows that the resulting distilled encoder features support more accurate action prediction.

We updated the main text to clarify the layer selection procedure.


W4: Contribution of future information (td=1t_d=1)

Thank you for suggesting this comparison. On RoboCasa-Kitchen, setting td=1t_d=1 yields 69.6% success, compared with 71.7% using td=0.7t_d=0.7. At td=1t_d=1, the teacher’s future frame inputs are pure noise, while its pretrained predictive knowledge and conditioning on the current observation and instruction remain available. These results suggest that transferring the teacher’s pretrained representations accounts for much of the benefit. Providing noisy demonstrated-future frames yields a further observed improvement of 2.1 percentage points. We therefore interpret demonstrated future information as an additional contributor to performance.


(Response continued in the next comment.)

AuthorsOpenReview ↗

Response to Reviewer KM8r [2/2]

W5: Sensitivity to teacher noise level

Thank you for suggesting this analysis. We evaluated teacher noise levels td∈0.5,0.7,1.0t_d \in \\{0.5, 0.7, 1.0\\} on RoboCasa-Kitchen using the 14B video backbone. DreamFree achieves the highest success rate of 71.7% with td=0.7t_d=0.7, compared with 71.4% at td=0.5t_d=0.5 and 69.6% at td=1.0t_d=1.0. Performance varies by 2.1 percentage points across the tested settings, suggesting that DreamFree is robust to the choice of teacher noise level within this range. We chose td=0.7t_d=0.7 because the teacher representations at this noise level achieved strong action probing performance.


W6: Robustness evaluation with LIBERO-Plus

Thank you for suggesting the evaluation. We additionally evaluate DreamFree on LIBERO-Plus.

ModelLIBERO (Standard)CameraRobotLanguageLightBackgroundNoiseLayoutLIBERO Plus Avg.
π0\pi_094.113.86.058.885.081.479.068.953.6
OpenVLA-OFT97.156.431.979.588.793.375.874.269.6
π0.5\pi_{0.5}96.975.477.585.696.994.689.785.785.7
Fast-WAM97.616.444.568.978.253.737.760.751.5
DreamFree98.625.567.579.283.457.235.767.958.3
DreamFree (+extra videos)98.066.255.977.092.385.181.871.774.6

As shown in the table, DreamFree trained on standard LIBERO demonstrations exhibits limited robustness, and training the visual encoder with additional LIBERO-Plus videos, without their action labels, substantially improves the results.

We hypothesize that the limited size and visual diversity of standard LIBERO demonstrations constrain the robustness of DreamFree. Additional videos help the visual encoder adapt to visual variations without additional action supervision.

Baseline results are taken from [1] and [2].

[1] Fei et al., 2025. LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action Models.

[2] Zhang et al., 2026. Do World Action Models Generalize Better than VLAs? A Robustness Study.


W7: Comparison with Cosmos Policy using Cosmos-Predict2.5

Thank you for pointing out this updated baseline. Cosmos Policy using Cosmos-Predict2.5 and DreamFree use the same released RoboCasa demonstration dataset for action learning, but rely on different pretrained video backbones and pretraining data: Cosmos-Predict2.5 for Cosmos Policy and Wan2.2 for DreamFree. Both use the same benchmark with 24 tasks across five evaluation scenes and 50 demonstrations per task.

Cosmos Policy using Cosmos-Predict2.5 reports 71.1% success, comparable to DreamFree’s 71.7%. Incorporating additional RoboCasa365-atomic videos without action labels further improves DreamFree to 74.4%, demonstrating the benefit of additional video training. DreamFree achieves both results without generating future video at inference.


W8: Fast-WAM 14B reproduction

We reproduced Fast-WAM 14B from the official code using the same Wan2.2-I2V-A14B backbone, native image-to-video conditioning, LoRA configuration, and RoboCasa demonstrations as DreamFree 14B. We retained Fast-WAM’s attention structure and joint video–action flow-matching objectives.

The action expert uses 40 layers with approximately 1B parameters, initialized by interpolating video-backbone weights. We jointly trained the video LoRA adapters, action expert, and proprioception encoder for the same number of epochs as DreamFree, with global batch size 32 and learning rate 10−410^{-4}. We added implementation details to Appendix B.1.


W9: Connection to latent action and motion-centric representation learning

Thank you for highlighting these connections. These approaches share our goal of learning useful information for robot control from videos without action annotations. LAPA, Moto, and UniVLA learn latent action or motion representations from visual transitions, while ViPRA and Motus integrate motion-centric latent actions with video modeling.

DreamFree focuses on transferring predictive knowledge already learned by a pretrained video diffusion model. We use its future visual features as supervision so that the encoder learns to anticipate the teacher’s representations from the current observation. This directly reuses the pretrained model’s representation space for control, without requiring a separately learned latent-action space.

Empirically, DreamFree achieves 98.6% average success on LIBERO, compared with the reported 97.7% for Motus and 95.2% for UniVLA, supporting the effectiveness of our approach. We added a discussion of these connections and clarified the motivation for our approach in Appendix E.1.


Please let us know if there are any remaining points we should clarify. We are happy to provide further details.

AuthorsOpenReview ↗

Follow-up on Our Response and Revisions for Reviewer KM8r

Thank you again for your time and valuable feedback on our paper.

In our response and revised manuscript, we:

  • Added teacher-noise ablations, a V-JEPA 2 teacher comparison, and LIBERO-Plus evaluations.
  • Clarified the Fast-WAM 14B reproduction and contextualized the updated Cosmos Policy results.
  • Expanded the discussion of DreamFree’s distinctions from related work and connections to latent action and motion-centric approaches.

As the author–reviewer discussion ends on November 18, we would appreciate hearing whether any concerns remain. We would be happy to provide further clarification before the deadline.

If our responses and revisions have satisfactorily addressed your concerns, we would be grateful if you would consider revising your score upward.

Yours sincerely, Authors

Reviewer_Rqg5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper is motivated by the fact that pretrained video diffusion models are good at video prediction, but require expensive iterative video denoising. The authors present a method to predict visual representation (rather than the video) from the current observation. This representation is then used to generate actions. This is done in two stages. In the first stage, a frozen video diffusion model extracts representations from current and future frames of a video demonstration. A visual encoder is trained to predict these representations from just the current observation. In the second stage, an action module is trained to map these representations to executable robot actions. This results in a world-action model that is called DreamFree. DreamFree demonstrates a 17.8x speedup compared to a WAM that with a similar video backbone, resulting in a latency similar to a VLA. Authors show that this method can achieve higher success rate than other VLAs and WAMs such as pi0.5 and DreamZero.

Critical strengths
  • Validation is presented on RoboCasa simulation and DROID real robot.

  • Latency of a WAM is improved to be similar to that of a pi0.5 VLA.

  • Success rate of the proposed DreamFree is better than the DreamZero WMA and the pi0.5 VLA.

  • Source code is provided to enable reproducibility of the results in the paper.

Critical weaknesses
  • Figures and symbols used in this paper are confusing - see below.

  • Fig. 1 block diagram is not too clear. What is the input to the encoder. What is the output of the action DiT? Why have both a_{1:H} in the top and bottom of the action DiT?

  • Fig. 2 is also not clear. The projector encoder seems to have outputs on both ends (arrows point out). What is the input to the encoder? Top inputs are l, f_1 ... f_h and bottom inputs are f_0 and noisy f_1..f_h.

  • The symbols f_0..f_h only appear in the Figures 1-2. The text doesn't seem to have this!!

  • Figures use f_0, ... f_h symbols for video frames but the text uses x_^0... x_^h which is confusing.

  • In text before Eq. (1) and in Eq. (1) The authors don't mention how i and j are related. The paper mentions h_i is the i-th layer hidden representation of the predictive encoder and j is layer j of the video diffusion model.

  • In Eq. (2), the encoder u_\theta is being trained to predict the entire video x_? First, the encoder is not supposed to generate a video. Even if it does, it should only predict a few frames rather than the entire video x_.

  • In Eq. (4), the same encoder u_\theta from Eq.(2) is now used as u_\phi to predict actions!! This is confusing.

  • It's unclear why the success rate of DreamFree is better than a traditional WAM that predicts video rather than representations. I understand the speedup, but not the increase in success rate. What component of the system enables better success rate?

Questions for authors
  • Clarity on some of points raised in the weaknesses could help improve the presentation of the paper and help readers better understand.
4 comments
AuthorsOpenReview ↗

Response to Reviewer Rqg5

We sincerely thank you for your time and effort in reviewing our paper. We appreciate your positive comments on:

  • Our simulation and real-world evaluations
  • VLA-comparable inference latency
  • Strong performance compared with the evaluated baselines
  • Publicly available source code

Please find our responses to your comments below.


W1: Fig. 1 block diagram

The encoder receives the current observation f0\text{f}_0 and the instruction ℓ\ell. The action chunk a1:Ha_{1:H} below the action DiT is the noisy input, while the action chunk above it is the action prediction obtained through iterative denoising. We updated the figure and the caption to clarify these points.


W2: Fig. 2 clarification

The projector takes the encoder’s current-frame features as input and maps them to the teacher’s future-frame representations. The arrow indicates alignment between the projected features and the teacher representation (lines 169-171 in the revised manuscript).

During encoder training (Fig. 2), the encoder receives the clean current frame latent f0\text{f}_{0}, noisy future frame latents f1:H\text{f}_{1:H}, and the instruction ℓ\ell. The structured attention mask prevents current frame tokens from attending to future frame tokens. Thus, the current frame features depend only on the current observation and the instruction. At inference, future frame tokens are omitted. f1:H\text{f}_{1:H} at the top depicts the denoised frame latents from the velocity field predicted by the video DiT.

We updated the figure to clarify the arrow and the models' input and output.


W3: f0,⋯ ,fH\text{f}_0, \cdots, \text{f}_H and x0,⋯ ,xHx_0, \cdots, x_H

We apologize for the inconsistent notation between the figures and text. We updated the main text to use f0,⋯ ,fH\text{f}_0, \cdots, \text{f}_H for the equation.


W4: Relation of ii and jj

When the encoder and teacher share the same architecture, we align corresponding layers, setting i=ji=j, as described in the appendix. We have updated the paper (lines 176-179) to clarify this and Appendix A.7 to explain this choice.


W5: Regarding the encoder generating video

The encoder is a video DiT initialized from a pretrained video diffusion model, retaining the architecture for video generation. During training, we retain the video model's flow matching objective (Eq. (2)) as an auxiliary loss to preserve its predictive capability. We call it an encoder because, at inference, we omit future-frame tokens and use only the current-frame hidden representations for action prediction.


W6: Prediction target and video-clip length in Eq. (2)

Eq.(2) trains the encoder to predict the velocity field used to denoise future-frame latents within a short video clip. f∗\text{f}_* represents a short video clip containing the current observation and a few future frames, rather than an entire demonstration.


W7: Notation for uθu_{\theta} and uϕu_{\phi}

Thank you for pointing out the notation. uθu_{\theta} and uϕu_{\phi} are two separate networks with independent parameters. uθu_{\theta} denotes the encoder and uϕu_{\phi} denotes the action decoder. We used uu for both models because both predict velocity fields under a flow matching objective. To clarify their roles, we now denote them uθvidu_{\theta}^{\text{vid}} and uϕactu_{\phi}^{\text{act}}, respectively, in the updated manuscript.


W8: success rate of DreamFree is better than a traditional WAM that predicts video rather than representations

Thank you for raising this question. Our explanation is that directly supervising the encoder with teacher features of strong action-probing performance helps it retain information useful for control. Video generation requires modeling visual details that may be less relevant to action selection. Instead, explicit representation supervision encourages the visual representation to capture action-relevant information about future states.

Our experiments identify representation distillation as a substantial contributor to DreamFree’s performance.

  • Removing distillation reduces RoboCasa success from 71.7% to 47.2% (Table 3)
  • Representation analysis shows that distillation improves action decodability (Fig. 3)

These findings demonstrate the benefit of distillation within DreamFree and are consistent with our proposed explanation.


Please let us know if there are any remaining points we should clarify. We are happy to provide further details.

AuthorsOpenReview ↗

Follow-up on Our Response and Revisions for Reviewer Rqg5

Thank you again for your time and valuable feedback on our paper.

In our response and revised manuscript, we:

  • Clarified the figures, notation, and training and inference procedures.
  • Discussed how representation distillation contributes to the performance gains, supported by the distillation ablation and representation analysis.

As the author–reviewer discussion ends on November 18, we would appreciate hearing whether any concerns remain. We would be happy to provide further clarification before the deadline.

If our responses and revisions have satisfactorily addressed your concerns, we would be grateful if you would consider revising your score upward.

Yours sincerely, Authors

AuthorsOpenReview ↗

Follow-up Before the Discussion Deadline for Reviewer Rqg5

Thank you again for your time and valuable feedback.

With the discussion ending on November 18, we would be grateful for an update on whether our responses have addressed your concerns. We would be happy to clarify any remaining questions.

Yours sincerely, Authors

AuthorsOpenReview ↗

Final Follow-up Before the Discussion Closes for Reviewer Rqg5

With less than 24 hours remaining in the discussion period, we would be grateful for an update on whether our responses have addressed your concerns. We would be happy to clarify any remaining questions before the discussion closes.

Thank you again for your time and consideration.

Yours sincerely, Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Thank You for Your Feedback: Summary of Our Revisions

We sincerely thank the reviewers for their careful evaluation and constructive feedback. We appreciate their recognition of the following strengths:

  • Broad evaluation: Strong empirical results supported by simulation, real-world experiments, and distillation analyses.
  • Efficient inference: VLA-comparable latency without generating future videos.
  • Video-only transfer: Transfer across tasks and robot embodiments, with policy adaptation through visual-encoder updates alone.
  • Reproducibility: Publicly available source code.

Our responses clarify DreamFree’s central contribution: transferring a pretrained video diffusion model’s predictive knowledge into current-observation representations through future representation distillation, enabling control without future video generation.


In response to the reviewers’ feedback, we have clarified the presentation, expanded the experimental analysis, and strengthened the discussion of related work. The main updates are listed below, with all changes highlighted in orange in the revised manuscript:

  • Fig. 1: Clarified the model inputs and outputs and explained aa and f\text{f} in the caption.

  • Fig. 2: Clarified the model inputs and outputs, corrected the arrow direction, and explained the colored squares.

  • Sections 3.1 and 3.2:

    • Clarified how the encoder repurposes a pretrained video diffusion model.
    • Standardized the notation to use f\text{f} consistently with the figures and distinguish the video model uθvidu^{\text{vid}}_{\theta} from the action model uϕactu^{\text{act}}_{\phi}.
    • Explained the relationship between layer indices ii and jj and the procedure for selecting teacher layer jj.
  • Appendix:

    • A.7: Added details on action and future probing.
    • B.1: Expanded the implementation details for Fast-WAM 14B.
    • D.3: Added an ablation on the teacher noise level tdt_d.
    • D.4: Added an ablation using a pretrained video encoder as an alternative visual teacher.
    • E.1: Clarified the technical differences from REPA, V-JEPA 2, and FLARE, and discussed connections to latent action and motion-centric representation learning.

Thank you again for your thorough review and thoughtful suggestions. We hope our responses have addressed your concerns, and we are willing to address any further questions you may have.

Yours sincerely,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.