- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors present a world model for dexterous manipulation that couples action generation, future prediction, and task-progress evaluation within a single model. This world model consists of three components: a policy, a simulator (action-conditioned video world model), and an evaluator (essentially a value function). The evaluator's scores are used for test-time planning and for policy optimization. They also present a curriculum to train the world model, progressing from monocular to stereo egocentric human data, then to robot teleoperation data, and finally to target-robot post-training. They showcase their contributions by pretraining on ~130K hours of egocentric human data and finetuning on a dual-arm, dexterous-hand robot for five real-world manipulation tasks. Their results show Motus2 outperforms and Motus2 variants with different training setups. They also include results that evaluate the planning and policy optimization loop, two long-horizon memory mechanisms, and a tactile expert in separate studies, each on two tasks.
Critical strengths
- The authors show success on five real-world manipulation tasks.
- The authors make their system work at a large scale with real data, i.e., ~130K raw hours of human data and over 100 hours of robot data. This is a substantial engineering effort. Moreover, they provide a scaling trend for stereo egocentric human data, where they show that human-action prediction error decreases approximately log-linearly with the amount of data.
Critical weaknesses
The novelty is unclear. Some of the writing implies that the novelty comes from "data scaling" and "model scaling." The claim on data scaling is never backed by concrete comparisons to how much data previous work used. The claim on model scaling points to their factorized architecture, which is neither novel (e.g., TD-MPC2 [Hansen et al., 2024]) nor consistent with the conventional meaning of model scaling, i.e., more parameters.
If the novelty is the closed loop between policy, simulator, and evaluator, there is prior work that uses a learned world model to improve policy performance, e.g., Dreamer [Hafner et al., 2023] and TD-MPC2 [Hansen et al., 2024].
If the novelty is the curriculum, I'm unconvinced. In Section 2.2, the authors claim that "the ordering matters," but never provide any evidence for this. I don't see why the whole model could not be trained at once on egocentric and/or robot data using the loss in Eq. (4).
The writing is often hard to follow. It often jumps straight into jargon without any context, e.g., in Section 2.2: "Pre-training learns joint video–action representations with mutual within-chunk visibility and no value query." It is not clear what "within-chunk visibility" or a "value query" are at this point in the paper.
Also, the use of the word "self-evolution" in the paper is not very precise. During policy optimization, only the action-related parameters are updated, while everything else in the world model remains frozen, so it is unclear in what sense the whole system self-evolves beyond finetuning the policy against the frozen evaluator.
The experimental results are also suspect. achieves 0% success on every task in Table 1, which makes me question whether the baseline was set up properly.
Questions for authors
- Can you elucidate why performs so poorly, i.e., 0% on every task? What is it about these tasks or this setup that makes fail completely? This would help me believe the baseline more.
- What prevents you from training the whole model at once on the egocentric and robot data, using the loss in Eq. (4)? Section 2.2 states that "the ordering matters," but I don't see an experiment supporting this. An ablation against single-stage training would help justify the curriculum as a contribution.
- How is your approach different from existing approaches that use a learned world model and value function to improve a policy, e.g., Dreamer or TD-MPC2? Clarifying this would help me understand the novelty of your contribution.
10 comments
R2. Meaning of “model scaling” and “self-evolution” (Terminology concerns).
In this paper, model scaling refers to expanding the unified model’s functional and modeling scope: from video–action modeling to action generation, consequence prediction, and value evaluation, enabling the three interfaces to jointly support decision-making and policy learning. Self-evolution refers to improving the action-generation policy using the model’s own predictions and evaluations. The policy interface generates candidates, the simulator predicts their consequences, and the evaluator supplies task-progress feedback. This feedback supports both action selection and policy updates. During policy optimization, the simulator and evaluator remain frozen, and only action-related parameters are updated. All tasks use 1k MBRL policy-optimization updates.
R3. Rationale for staged training (Curriculum concerns; Question 2).
Our training stages address three successive transfer objectives: learning general interaction priors from egocentric human data, aligning these priors with robot observations and actions, and adapting them to the target task. This arrangement follows staged transfer recipes in human-to-robot learning. EgoScale uses human pretraining, aligned human–robot mid-training, and task-specific post-training. Motus progresses from video-model training to latent-action pretraining and target-robot SFT. EgoVLA learns from egocentric human demonstrations before robot fine-tuning. In Motus2, the downstream initialization comparison measures the benefits of human pretraining and robot-domain adaptation. Under the same target-task SFT data and procedure, WAN-SFT, Pretrain-SFT, and Midtrain-SFT achieve average success rates of 0%, 51%, and 84%, respectively. These results demonstrate the contributions of general interaction priors and subsequent robot-domain adaptation to target-task control.
R1. Core contribution and relationship to prior world-model methods (Novelty concerns; Question 3).
Thank you for your questions about the contribution, training curriculum, and terminology. We address these points below.
The central contribution of Motus2 is to extend unified video–action modeling to policy learning driven by the model’s own predictions and evaluations. The research question is how to organize action generation, consequence prediction, and evaluation within the same generative model so that these capabilities support both decision-making and improvement of the action-generation policy. Dreamer trains an actor–critic using latent trajectories imagined by a world model. TD-MPC2 uses learned latent dynamics, rewards, and values for trajectory optimization. Motus2 establishes the conditional relationships among action generation, visual consequence prediction, and value evaluation within a shared generative video–action model, using an action-first information flow and trajectory-dependent supervision routing. Specifically, the policy interface independently proposes candidate actions, the simulator interface predicts their visual consequences, and the evaluator interface assesses task progress. The action-first attention mask prevents future information from leaking into action generation. Supervision routing assigns appropriate objectives to successful, failed, and suboptimal interactions, allowing the model to learn unsuccessful outcomes while avoiding imitation of unsuccessful actions. The model’s own evaluation feedback then updates the action policy through DiffusionNFT, improving both action selection and subsequent action generation. As a related unified modeling approach, Cosmos Policy uses action, future-state, and value prediction for Best-of-N planning. Motus2 further uses prediction and evaluation feedback to update the parameters of the action interface, connecting the unified model’s decision-making capabilities to policy learning. The contribution we emphasize is this structural and training extension of a unified model, together with the resulting self-improvement loop that can be trained and deployed.