acceptodds
Under review as a conference paper at ICLR 2027

Motus2: A Self-Evolving General World Model for Dexterous Manipulation

12 views

Abstract

General embodied agents should perceive, predict, act, evaluate, and improve within a unified system. World models have shown great promise in building such agents, yet existing models typically append an action output head to a world simulator, without coupling them into a closed decision-and-learning loop for policy improvement. We present Motus2, a self-evolving general world model for dexterous manipulation. Motus2 advances world modeling through model scaling and data scaling. For model scaling, a single model with shared weights exposes three control interfaces: a policy (world-action model), a simulator (action-conditioned world model), and an evaluator (value model). The policy proposes candidate action chunks, the simulator predicts their visual consequences, and the evaluator assesses the predicted outcomes. Their coupling forms a closed decision-and-learning loop for policy improvement. This formulation uses curated expert demonstrations for action learning, while failed and suboptimal interactions provide valuable evidence for dynamics modeling and value learning. For data scaling, Motus2 progresses from large-scale monocular egocentric data to synchronized stereo egocentric data, followed by robot-domain adaptation with robot trajectories and supplementary human-robot alignment data. Motus2 further studies global-autoregressive and hybrid-memory extensions of its sliding-window context, adds tactile feedback for contact-aware control, and is instantiated on a fully biomimetic platform with stereo vision, dual arms, dual dexterous hands, and tactile sensing. Together, egocentric data scaling and closed-loop general world model scaling provide a general path toward self-evolving dexterous manipulation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
General World Models, Embodied Foundation Models, Dextrous Manipulation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_yhDLOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors present a world model for dexterous manipulation that couples action generation, future prediction, and task-progress evaluation within a single model. This world model consists of three components: a policy, a simulator (action-conditioned video world model), and an evaluator (essentially a value function). The evaluator's scores are used for test-time planning and for policy optimization. They also present a curriculum to train the world model, progressing from monocular to stereo egocentric human data, then to robot teleoperation data, and finally to target-robot post-training. They showcase their contributions by pretraining on ~130K hours of egocentric human data and finetuning on a dual-arm, dexterous-hand robot for five real-world manipulation tasks. Their results show Motus2 outperforms π0.5\pi_{0.5} and Motus2 variants with different training setups. They also include results that evaluate the planning and policy optimization loop, two long-horizon memory mechanisms, and a tactile expert in separate studies, each on two tasks.

Critical strengths
  1. The authors show success on five real-world manipulation tasks.
  2. The authors make their system work at a large scale with real data, i.e., ~130K raw hours of human data and over 100 hours of robot data. This is a substantial engineering effort. Moreover, they provide a scaling trend for stereo egocentric human data, where they show that human-action prediction error decreases approximately log-linearly with the amount of data.
Critical weaknesses

The novelty is unclear. Some of the writing implies that the novelty comes from "data scaling" and "model scaling." The claim on data scaling is never backed by concrete comparisons to how much data previous work used. The claim on model scaling points to their factorized architecture, which is neither novel (e.g., TD-MPC2 [Hansen et al., 2024]) nor consistent with the conventional meaning of model scaling, i.e., more parameters.

If the novelty is the closed loop between policy, simulator, and evaluator, there is prior work that uses a learned world model to improve policy performance, e.g., Dreamer [Hafner et al., 2023] and TD-MPC2 [Hansen et al., 2024].

If the novelty is the curriculum, I'm unconvinced. In Section 2.2, the authors claim that "the ordering matters," but never provide any evidence for this. I don't see why the whole model could not be trained at once on egocentric and/or robot data using the loss in Eq. (4).

The writing is often hard to follow. It often jumps straight into jargon without any context, e.g., in Section 2.2: "Pre-training learns joint video–action representations with mutual within-chunk visibility and no value query." It is not clear what "within-chunk visibility" or a "value query" are at this point in the paper.

Also, the use of the word "self-evolution" in the paper is not very precise. During policy optimization, only the action-related parameters are updated, while everything else in the world model remains frozen, so it is unclear in what sense the whole system self-evolves beyond finetuning the policy against the frozen evaluator.

The experimental results are also suspect. π0.5\pi_{0.5} achieves 0% success on every task in Table 1, which makes me question whether the baseline was set up properly.

Questions for authors
  1. Can you elucidate why π0.5\pi_{0.5} performs so poorly, i.e., 0% on every task? What is it about these tasks or this setup that makes π0.5\pi_{0.5} fail completely? This would help me believe the baseline more.
  2. What prevents you from training the whole model at once on the egocentric and robot data, using the loss in Eq. (4)? Section 2.2 states that "the ordering matters," but I don't see an experiment supporting this. An ablation against single-stage training would help justify the curriculum as a contribution.
  3. How is your approach different from existing approaches that use a learned world model and value function to improve a policy, e.g., Dreamer or TD-MPC2? Clarifying this would help me understand the novelty of your contribution.
10 comments
AuthorsOpenReview ↗

R1. Core contribution and relationship to prior world-model methods (Novelty concerns; Question 3).

Thank you for your questions about the contribution, training curriculum, and terminology. We address these points below.

The central contribution of Motus2 is to extend unified video–action modeling to policy learning driven by the model’s own predictions and evaluations. The research question is how to organize action generation, consequence prediction, and evaluation within the same generative model so that these capabilities support both decision-making and improvement of the action-generation policy. Dreamer trains an actor–critic using latent trajectories imagined by a world model. TD-MPC2 uses learned latent dynamics, rewards, and values for trajectory optimization. Motus2 establishes the conditional relationships among action generation, visual consequence prediction, and value evaluation within a shared generative video–action model, using an action-first information flow and trajectory-dependent supervision routing. Specifically, the policy interface independently proposes candidate actions, the simulator interface predicts their visual consequences, and the evaluator interface assesses task progress. The action-first attention mask prevents future information from leaking into action generation. Supervision routing assigns appropriate objectives to successful, failed, and suboptimal interactions, allowing the model to learn unsuccessful outcomes while avoiding imitation of unsuccessful actions. The model’s own evaluation feedback then updates the action policy through DiffusionNFT, improving both action selection and subsequent action generation. As a related unified modeling approach, Cosmos Policy uses action, future-state, and value prediction for Best-of-N planning. Motus2 further uses prediction and evaluation feedback to update the parameters of the action interface, connecting the unified model’s decision-making capabilities to policy learning. The contribution we emphasize is this structural and training extension of a unified model, together with the resulting self-improvement loop that can be trained and deployed.

AuthorsOpenReview ↗

R2. Meaning of “model scaling” and “self-evolution” (Terminology concerns).

In this paper, model scaling refers to expanding the unified model’s functional and modeling scope: from video–action modeling to action generation, consequence prediction, and value evaluation, enabling the three interfaces to jointly support decision-making and policy learning. Self-evolution refers to improving the action-generation policy using the model’s own predictions and evaluations. The policy interface generates candidates, the simulator predicts their consequences, and the evaluator supplies task-progress feedback. This feedback supports both action selection and policy updates. During policy optimization, the simulator and evaluator remain frozen, and only action-related parameters are updated. All tasks use 1k MBRL policy-optimization updates.

AuthorsOpenReview ↗

R3. Rationale for staged training (Curriculum concerns; Question 2).

Our training stages address three successive transfer objectives: learning general interaction priors from egocentric human data, aligning these priors with robot observations and actions, and adapting them to the target task. This arrangement follows staged transfer recipes in human-to-robot learning. EgoScale uses human pretraining, aligned human–robot mid-training, and task-specific post-training. Motus progresses from video-model training to latent-action pretraining and target-robot SFT. EgoVLA learns from egocentric human demonstrations before robot fine-tuning. In Motus2, the downstream initialization comparison measures the benefits of human pretraining and robot-domain adaptation. Under the same target-task SFT data and procedure, WAN-SFT, Pretrain-SFT, and Midtrain-SFT achieve average success rates of 0%, 51%, and 84%, respectively. These results demonstrate the contributions of general interaction priors and subsequent robot-domain adaptation to target-task control.

Reviewer_WsAuOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work presents Motus2, a general world model that combines a policy, a simulator, and a value model through a mixture of transformers and an action-first attention mask to support self-evolution. Multiple candidate action chunks generated in policy mode can then be used in simulation mode to predict the corresponding future visual latents. In evaluation mode, the value model scores these action-future pairs to support Best-of-N selection of action chunks or DiffusionNFT-style policy optimization. For long-horizon tasks, the hybrid-memory variant conditions action generation on both a recent observation window and memory tokens representing earlier history. A tactile-conditioned action refinement expert can also be integrated for contact-rich tasks. For data scaling, the two-stage curriculum begins with video-only pretraining on monocular egocentric data, moving from low to high resolution, and then introduces joint video-action pretraining on synchronized stereo data. Robot-domain mid-training then incorporates robot trajectories and supplementary human-robot alignment data. From this stage onward, trajectory-dependent supervision routing assigns different types of interactions to the appropriate training modes, while subsequent MBRL post-training uses value-guided policy updates to support self-evolution.

Critical strengths
  1. The proposed method unifies action generation, action-conditioned simulation, and outcome evaluation through shared parameters and an action-first mask. Its supervision design also uses unsuccessful interactions for dynamics and value learning, providing a coherent foundation for policy improvement from predicted consequences.
  2. The proposed method develops a structured data-scaling curriculum from monocular egocentric video to stereo video paired with human actions, followed by robot-domain adaptation with robot trajectories and human-robot alignment data. These stages connect broad visual priors, spatial and hand-motion information, and executable robot control.
  3. The proposed method connects Best-of-N planning and DiffusionNFT-based MBRL through the same branch-evaluation mechanism. The resulting scores support both selecting current actions and improving future action proposals, giving self-evolution a concrete decision-and-learning process.
  4. The proposed method accommodates visual history and tactile feedback through complementary extensions to context retention and action refinement. These extensions address delayed visual evidence and ambiguous contact states, providing flexible ways to support different information requirements in dexterous manipulation.
  5. Under the same target-task SFT procedure, initialization comparisons report 51% average success after egocentric pretraining and 84% after robot-domain mid-training. Global autoregression outperforms hybrid memory in both simulation and real-robot tests, while tactile feedback improves success on both contact-rich tasks. These experiments provide concrete support for the training curriculum and the context and sensing extensions.
Critical weaknesses
  1. The self-evolution gains have limited statistical support. MBRL and planning are evaluated on only two tasks with 20 rollouts per task. Planning provides a net gain of one successful rollout across 40 evaluations, while MBRL provides three. Without repeated training runs, paired outcome analysis, or uncertainty estimates, the reliability of these improvements remains unclear.
  2. The evalator’s reliability on generated futures is not established. The qualitative value curves use recorded observations, without direct quantitative validation that scores on generated futures correctly rank candidate actions according to their actual execution outcomes. This leaves the feedback supporting planning and policy optimization insufficiently verified.
  3. The sources of improvement are not sufficiently isolated. The benefit of parameter sharing is not established through a separate-module comparison. MBRL also includes ground-truth action anchors without continued-SFT or shuffled-score controls, leaving the contributions of additional supervised exposure and informative evaluator feedback unclear.
  4. The proposed method lacks a compute-performance evaluation across planning candidate counts and visual-history lengths. The latency, computational cost, and memory required for the reported performance gains therefore remain unclear.
  5. Within-task real-robot generalization is insufficiently characterized. Although 20 initial configurations are evaluated per task, the separation between training and evaluation object instances, poses, and scene layouts is unspecified, and performance under controlled held-out variations is not separately reported.
Questions for authors
  1. Can you report repeated training runs, paired rollout outcomes, and uncertainty estimates for Table 2 to establish the reliability of the MBRL and planning gains?
  2. Can you quantitatively evaluate whether scores on generated futures correctly rank candidate actions from the same prefix according to actual execution outcomes? A comparison with rankings obtained from recorded futures would help assess the evaluator’s reliability.
  3. Can you isolate the benefit of parameter sharing through a separate-module comparison under comparable data and training budgets? Can you also compare MBRL with continued-SFT and shuffled-score controls while matching trainable parameters, policy-update budgets, and ground-truth exposure?
  4. Can you report success rate, end-to-end latency, computational cost, and peak memory across planning candidate counts and visual-history lengths on fixed hardware?
  5. Can you clarify which object instances, poses, and scene layouts are held out from training, and report performance under controlled within-task variations?
7 comments

Thank you for recognizing the unified formulation and the connection between action selection and policy learning. We address the additional optimization controls, parameter sharing, computational cost, and generalization evaluation below.

AuthorsOpenReview ↗

R1. Additional MBRL controls (Critical Weaknesses 1 and 3; related parts of Questions 1 and 3).

The original tasks already have relatively high SFT success rates, leaving less room for further improvement. To examine policy optimization more clearly, we added the more challenging opening-a-cola-can task, on which the initial SFT success rate is 25%. On both tasks, continued-SFT and MBRL reuse the existing data, with no newly collected data introduced. Continued-SFT uses 5k additional supervised updates, while MBRL uses 1k policy-optimization updates, with the same MBRL configuration across tasks.

TaskSFTContinued-SFTMBRLShuffled-score
Multi-object grasping14/20 (70%)16/20 (80%)17/20 (85%)14/20 (70%)
Opening a cola can5/20 (25%)9/20 (45%)11/20 (55%)7/20 (35%)

On opening a cola can, MBRL increases success from 25% to 55%, compared with 45% for continued-SFT and 35% for shuffled-score. On multi-object grasping, MBRL achieves 85%, compared with 80% for continued-SFT and 70% for shuffled-score. Continued-SFT examines the benefit of further supervised training on existing demonstrations. The shuffled-score control permutes evaluation scores among policy candidates from the same history prefix, examining the contribution of informative evaluator feedback. MBRL achieves the highest observed success count on both tasks.

AuthorsOpenReview ↗

R2. Role of parameter sharing (Critical Weakness 3; parameter-sharing component of Question 3).

Parameter sharing first supports a decision-and-learning loop within the unified model. The policy, simulator, and evaluator interfaces use the same backbone for action generation, consequence prediction, and progress evaluation. Their evaluations then guide action selection and policy updates, allowing the unified model’s predictive and evaluative capabilities to participate in its own policy learning. Second, the shared backbone allows supervision from different interactions to accumulate in a common representation. Dynamics learning teaches how actions change the environment, while value learning provides supervision about how those changes advance the task. Both shape the representation used by all three interfaces. A successful trajectory contains imitation targets, environmental transitions, and task-progress information. Failed and suboptimal trajectories also provide useful dynamics and outcome supervision. Through trajectory-dependent supervision routing, a trajectory that is unsuitable for action imitation can still train the shared backbone through its dynamics and outcome information. All three interfaces can therefore draw on the knowledge accumulated from these interactions. Integrating the value interface into the backbone both reuses its learned dynamics representations for evaluation and allows task-progress supervision to shape the common representation. Related studies support this design rationale: VLAC demonstrates benefits of task-progress learning for action generation; the ablations in Cosmos Policy demonstrate benefits of joint prediction objectives for policy performance; and World Value Models demonstrate the utility of world-model temporal representations for value estimation. Motus2 organizes these complementary signals within a unified video–action backbone and further uses its evaluation feedback for policy optimization.

Reviewer_TFfyOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Motus2 proposes a shared-parameter world model coupling a policy generating action chunks, an action conditioned simulator predicting future videos, and an evaluator scoring predicted outcomes as task progress. This model can be trained on multi-source data, including expert demonstrations for action learning, and failed and suboptimal interaction data are used to train the simulator and evaluator. Training uses egocentric data for pertaining, robot domain data for mid training, then target robot post training. The model also adds long history context and a tactile expert as post trained branches. Results on nine real-world tasks on three bimanual dexterous platforms demonstrate the effectiveness of the system.

Critical strengths
  • The system is very complete and comprehensive, from large-scale data curation, multi-stage training pipeline, to incorporating tactile refinement and demonstrating nine real-world tasks across three bimanual dexterous platforms.
  • The result is strong, showing clear scaling laws of stereo egocentric human data for pretraining, and significantly increased real-world success rates compared to baseline methods.
Critical weaknesses
  • Action generation, future prediction, and progress evaluation are all existing formulations, and the paper assembles them rather than introducing them. Motus already realizes the policy and simulator interfaces, the value target is VLAC's relative progress formulation, the optimizer is DiffusionNFT, the hybrid memory is MemoryWAM, the tactile expert follows T-Rex, the backbone is Wan 2.2, and the asynchronous pipeline follows AcceRL. The new algorithm elements are the action first mask and the supervision routing, which should be clarified in the writing and not over-claimed.
  • The paper lacks analysis of each components. One concern I have is the evaluator scores predicted video rather than real outcomes, and DiffusionNFT then moves the policy toward whatever it rates highly, but what if the simulator predicts "false positives", i.e., due to not sufficient coverage of failure data, the simulator may overfit to success and generates videos that look successful even though the action wouldn't lead to success in the real world. More analysis such as simulator prediction error, and comparison of values on imagined versus real-world execution rollouts.
Questions for authors
  • Will the mode, code, and data be open-sourced?
  • The evaluation protocol should be specified in more detail, including what varies across the 20 configurations per task, whether those configurations are seen or unseen in the SFT data, why each task is chosen, what challenges it has and what capabilities it tests.
  • The real robot results only report success rate, more failure mode analysis of different methods and summarizing findings from them would better support the paper's claims.
  • Ideally the scaling law can also be validated with task success rates rather than validation loss.
  • What's the inference cost of best of N planning and achieved closed-loop control rate? 1x policy rollout videos would also help evaluate overall system performance.
  • Have you seen compounding error in the loop? For example video generation error coupled with evaluator value prediction error, then causing planning error.
8 comments

Thank you for recognizing the completeness of our system and its real-world manipulation results. We address the contribution, evaluation protocol, data scale, inference cost, and release plan below.

AuthorsOpenReview ↗

R1. Core contribution and attribution (Critical Weakness 1).

We agree with your attribution of the components. Relative to Motus, the two specific designs we introduce are the action-first attention mask and trajectory-dependent supervision routing. The mask lets the model generate actions before predicting and evaluating their consequences. Supervision routing lets it learn from the consequences of failed actions without imitating them. Our system contribution is to combine these designs with existing progress targets and policy optimization methods so that the model’s own predictions and scores can improve its action policy.

AuthorsOpenReview ↗

R2. Evaluation protocol, task selection, and generalization (Question 2).

Each task uses 20 initial configurations that vary object position, orientation, scene layout, and background. The evaluation objects also appear in SFT, and training and evaluation layouts can overlap. We therefore do not characterize all 20 configurations as unseen. Each method is evaluated once on each of the same configurations. Place Ball and Put Phone test localization and placement, Attach Eraser and Screw Bulb test precise alignment and sustained contact, and Multi-Finger tests finger coordination. The two memory tasks require acting on earlier cues, while the two tactile tasks probe grasp adjustment and sustained bimanual coordination, respectively.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.