acceptodds
Under review as a conference paper at ICLR 2027

MM-World: A Unified Video World Model for Egocentric Navigation and Manipulation

Abstract

Action-conditioned video world models enable users to explore and interact with virtual worlds through their own actions. However, existing models typically focus on either navigation or local manipulation, leaving their combination poorly supported, even though most real-world tasks require moving and manipulating in tandem. To address this, we present MM-World, a unified video world model for egocentric navigation and manipulation controlled by head and hand motion. Our key insight is to express these heterogeneous action-conditioning signals in a shared image space. We reproject scene geometry reconstructed from the initial image along the head trajectory and overlay hands mesh rendered in their prescribed poses. The resulting visual condition reuses the pretrained model's native visual-conditioning pathway without action-specific input modules, while providing a shared interface for joint training on navigation and manipulation videos. We further adopt an autoregressive generation scheme to support long-horizon interactions. Moreover, to evaluate these capabilities comprehensively, we curate a unified dataset and establish a four-track benchmark with 846 test cases spanning navigation, manipulation, mobile manipulation, and long-horizon generation. Extensive experiments show that MM-World outperforms previous methods by a large margin across visual quality and action-following.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
video world model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_5cLHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MM-World generates egocentric video from one image, a static scene caption, and head and hand trajectories. Head motion is rendered by lifting the first frame to a point cloud with Pi3X depth, with hand pixels and low-confidence points removed, and reprojecting it along the camera path. Hand motion is rendered as WiLoR-estimated MANO meshes pasted over the scene render. The composite and its mask enter Wan2.1-I2V-14B through the backbone's existing 36-channel input, and SCoPE adds Plücker rays to every attention block. Only LoRA and SCoPE are trained, on 2.52M clips from DL3DV, EgoDex, EgoVerse and the in-house KAI-Ego. A LongLive-style distilled autoregressive variant handles long sequences.

The paper also builds an 846-case benchmark: navigation (156 DL3DV clips), manipulation in three tiers (623 clips), mobile manipulation (30 KAI-Ego clips) and long-horizon generation (37 sequences). Against released checkpoints run through their own interfaces, MM-World is best on almost every short-horizon metric. A matched-training ablation shows that the scene render is what carries head motion.

Critical strengths
  1. The conditioning design is simple but efficiant, and leaves the backbone unchanged. Both actions end up in the 16+4+16-channel input that Wan2.1-I2V already uses for its conditioning video and mask, no input layer is added, and navigation clips use the same format with an empty hand render. The paper handles the details that matter in practice: it removes first-frame hand pixels so hands are not frozen into the scene, keeps only confident depth, and randomizes the SCoPE gate against scale error.

  2. The short-horizon results improve a lot and come from explicit held-out splits where it matters most. On navigation, MM-World beats eight camera-controlled world models that all receive the trajectory. On mobile manipulation, PSNR is 5–6 dB higher and MPJPE 33–43% lower than the strongest baseline. The supplementary video, including control from a real user's headset and hand tracking, matches these numbers.

  3. The render ablation is controlled and reported honestly. Four models share initialization, seed, data order and step count and differ only in how many first-frame points they keep. The render carries head motion (DL3DV RotErr 6.37° vs. 33.16° with rays only, PSNR 15.24 vs. 11.96 dB), hand accuracy does not depend on it (KAI-Ego MPJPE 8.14 vs. 7.97), halving the points costs little, and mixing render-free clips into training cuts the failure without a render from 99.2° to 54.1° (Tables 4, 12). The authors also report a second checkpoint, note that the High/Low/Random ranking changes between checkpoints, and give the oracle-depth gap (14.50 to 15.46 dB, Table 14).

Critical weaknesses
  1. The advance over the closest render-conditioned methods is not clearly established. The paper overstates what is new relative to EgoSim and DWM. The stated key insight (L84–90) is to express head and hand actions as one image-space render. DWM already conditions a video diffusion model on static-scene renders along a camera trajectory plus egocentric hand-mesh renders, and trains it for joint locomotion and manipulation. EgoSim combines a first-frame point-cloud render, projected hand keypoints and a validity mask. What is new is the MANO-mesh overlay on a confidence-filtered first-frame cloud through the backbone's native input, co-training with navigation video, the ray conditioning, and the scale of the system. That is a real system contribution, but the paper should be positioned at that level. Table 2 shows a large system-level gain over EgoSim (EgoDex PSNR 29.85 vs. 20.34 dB), and the render ablation supports the value of scene rendering. These results do not isolate the contribution of MM-World's added design choices alongside differences in data, training and inference configuration (see Q1).

  2. The navigation advantage is not established against the closest render-conditioned methods. Sec. 5.2 (L363–365) explains the PSNR gain by first-frame surfaces reaching each target view as pixel-aligned renders, whereas the baselines must regenerate them. GEN3C (CVPR 2025) and ViewCrafter (2024), both cited at L122 and both with public code, condition on exactly such single-image point-cloud renders along a camera path. Without one of them, Table 1 shows that MM-World beats ray- and action-conditioned world models, but not that it improves on render-conditioned camera control (see Q2).

  3. The manipulation comparison does not separate the method from in-domain training. MM-World trains on 2.46M manipulation clips from the same three sources as the test tiers, while every baseline is a released checkpoint trained elsewhere. C.1 states benchmark clips are held out (lines 881–882), and D.1 states the split unit only for DL3DV (scene) and the mobile track (parent sequence). For the three manipulation tiers it describes how test clips were picked, but not whether those groups, episodes or demonstrators are excluded from training. KAI-Ego alone contributes 1.08M training clips cut from 139K recordings. As it stands, the 5–6 dB margins in Table 2 measure system performance on the authors' own data sources, not the effect of the conditioning design. (see Q3)

  4. The manipulation RotErr comparison needs a check for alignment-induced offsets. On the manipulation tiers, RotErr mostly measures the Umeyama alignment rather than camera following. In Table 8b, five of six baselines have RotErr of 87–109° on EgoDex and KAI-Ego, but rotational RPE of only 0.16–0.27° per frame. For an 81-frame sequence, 80 such transitions would accumulate only 13–22° of rotation discrepancy. The gap may suggest a large initial offset, which needs direct measurement on a common valid set. Camera following can be checked on the existing outputs by fixing each trajectory's coordinates to its first camera pose, comparing generated orientations against the supplied camera controls for subsequent frames, and reporting both mean angular error and rotational RPE at short and longer time intervals (see Q4).

  5. The AR training horizon and actual evaluated horizon are unclear. Tables 10–11 report AR results. The missing information is how these lengths relate to training. Stage 1 gives no teacher-forcing video length; Stage 2 describes an initial 5s-window and later windows such as 5–10s and 10–15s, but gives no maximum streaming rollout or number of windows. The 21-latent-frame attention window is a local context, not the total training horizon. Thus, the tables do not establish whether the model is tested beyond its training horizon. Please give the training lengths and final AR checkpoint configuration. Also, D.3 ends evaluation when either video ends, while D.2 keeps native frame rates and rounds baseline lengths to valid chunks. The nominal reference durations do not identify the actual generated and scored duration; frame counts, rates and timestamp matching are needed.

Questions for authors

The short-horizon results are strong, the render ablation is controlled, and I view the system positively. I would raise the score if the response clarifies the added contribution and provides convincing evidence, or a justified boundary, for navigation against the closest rendering-based methods.

  1. What is the added methodological contribution over DWM and EgoSim, and which existing comparisons or ablations support its benefit?
  2. What evidence establishes a navigation advantage over the closest render-conditioned camera methods, such as GEN3C and ViewCrafter? If they were excluded, why, and what navigation claim remains supported by the current comparisons?
  3. How do joint head/hand motion and scene familiarity in training compare with the mobile test clips? What evidence supports composition beyond training and an actual contribution from navigation co-training?
  4. After recomputing camera following in coordinates fixed by each trajectory's first pose, do the main rotation and translation gains remain? Please use the protocol and coverage checks in W4 to distinguish control error from alignment offsets.
  5. What are the video lengths in Stage 1 and the maximum streaming rollout in Stage 2 for the final AR checkpoint? Do the actual long-horizon evaluations exceed that training horizon?
  6. As generation length increases, do object states, contact and the overall manipulation remain coherent?
9 comments
AuthorsOpenReview ↗

Response to Reviewer 5cLH (1/6)

W1 & Q1: Contribution, System Comparison, and Component Ablations

Thank you for suggesting that we clarify our contribution and the sources of our gains. We agree that MM-World’s primary contribution is its system design and integration for single-image egocentric mobile manipulation: unified visual conditioning and a shared training interface let one model learn viewpoint changes and hand interactions jointly from existing navigation and manipulation data.

1. Differences from DWM and EgoSim, and system comparison. DWM assumes a static 3D scene and conditions on trajectory scene renders plus hand-mesh renders, so scene appearance and view changes come mainly from rendering existing 3D assets. MM-World instead estimates a partial point cloud from a single initial image at inference, generating both hand interactions and newly revealed regions under camera motion.

EgoSim renders a first-frame point cloud, projected hand keypoints, and a validity mask, encoding scene and hand separately before concatenation. MM-World uses MANO meshes and composites scene and hand renders into one visual condition fed through the backbone’s native visual pathway; navigation samples simply omit the hand render under the same format and objective.

For the system comparison between EgoSim and MM-World, we sampled 10K clips each from EgoDex, KAI-Ego, and EgoVerse (30K total), initialized both from their respective Wan2.1 14B backbones, and trained for 3,000 steps with the same data and seed, adapting EgoSim to MM-World’s scene point-cloud rendering pipeline to align geometric scale:

DatasetMethodPSNR↑SSIM↑LPIPS↓RotErr↓ATE↓RPE rot↓RPE trans↓Flow-EPE↓FVD↓ROI-LPIPS↓RA-MPJPE↓PA-MPJPE↓PA-MPVPE↓
EgoDexEgoSim28.41190.90050.094146.99520.02460.14470.00490.654587.39620.176712.98285.33065.0080
MM-World29.15850.90730.084747.77620.02390.13410.00470.599175.03400.152611.05204.73764.4521
KAI-EgoEgoSim24.03020.81550.137338.82520.02930.11350.00500.5338113.28350.17608.51793.18733.0339
MM-World25.15980.83170.109534.17140.02880.10120.00480.472699.00960.14337.10482.82752.6812
EgoVerseEgoSim20.40380.71870.208425.80480.05940.29310.01161.6816152.38770.265916.36815.85405.5211
MM-World20.94780.73650.193623.02190.05580.24000.01061.3587144.53980.242113.44515.26934.9700

RA-MPJPE: root-aligned MPJPE; hand errors are in mm.

With matched data and steps, MM-World outperforms EgoSim across all splits: PSNR rises by 0.54–1.13 dB, while LPIPS, FVD, and RA-MPJPE drop by 7.1%–20.2%, 5.1%–14.1%, and 14.9%–17.9%. We will clarify this system contribution relative to prior render conditioning, hand representations, and ray encoding in the revision.

2. Separating system performance from component contributions. Beyond §5.4’s render-coverage study, we remove scene rendering (w/o scene render) or remove SCoPE (w/o SCoPE), training all variants on the same 30K manipulation + 10K navigation clips for 3,000 steps under identical inference settings:

TaskConfigPSNR↑SSIM↑LPIPS↓RotErr↓ATE↓RPE rot↓RPE trans↓Flow-EPE↓FVD↓ROI-LPIPS↓RA-MPJPE↓PA-MPJPE↓PA-MPVPE↓
NavigationMM-World14.38170.41850.42378.19600.11600.38490.02755.7568218.7553————
w/o SCoPE14.24510.41270.41808.18670.12670.40610.02935.9578202.3371————
w/o scene render11.34070.25910.623663.60630.63521.90670.098520.2066359.4590————
Mobile manipulationMM-World17.33290.59070.323330.46840.06300.26640.01051.7737743.98720.312113.23565.97505.6081
w/o SCoPE17.08200.57940.328223.11900.06760.27640.01071.8533775.74290.308513.09105.88705.5222
w/o scene render14.06170.46160.494274.22530.14220.73330.02074.72891107.07730.366513.45925.82615.4657

Removing either component worsens PSNR, SSIM, ATE, and Flow-EPE on both tasks. Scene rendering has the larger impact (−3.04 dB PSNR on navigation, −3.27 dB on mobile manipulation), while SCoPE adds gains in PSNR, SSIM, LPIPS, ATE, and Flow-EPE on mobile manipulation. For hand representation, the EgoSim comparison above serves as a practical reference for MANO mesh vs. skeleton conditioning, reflecting the full designs including their scene–hand encoding and fusion pathways.

AuthorsOpenReview ↗

Response to Reviewer 5cLH (2/6)

W2 & Q2: Comparison with Render-Conditioned Navigation Methods

Thank you for pointing out this gap in our comparisons. We agree that GEN3C and ViewCrafter also condition generation on point-cloud renders along camera trajectories and support single-image novel view synthesis in the navigation track. We therefore added comparisons with both methods.

To improve comparability, we replaced ViewCrafter’s DUSt3R and GEN3C’s MoGE point-cloud estimation pipelines with Pi3X, as used by MM-World, to align the scales of the point clouds and camera trajectories. We also adapted the frame counts and resolutions based on each model’s original configuration. The table below reports the complete navigation results for these adapted baselines and the MM-World main model from our submission.

MethodPSNR↑SSIM↑LPIPS↓RotErr↓ATE↓RPE rot↓RPE trans↓Flow-EPE↓FVD↓
ViewCrafter + Pi3X12.61060.34060.538349.29110.71681.32840.104110.4704395.3997
GEN3C + Pi3X13.24420.35790.549820.29150.30130.64070.05517.9020291.6840
MM-World14.50360.42400.42325.99070.09530.31570.02395.5838207.8224

MM-World outperforms both adapted baselines on all metrics. Compared with GEN3C and ViewCrafter, PSNR improves by 1.26 dB and 1.89 dB, LPIPS decreases by 23.0% and 21.4%, and FVD decreases by 28.8% and 47.4%, respectively. These gains span image reconstruction, perceptual quality, and video quality. Flow-EPE also decreases by 29.3% and 46.7%, indicating closer agreement with the motion in the reference videos.

Together with the “without scene rendering” ablation in Part 1 (W1 & Q1), these results support the value of scene rendering as a camera-conditioning signal for improving camera control and spatial consistency. We will add these results and adaptation details to the revision and revise §5.2 to distinguish this established conditioning mechanism from the performance gains achieved by the complete MM-World system.

AuthorsOpenReview ↗

Response to Reviewer 5cLH (3/6)

W3: Manipulation Data Splits and Matched-Data Baseline Comparisons

Thank you for highlighting the need to clarify our data splits and separate model design from in-domain training. Below we detail the split protocols and expand the matched-data comparison to additional manipulation baselines; navigation co-training is examined in Part 4 (Q3).

1. Train/test splits and scene familiarity.

The three manipulation test splits use different partitioning protocols. For EgoDex, we hold out independent test episodes within each task. Training and test scenes and objects are similar, with differences mainly in the action sequences. This split therefore evaluates new action sequences in familiar tasks and environments. Its controlled setting and similar appearance distributions are also consistent with its higher PSNR and SSIM.

For KAI-Ego and EgoVerse, we strictly separate training and test scenes, ensuring that the test scenes do not appear in MM-World’s training data. These splits evaluate manipulation generation in unseen scenes. The source videos and parent sequences of the mobile manipulation test clips are also excluded from training, and their scenes are unseen during training.

We will clarify these protocols in the revision, explicitly distinguishing episode-level and scene-level splits, and release the corresponding split lists with the benchmark.

2. System gains remain across four methods under matched training data and steps.

Building on the matched-data protocol in Part 1 (W1 & Q1)—10K clips each from EgoDex, KAI-Ego, and EgoVerse (30K total), 3,000 steps, and the same random seed—we further compare MM-World and EgoSim with JointControlVideo and Generated Reality (Gen. Reality), all initialized from their respective pretrained backbones. As in Part 1, EgoSim uses MM-World’s scene point-cloud rendering pipeline to align geometric scale, and Gen. Reality is adapted from its 5B base to Wan2.1-Fun-14B-Control so all four methods use 14B pretrained backbones:

DatasetMethodPSNR↑SSIM↑LPIPS↓RotErr↓ATE↓RPE rot↓RPE trans↓Flow-EPE↓FVD↓ROI-LPIPS↓RA-MPJPE↓PA-MPJPE↓PA-MPVPE↓
EgoDexJointControlVideo25.92570.84110.151783.84560.03300.23450.00591.0369103.11160.211314.24626.01505.6590
Gen. Reality (14B)26.39540.84220.148279.25700.03250.23010.00580.996394.31070.199713.28455.30124.9845
EgoSim28.41190.90050.094146.99520.02460.14470.00490.654587.39620.176712.98285.33065.0080
MM-World29.15850.90730.084747.77620.02390.13410.00470.599175.03400.152611.05204.73764.4521
KAI-EgoJointControlVideo22.09170.71130.177870.90480.04060.16680.00650.8601147.01860.18098.17023.34123.1642
Gen. Reality (14B)22.39270.71800.175365.61380.04130.16620.00630.8095138.23290.17067.66072.98332.8317
EgoSim24.03020.81550.137338.82520.02930.11350.00500.5338113.28350.17608.51793.18733.0339
MM-World25.15980.83170.109534.17140.02880.10120.00480.472699.00960.14337.10482.82752.6812
EgoVerseJointControlVideo16.36050.55060.3767101.43700.11400.73240.02024.1580235.61240.333518.15066.71626.3299
Gen. Reality (14B)16.33150.54930.376889.94130.11400.73230.02013.9633223.64570.316216.26455.78805.4624
EgoSim20.40380.71870.208425.80480.05940.29310.01161.6816152.38770.265916.36815.85405.5211
MM-World20.94780.73650.193623.02190.05580.24000.01061.3587144.53980.242113.44515.26934.9700

RA-MPJPE: root-aligned MPJPE; hand errors are in mm.

Among all four methods, MM-World achieves the best results on all 13 metrics on KAI-Ego and EgoVerse, and on 12 of 13 on EgoDex, substantially outperforming JointControlVideo and Gen. Reality (14B) while maintaining clear margins over EgoSim in image/video quality, motion accuracy, and hand-pose accuracy. Thus, MM-World’s gains are not merely a byproduct of in-domain data access, consistent with the system contribution discussed in Part 1 (W1 & Q1).

Reviewer_vBVAOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents MM-World, a unified video world model for egocentric navigation and manipulation, conditioned on head and hand motions. The central idea is to represent these head and hand motion signals in a shared image space by reprojecting a 3D scene reconstructed from the initial reference image and overlaying MANO rendered hand meshes. The resulting unified visual condition with camera ray conditioning guides a retrained video diffusion model without requiring separate action-specific conditioning modules. Experiments demonstrate strong performance in visual quality, camera controllability and hand-motion accuracy compared with existing methods.

Critical strengths
  1. Meaningful problem formulation for more general egocentric interactive world modeling
  2. Simple and intuitive unified conditioning framework
  3. Evaluation on various benchmark and strong experimental results.
Critical weaknesses

1. Limited modeling and analysis of interaction-induced object-state changes. The proposed 3D scene representation is constructed form the initial image and remains fixed throughout generation. However, hand-object interactions inherently change object states, including their positions, orientations and contact relationships. Although the generator is expected to synthesize these changes implicitly, it remains unclear how the model resolves potential inconsistencies between the static scene condition and the evolving world. The paper provides limited analysis of whether such conflicts lead to object duplication, incorrect occlusion, or inconsistent object states, particularly under substantial camera movements or revisiting previously observed regions.

2. Insufficient ablations to justify the unified conditioning strategy. The central contribution seems to be the unified representation of head and hand motions in a shared image space. However, the ablation studies do not directly evaluate the advantages of unified conditioning over separate head- and hand-conditioning. Also, the individual contributions of camera-ray conditioning, hand rendering, reference-image conditioning, and joint training are not sufficiently isolated, making it difficult to determine which components are responsible for the observed improvements and whether the proposed unified conditioning is necessary.

3. Lack of direct evaluation of manipulation-induced object-state changes. Although the paper evaluates various metrics including visual quality, camera accuracy and hand-motion accuracy, these metrics do not directly measure whether the intended object manipulation is successfully performed. Accurate hand motion accuracy does not necessarily imply physically consistent hand-object interactions or correct object-state transitions. (This would be aligned with the Weakness 1.) There should be several qualitative or quantitative analysis whether the model captures the physical consequences of hand actions rather than only their visual appearance.

4. Limited validation of long-horizon mobile manipulation and state persistence. The paper introduces an autoregressive extension and provides long-horizon results.however, these experiments evaluate navigation and manipulation separately, while long-horizon mobile manipulation which combines these two controls remains insufficiently explored. In particular, it is unclear whether manipulation-inched object-state changes persist across autoregressive chunks, especially when the camera moves away from an interacted object (or interested?) and later returns. Such evaluation could be important for establishing the effectiveness of this framework over extended horizons.

Questions for authors

Q1. How does the model handle manipulated objects whose states differ from the initial 3D scene representation? Could the. uthors analyze potential inconsistencies, for example, in the case when revisiting manipulated regions?

Q2. Could the authors compare unified conditioning against separate head- and hand-conditioning pathways? Ablations of camera ray conditioning, hand rendering, reference-image conditioning, and joint training would also help clarify their contributions.

Q3. Could the authors evaluate object displacement accuracy, hand-object contact consistency or manipulation success to verify that the model captures interaction-induced state changes?

Q4. Could the authors evalaute long-horizon mobile manipulation, particularly when modified object states persist across autoregressive chunks and camera revisits?

8 comments
AuthorsOpenReview ↗

Response to Reviewer vBVA (1/5)

W1 & Q1: Static Scene Conditioning and Dynamic Object States

Thank you for highlighting the need for object-centric evaluation and analysis. We agree that this deserves further study. Below, we explain how the model generates object states that differ from the initial scene, focusing on the bidirectional model underlying MM-World’s main experiments and contributions.

There is indeed a tension between static scene rendering and dynamic interaction generation. We use scene rendering primarily to constrain the static background under camera motion, improving camera control as supported by our ablations. To keep the design simple, MM-World does not introduce an additional model to separate static and dynamic regions, which could introduce bias or errors. Instead, it learns interaction dynamics conditioned on hand actions through end-to-end training.

During training, the scene renderings contain objects in their initial states, while the model must predict plausible object poses and physical state changes in response to hand actions within an attention window spanning the entire clip. This encourages the model to learn real-world interaction dynamics implicitly rather than copy the scene renderings pixel by pixel, while also synthesizing novel views of the static scene as the camera moves. Occlusions and revisits within this window are also covered during training, and the model can generate physically plausible object responses.

We have added qualitative examples to Appendix E.5 of the revised paper (Fig. 8, p. 35, lines 1836–1880). The figure presents selected 81-frame clips from the bidirectional model, showing ball rolling, throwing and catching, and object relocation with camera revisiting. In the revisit example, the generated result retains the ball’s new placement in a neighboring shelf compartment and leaves the original compartment empty when the view returns. The caption also specifies that MM-World, EgoSim, and Wan2.1 Fun were each trained on 30K manipulation clips for 3,000 steps. Our response to W3/Q3 further quantifies object-state fidelity.

W3 & Q3: Object Trajectories and Object Responses

Thank you for suggesting object-centric evaluation. We agree that this is a valuable perspective: accurate hand motion does not necessarily imply correct object responses. We therefore added ObjMC, which compares 2D point trajectories on target objects in generated and ground-truth videos, complementing our visual quality, camera, and hand metrics.

Evaluation protocol. We use Qwen3-VL and SAM3 to identify target objects, first-frame segmentation masks, and query points in the ground-truth videos, then track the sampled object points with CoTracker3. Excluding the initialization frame, we compute 2D Euclidean distances between corresponding points visible in both videos within their overlapping image regions. ObjMC averages these distances over valid point–frame pairs within each clip, then equally across scorable clips. Lower values are better.

We also report the GT-visible point retention rate: the fraction of GT-visible point–frame pairs that are also visible in the generated video. The denominator is restricted to the overlapping image regions and query points initialized in the first frame, and includes trackable clips with no jointly visible point–frame pairs. This measures tracking coverage alongside trajectory error, avoiding an assessment based solely on low errors over a small visible subset.

Results. Each table entry reports ObjMC (pixels, ↓) / GT-visible point retention rate (↑).

MethodEgoDexKAI-EgoEgoVerse
EgoSim12.510 / 83.5%21.964 / 86.4%21.614 / 78.8%
RynnWorld-Teleop18.634 / 86.0%17.305 / 78.7%29.151 / 66.2%
Hand2World8.154 / 10.6%11.442 / 43.0%11.859 / 12.6%
Wan2.2-Fun-5B-Control14.723 / 75.5%17.989 / 81.8%33.159 / 72.7%
Generated Reality14.205 / 77.7%19.738 / 80.9%32.949 / 71.7%
Wan2.1-Fun-14B-Control15.747 / 81.9%23.162 / 84.1%37.569 / 81.0%
MM-World5.850 / 85.7%4.760 / 91.0%8.620 / 84.4%

The evaluation set consists of the 623 test clips in the manipulation track. MM-World achieves the lowest ObjMC across all three difficulty tiers, with retention rates of 85.7%, 91.0%, and 84.4%, respectively. These results indicate closer agreement with ground-truth object trajectories on the evaluable observations, providing additional evidence of object motion accuracy.

We will further refine the benchmark’s object-centric metrics to provide a more comprehensive evaluation. Thank you again for this valuable suggestion.

AuthorsOpenReview ↗

Response to Reviewer vBVA (2/5)

W2 & Q2: Camera-Control Ablations, Architecture Comparison, and Joint Training

Thank you for this suggestion. We added three sets of experiments: camera-control ablations, an architecture comparison with matched training data and steps, and a joint-training ablation. All models in these experiments are trained for 3,000 steps. We report the complete metrics below and will include them in the revised paper.

1. Camera-control ablations: scene rendering and camera rays.

Scene rendering projects the initial scene’s appearance and spatial layout into target views, while SCoPE provides explicit camera-ray geometry, including in regions not covered by the scene render. To evaluate their contributions, we use the same 30K manipulation and 10K navigation training clips and separately remove scene rendering or SCoPE. Results are reported for navigation and mobile manipulation:

TaskConfigurationPSNR ↑SSIM ↑LPIPS ↓RotErr ↓ATE ↓RPE rot ↓RPE trans ↓Flow-EPE ↓FVD ↓ROI-LPIPS ↓RA-MPJPE ↓PA-MPJPE ↓PA-MPVPE ↓
NavigationMM-World14.38170.41850.42378.19600.11600.38490.02755.7568218.7553————
without SCoPE14.24510.41270.41808.18670.12670.40610.02935.9578202.3371————
without scene rendering11.34070.25910.623663.60630.63521.90670.098520.2066359.4590————
Mobile manipulationMM-World17.33290.59070.323330.46840.06300.26640.01051.7737743.98720.312113.23565.97505.6081
without SCoPE17.08200.57940.328223.11900.06760.27640.01071.8533775.74290.308513.09105.88705.5222
without scene rendering14.06170.46160.494274.22530.14220.73330.02074.72891107.07730.366513.45925.82615.4657

Scene rendering provides substantial gains in image and motion quality: compared with the variant without it, PSNR increases by 3.04 dB on navigation and 3.27 dB on mobile manipulation, while LPIPS decreases by 32.1% and 34.6% and Flow-EPE by 71.5% and 62.5%, respectively. SCoPE provides further benefits alongside scene rendering, increasing PSNR by 0.14 and 0.25 dB and reducing ATE by 8.4% and 6.8% on the two tasks. On mobile manipulation, it also reduces Flow-EPE and FVD by 4.3% and 4.1%, respectively.

AuthorsOpenReview ↗

Response to Reviewer vBVA (3/5)

W2 & Q2: Camera-Control Ablations, Architecture Comparison, and Joint Training

2. Architecture comparison with matched training data and steps.

To compare different conditioning designs, we trained MM-World, JointControlVideo, EgoSim, and Generated Reality on the same manipulation data. The table below summarizes the action and scene conditions used in their original papers:

MethodMANO mesh renderingHand skeleton renderingNumerical hand conditioningSeparate camera conditioningScene rendering
JointControlVideo——✓——
EgoSim—✓——✓
Generated Reality—✓✓✓—
MM-World✓——✓✓

MM-World combines scene appearance, hand shape and pose, and explicit camera geometry. It composites scene and hand renders in a shared image space, jointly encodes them, and uses the pretrained model’s native visual conditioning pathway. EgoSim encodes scene and hand conditions separately before concatenation. MM-World’s shared representation supports camera and hand control and lets navigation samples without hand actions use the same interface, enabling the joint training evaluated in Point 3.

We sampled 10K clips each from EgoDex, KAI-Ego, and EgoVerse to form a shared 30K manipulation training set. All four methods were trained from their respective pretrained base models, using the same training data and random seed for 3,000 steps. EgoSim uses the same scene point-cloud rendering pipeline as MM-World to ensure a consistent geometric scale. Generated Reality is adapted from its original 5B backbone to Wan2.1-Fun-14B-Control, so all four methods use 14B backbones. The complete results are shown below:

DatasetMethodPSNR ↑SSIM ↑LPIPS ↓RotErr ↓ATE ↓RPE rot ↓RPE trans ↓Flow-EPE ↓FVD ↓ROI-LPIPS ↓RA-MPJPE ↓PA-MPJPE ↓PA-MPVPE ↓
EgoDexJointControlVideo25.92570.84110.151783.84560.03300.23450.00591.0369103.11160.211314.24626.01505.6590
Generated Reality (14B adaptation)26.39540.84220.148279.25700.03250.23010.00580.996394.31070.199713.28455.30124.9845
EgoSim28.41190.90050.094146.99520.02460.14470.00490.654587.39620.176712.98285.33065.0080
MM-World29.15850.90730.084747.77620.02390.13410.00470.599175.03400.152611.05204.73764.4521
KAI-EgoJointControlVideo22.09170.71130.177870.90480.04060.16680.00650.8601147.01860.18098.17023.34123.1642
Generated Reality (14B adaptation)22.39270.71800.175365.61380.04130.16620.00630.8095138.23290.17067.66072.98332.8317
EgoSim24.03020.81550.137338.82520.02930.11350.00500.5338113.28350.17608.51793.18733.0339
MM-World25.15980.83170.109534.17140.02880.10120.00480.472699.00960.14337.10482.82752.6812
EgoVerseJointControlVideo16.36050.55060.3767101.43700.11400.73240.02024.1580235.61240.333518.15066.71626.3299
Generated Reality (14B adaptation)16.33150.54930.376889.94130.11400.73230.02013.9633223.64570.316216.26455.78805.4624
EgoSim20.40380.71870.208425.80480.05940.29310.01161.6816152.38770.265916.36815.85405.5211
MM-World20.94780.73650.193623.02190.05580.24000.01061.3587144.53980.242113.44515.26934.9700

RA-MPJPE denotes root-aligned MPJPE. RA-MPJPE, PA-MPJPE, and PA-MPVPE are reported in mm.

In this setting, MM-World achieves the best results on all 13 metrics for KAI-Ego and EgoVerse, and on 12 of 13 metrics for EgoDex. Across the three datasets, PSNR improves by 3.07–4.59 dB over JointControlVideo and 2.76–4.62 dB over Generated Reality.

Compared with EgoSim, which also uses scene and hand rendering, MM-World improves PSNR by 0.54–1.13 dB and reduces LPIPS and FVD by 7.1%–20.2% and 5.1%–14.1%, respectively. Hand-region ROI-LPIPS, RA-MPJPE, and Flow-EPE decrease by 9.0%–18.6%, 14.9%–17.9%, and 8.5%–19.2%, respectively. These gains span generation quality, hand appearance and pose, and motion accuracy, supporting the effectiveness of MM-World’s overall conditioning design under matched task-specific training data and training steps.

Reviewer_duAMOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposed a video world model that jointly considers navigation and manipulation. They tackle this by conditioning a video generation model with both hand and head motion, where the head is responsible for navigation and the hands for manipulation. Both conditions are combined into a spatial prompt and injected into the video generation model through its existing conditioning pipeline. The method is trained and evaluated on a benchmark the authors construct from four different sources for both tasks, concluding with some ablations to validate design choices.

Critical strengths
  • Jointly considering manipulation and navigation makes sense and is a timely and important topic.
  • The figures are clear and contribute to a better understanding of the paper.
Critical weaknesses

My two main concerns relate to the evaluation protocol and the novelty of the proposed approach. First, the evaluation seems to contain a critical issue: Section 5.1 states that MM-World is trained on the four data sources of the introduced benchmark, while the baseline performances appear to come from publicly released checkpoints. If so, these publicly released checkpoints are not trained on the proposed benchmark, and the main result tables do not report comparable numbers. Second, the main contribution is a spatial prompt accounting for hands and camera motion. These spatial prompts have been extensively used to condition egocentric video generation for navigation and manipulation, see, for instance, [1] for a recent example, and a comparison to such methods is missing.

[1] Zhang, Chenyangguang, et al. "Controllable egocentric video generation via occlusion-aware sparse 3d hand joints." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2026.

Questions for authors
  • Are the main results reported for models trained on the same datasets? Would the gains in performance still hold if they are?
  • The benchmark contains an in-house dataset and is therefore not usable by the public and can not be seen as a major contribution.
  • The claim that "Splitting these actions into dedicated modules would fragment the control interface", i.e., that separate conditioning streams would be insufficient, is not empirically justified.
  • As the conditioning is a major contribution, a comparison to other related works, such as EgoControl and [1], would strengthen the paper. Moreover, a discussion on how the conditioning differs from previous spatial prompt approaches is needed.
  • A discussion on computational cost is missing.
  • The writing is somewhat dense at points, and some sections appear out of order. For instance, it does not become clear why masks are needed until section 3.3, despite the discussion on mask construction appearing in the section before.
4 comments
AuthorsOpenReview ↗

Response to Reviewer duAM (1/3)

W1 & Q1: Evaluation Fairness and Comparison with Matched Training Data

Thank you for your attention to evaluation fairness. The original main tables compare MM-World with released baseline checkpoints rather than models retrained on the same data, evaluating complete systems under our benchmark protocol while differences in training histories remain. To test whether the gains persist with matched task-specific training data, we added a controlled manipulation comparison under identical data and training steps, where MM-World retains clear advantages.

1. Navigation track. We evaluate on clips from the official DL3DV evaluation partition, whose scenes are excluded from MM-World’s training data. Navigation baselines such as SANA-WM, SolarWM, and Lingbot-World 2.0 also include DL3DV within larger multi-source corpora containing substantial additional data. MM-World is thus not uniquely advantaged by access to DL3DV; evaluated on the same held-out split, the released checkpoints provide a reasonable basis for a fair system-level comparison.

2. Manipulation track and matched-data comparison. Lacking a standard community test set, we built a manipulation benchmark spanning laboratory (EgoDex), everyday indoor (KAI-Ego), and in-the-wild (EgoVerse) settings. EgoDex holds out test episodes within each task, while KAI-Ego and EgoVerse use scene-level splits with all test scenes excluded from MM-World’s training.

To control for task-specific training data, we sampled 10K clips per dataset (30K total) and trained MM-World and the baselines from their respective pretrained base models for 3,000 steps using the same data and random seed. For fair adaptation, EgoSim uses MM-World’s scene point-cloud rendering pipeline for consistent geometric scale, and Generated Reality (Gen. Reality) is adapted from its 5B base to Wan2.1-Fun-14B-Control so all compared models share a 14B backbone:

DatasetMethodPSNR↑SSIM↑LPIPS↓RotErr↓ATE↓RPE rot↓RPE trans↓Flow-EPE↓FVD↓ROI-LPIPS↓RA-MPJPE↓PA-MPJPE↓PA-MPVPE↓
EgoDexJointControlVideo25.92570.84110.151783.84560.03300.23450.00591.0369103.11160.211314.24626.01505.6590
Gen. Reality (14B)26.39540.84220.148279.25700.03250.23010.00580.996394.31070.199713.28455.30124.9845
EgoSim28.41190.90050.094146.99520.02460.14470.00490.654587.39620.176712.98285.33065.0080
MM-World29.15850.90730.084747.77620.02390.13410.00470.599175.03400.152611.05204.73764.4521
KAI-EgoJointControlVideo22.09170.71130.177870.90480.04060.16680.00650.8601147.01860.18098.17023.34123.1642
Gen. Reality (14B)22.39270.71800.175365.61380.04130.16620.00630.8095138.23290.17067.66072.98332.8317
EgoSim24.03020.81550.137338.82520.02930.11350.00500.5338113.28350.17608.51793.18733.0339
MM-World25.15980.83170.109534.17140.02880.10120.00480.472699.00960.14337.10482.82752.6812
EgoVerseJointControlVideo16.36050.55060.3767101.43700.11400.73240.02024.1580235.61240.333518.15066.71626.3299
Gen. Reality (14B)16.33150.54930.376889.94130.11400.73230.02013.9633223.64570.316216.26455.78805.4624
EgoSim20.40380.71870.208425.80480.05940.29310.01161.6816152.38770.265916.36815.85405.5211
MM-World20.94780.73650.193623.02190.05580.24000.01061.3587144.53980.242113.44515.26934.9700

RA-MPJPE: root-aligned MPJPE; hand errors are in mm.

MM-World achieves the best results on all 13 metrics for KAI-Ego and EgoVerse, and on 12 of 13 for EgoDex. Over EgoSim, PSNR improves by 0.54–1.13 dB, while LPIPS, FVD, hand ROI-LPIPS, RA-MPJPE, and Flow-EPE drop by 7.1%–20.2%, 5.1%–14.1%, 9.0%–18.6%, 14.9%–17.9%, and 8.5%–19.2%, respectively, confirming our gains under matched data and steps.

3. Mobile-manipulation and long-horizon tracks. Both tracks also evaluate on scenes unseen during MM-World training. We formulate and benchmark egocentric mobile manipulation as a novel task; given the scarcity of dedicated joint data, our unified conditioning interface enables joint training on existing navigation and manipulation data.

AuthorsOpenReview ↗

Response to Reviewer duAM (2/3)

W2 & Q4: Method Novelty and Comparison with Related Work

Thank you for suggesting that we clarify our contributions and compare with related work. MM-World addresses egocentric mobile manipulation from a single image through unified visual conditioning and a shared training interface. This allows a single model to learn from existing navigation and manipulation data and support joint control of camera motion and hand interactions.

1. Unified visual conditioning and its role. Mobile manipulation requires the model to handle both navigation-scale viewpoint changes and local hand–object interactions. These two types of actions differ in their representations and training data. We represent camera motion by reprojecting the initial scene into the target views, and hand motion by rendering MANO hands in the target poses. We then combine these renders into a visual condition and a validity mask in a shared image space.

This makes navigation a special case of the unified representation without hand rendering. Navigation and manipulation samples can therefore share the same condition format, pretrained visual conditioning pathway, and training objective to train a single model. Given the scarcity of dedicated mobile-manipulation data, this interface provides a direct way to use both types of existing data.

2. Differences from prior spatial conditioning methods. The work you cited as [1], JointControlVideo, uses numerical 3D hand joint trajectories as explicit action inputs, without a separate camera trajectory input. EgoControl uses 3D full-body poses to jointly represent camera and body motion. MM-World explicitly takes camera trajectories and hand actions, combines scene reprojection and hand rendering into a unified visual condition, and uses the shared interface to jointly train on navigation and manipulation data.

The table below summarizes the action and scene conditions used in each method’s original paper:

MethodMANO mesh renderingHand skeleton renderingNumerical hand conditioningSeparate camera conditioningScene rendering
JointControlVideo——✓——
EgoSim—✓——✓
Gen. Reality—✓✓✓—
MM-World✓——✓✓

3. Additional comparisons with related methods. To demonstrate the advantages of our framework, we compared MM-World with JointControlVideo, EgoSim, and Generated Reality (Gen. Reality) using matched training data. All methods used the same 30K manipulation training set, the same random seed, and 3,000 training steps. The complete 13-metric results in W1/Q1 show consistent gains in generation quality, hand-pose accuracy, and motion accuracy across EgoDex, KAI-Ego, and EgoVerse under matched task-specific training data.

AuthorsOpenReview ↗

Response to Reviewer duAM (3/3)

Q3: Unified and Separate Conditioning Pathways

Thank you for pointing this out. By “fragment the control interface,” we meant that different actions are handled through separate conditioning pathways, not that separate pathways cannot support joint control. Indeed, our mobile-manipulation benchmark also evaluates existing baselines that use separate conditioning streams and support both navigation and manipulation.

Our goal is to let navigation and manipulation share the same visual conditioning format and processing pathway. For navigation samples without hand actions, the unified visual condition naturally reduces to scene rendering, without requiring a separate hand-conditioning input to the generator. We will revise the wording to clarify that unified conditioning is an effective design choice that supports joint training and reuses the pretrained model’s native visual conditioning pathway, rather than a necessary condition for joint control.

Q2: Public Availability of the Benchmark

Thank you for your concern about the benchmark’s public availability. We will release a complete benchmark package for independent evaluation, covering navigation, manipulation, mobile manipulation, and long-horizon generation.

Among the egocentric manipulation datasets used in our work, EgoDex and EgoVerse are publicly available, while KAI-Ego is not yet public. KAI-Ego was collected in-house across diverse everyday indoor settings. In addition to the complete benchmark package, we will release a subset of KAI-Ego training data upon acceptance.

The benchmark provides a unified basis for evaluating navigation, manipulation, and their combination in mobile manipulation. Through these releases, we hope to support the community in developing data and methods for mobile manipulation and improving interaction quality in video world models.

Q5: Computational Cost

Thank you for pointing out the missing discussion of computational cost. MM-World is built on Wan2.1 14B. We train the main model on 64 A100 GPUs with a batch size of 1 per GPU, at approximately 3,000 steps per day. Generating an 81-frame video at 480p resolution takes about 10 minutes on a single A100, with most of the time spent on the 50 denoising steps. The autoregressive variant uses sliding-window attention, KV caching, and few-step distillation. Its training and acceleration procedures draw on LongLive. We will include these settings and computational cost details in the revised paper to support reproducibility.

Q6: Writing Clarity and the Introduction of Masks

Thank you for your suggestions on the paper’s clarity and organization. We will simplify the wording and improve the order of presentation in the revised paper. In particular, Section 3.2 will explain what the masks do and why they are needed before describing how they are constructed. Thank you again for your careful reading and specific suggestions on the flow of the paper. We greatly appreciate the time and care you have devoted to our work.

Dear Reviewer duAM,

As the discussion period nears its close, we understand you have a busy schedule and sincerely appreciate your time. We wanted to gently check whether our response addresses your concerns:

  • Matched-data fairness (W1, Q1, Q4): Training MM-World, JointControlVideo [1], EgoSim, and Generated Reality (14B) from 14B backbones on the same data and steps confirms that our gains clearly hold, alongside our clarification of the held-out evaluation split.
  • Positioning & related work (W2, Q3, Q4): We contrasted our unified visual interface with [1] and EgoControl, highlighted its role in enabling joint navigation–manipulation training, and refined our wording regarding separate conditioning pathways.
  • Benchmark release, cost & clarity (Q2, Q5, Q6): We confirmed the public release of a complete standalone benchmark package (plus a subset of KAI-Ego training data upon acceptance), reported training/inference costs.

If anything remains unclear, we would be glad to clarify further before the discussion window closes. Thank you again!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.