- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
MM-World generates egocentric video from one image, a static scene caption, and head and hand trajectories. Head motion is rendered by lifting the first frame to a point cloud with Pi3X depth, with hand pixels and low-confidence points removed, and reprojecting it along the camera path. Hand motion is rendered as WiLoR-estimated MANO meshes pasted over the scene render. The composite and its mask enter Wan2.1-I2V-14B through the backbone's existing 36-channel input, and SCoPE adds Plücker rays to every attention block. Only LoRA and SCoPE are trained, on 2.52M clips from DL3DV, EgoDex, EgoVerse and the in-house KAI-Ego. A LongLive-style distilled autoregressive variant handles long sequences.
The paper also builds an 846-case benchmark: navigation (156 DL3DV clips), manipulation in three tiers (623 clips), mobile manipulation (30 KAI-Ego clips) and long-horizon generation (37 sequences). Against released checkpoints run through their own interfaces, MM-World is best on almost every short-horizon metric. A matched-training ablation shows that the scene render is what carries head motion.
Critical strengths
-
The conditioning design is simple but efficiant, and leaves the backbone unchanged. Both actions end up in the 16+4+16-channel input that Wan2.1-I2V already uses for its conditioning video and mask, no input layer is added, and navigation clips use the same format with an empty hand render. The paper handles the details that matter in practice: it removes first-frame hand pixels so hands are not frozen into the scene, keeps only confident depth, and randomizes the SCoPE gate against scale error.
-
The short-horizon results improve a lot and come from explicit held-out splits where it matters most. On navigation, MM-World beats eight camera-controlled world models that all receive the trajectory. On mobile manipulation, PSNR is 5–6 dB higher and MPJPE 33–43% lower than the strongest baseline. The supplementary video, including control from a real user's headset and hand tracking, matches these numbers.
-
The render ablation is controlled and reported honestly. Four models share initialization, seed, data order and step count and differ only in how many first-frame points they keep. The render carries head motion (DL3DV RotErr 6.37° vs. 33.16° with rays only, PSNR 15.24 vs. 11.96 dB), hand accuracy does not depend on it (KAI-Ego MPJPE 8.14 vs. 7.97), halving the points costs little, and mixing render-free clips into training cuts the failure without a render from 99.2° to 54.1° (Tables 4, 12). The authors also report a second checkpoint, note that the High/Low/Random ranking changes between checkpoints, and give the oracle-depth gap (14.50 to 15.46 dB, Table 14).
Critical weaknesses
-
The advance over the closest render-conditioned methods is not clearly established. The paper overstates what is new relative to EgoSim and DWM. The stated key insight (L84–90) is to express head and hand actions as one image-space render. DWM already conditions a video diffusion model on static-scene renders along a camera trajectory plus egocentric hand-mesh renders, and trains it for joint locomotion and manipulation. EgoSim combines a first-frame point-cloud render, projected hand keypoints and a validity mask. What is new is the MANO-mesh overlay on a confidence-filtered first-frame cloud through the backbone's native input, co-training with navigation video, the ray conditioning, and the scale of the system. That is a real system contribution, but the paper should be positioned at that level. Table 2 shows a large system-level gain over EgoSim (EgoDex PSNR 29.85 vs. 20.34 dB), and the render ablation supports the value of scene rendering. These results do not isolate the contribution of MM-World's added design choices alongside differences in data, training and inference configuration (see Q1).
-
The navigation advantage is not established against the closest render-conditioned methods. Sec. 5.2 (L363–365) explains the PSNR gain by first-frame surfaces reaching each target view as pixel-aligned renders, whereas the baselines must regenerate them. GEN3C (CVPR 2025) and ViewCrafter (2024), both cited at L122 and both with public code, condition on exactly such single-image point-cloud renders along a camera path. Without one of them, Table 1 shows that MM-World beats ray- and action-conditioned world models, but not that it improves on render-conditioned camera control (see Q2).
-
The manipulation comparison does not separate the method from in-domain training. MM-World trains on 2.46M manipulation clips from the same three sources as the test tiers, while every baseline is a released checkpoint trained elsewhere. C.1 states benchmark clips are held out (lines 881–882), and D.1 states the split unit only for DL3DV (scene) and the mobile track (parent sequence). For the three manipulation tiers it describes how test clips were picked, but not whether those groups, episodes or demonstrators are excluded from training. KAI-Ego alone contributes 1.08M training clips cut from 139K recordings. As it stands, the 5–6 dB margins in Table 2 measure system performance on the authors' own data sources, not the effect of the conditioning design. (see Q3)
-
The manipulation RotErr comparison needs a check for alignment-induced offsets. On the manipulation tiers, RotErr mostly measures the Umeyama alignment rather than camera following. In Table 8b, five of six baselines have RotErr of 87–109° on EgoDex and KAI-Ego, but rotational RPE of only 0.16–0.27° per frame. For an 81-frame sequence, 80 such transitions would accumulate only 13–22° of rotation discrepancy. The gap may suggest a large initial offset, which needs direct measurement on a common valid set. Camera following can be checked on the existing outputs by fixing each trajectory's coordinates to its first camera pose, comparing generated orientations against the supplied camera controls for subsequent frames, and reporting both mean angular error and rotational RPE at short and longer time intervals (see Q4).
-
The AR training horizon and actual evaluated horizon are unclear. Tables 10–11 report AR results. The missing information is how these lengths relate to training. Stage 1 gives no teacher-forcing video length; Stage 2 describes an initial 5s-window and later windows such as 5–10s and 10–15s, but gives no maximum streaming rollout or number of windows. The 21-latent-frame attention window is a local context, not the total training horizon. Thus, the tables do not establish whether the model is tested beyond its training horizon. Please give the training lengths and final AR checkpoint configuration. Also, D.3 ends evaluation when either video ends, while D.2 keeps native frame rates and rounds baseline lengths to valid chunks. The nominal reference durations do not identify the actual generated and scored duration; frame counts, rates and timestamp matching are needed.
Questions for authors
The short-horizon results are strong, the render ablation is controlled, and I view the system positively. I would raise the score if the response clarifies the added contribution and provides convincing evidence, or a justified boundary, for navigation against the closest rendering-based methods.
- What is the added methodological contribution over DWM and EgoSim, and which existing comparisons or ablations support its benefit?
- What evidence establishes a navigation advantage over the closest render-conditioned camera methods, such as GEN3C and ViewCrafter? If they were excluded, why, and what navigation claim remains supported by the current comparisons?
- How do joint head/hand motion and scene familiarity in training compare with the mobile test clips? What evidence supports composition beyond training and an actual contribution from navigation co-training?
- After recomputing camera following in coordinates fixed by each trajectory's first pose, do the main rotation and translation gains remain? Please use the protocol and coverage checks in W4 to distinguish control error from alignment offsets.
- What are the video lengths in Stage 1 and the maximum streaming rollout in Stage 2 for the final AR checkpoint? Do the actual long-horizon evaluations exceed that training horizon?
- As generation length increases, do object states, contact and the overall manipulation remain coherent?
9 comments
Response to Reviewer 5cLH (2/6)
W2 & Q2: Comparison with Render-Conditioned Navigation Methods
Thank you for pointing out this gap in our comparisons. We agree that GEN3C and ViewCrafter also condition generation on point-cloud renders along camera trajectories and support single-image novel view synthesis in the navigation track. We therefore added comparisons with both methods.
To improve comparability, we replaced ViewCrafter’s DUSt3R and GEN3C’s MoGE point-cloud estimation pipelines with Pi3X, as used by MM-World, to align the scales of the point clouds and camera trajectories. We also adapted the frame counts and resolutions based on each model’s original configuration. The table below reports the complete navigation results for these adapted baselines and the MM-World main model from our submission.
| Method | PSNR↑ | SSIM↑ | LPIPS↓ | RotErr↓ | ATE↓ | RPE rot↓ | RPE trans↓ | Flow-EPE↓ | FVD↓ |
|---|---|---|---|---|---|---|---|---|---|
| ViewCrafter + Pi3X | 12.6106 | 0.3406 | 0.5383 | 49.2911 | 0.7168 | 1.3284 | 0.1041 | 10.4704 | 395.3997 |
| GEN3C + Pi3X | 13.2442 | 0.3579 | 0.5498 | 20.2915 | 0.3013 | 0.6407 | 0.0551 | 7.9020 | 291.6840 |
| MM-World | 14.5036 | 0.4240 | 0.4232 | 5.9907 | 0.0953 | 0.3157 | 0.0239 | 5.5838 | 207.8224 |
MM-World outperforms both adapted baselines on all metrics. Compared with GEN3C and ViewCrafter, PSNR improves by 1.26 dB and 1.89 dB, LPIPS decreases by 23.0% and 21.4%, and FVD decreases by 28.8% and 47.4%, respectively. These gains span image reconstruction, perceptual quality, and video quality. Flow-EPE also decreases by 29.3% and 46.7%, indicating closer agreement with the motion in the reference videos.
Together with the “without scene rendering” ablation in Part 1 (W1 & Q1), these results support the value of scene rendering as a camera-conditioning signal for improving camera control and spatial consistency. We will add these results and adaptation details to the revision and revise §5.2 to distinguish this established conditioning mechanism from the performance gains achieved by the complete MM-World system.
Response to Reviewer 5cLH (3/6)
W3: Manipulation Data Splits and Matched-Data Baseline Comparisons
Thank you for highlighting the need to clarify our data splits and separate model design from in-domain training. Below we detail the split protocols and expand the matched-data comparison to additional manipulation baselines; navigation co-training is examined in Part 4 (Q3).
1. Train/test splits and scene familiarity.
The three manipulation test splits use different partitioning protocols. For EgoDex, we hold out independent test episodes within each task. Training and test scenes and objects are similar, with differences mainly in the action sequences. This split therefore evaluates new action sequences in familiar tasks and environments. Its controlled setting and similar appearance distributions are also consistent with its higher PSNR and SSIM.
For KAI-Ego and EgoVerse, we strictly separate training and test scenes, ensuring that the test scenes do not appear in MM-World’s training data. These splits evaluate manipulation generation in unseen scenes. The source videos and parent sequences of the mobile manipulation test clips are also excluded from training, and their scenes are unseen during training.
We will clarify these protocols in the revision, explicitly distinguishing episode-level and scene-level splits, and release the corresponding split lists with the benchmark.
2. System gains remain across four methods under matched training data and steps.
Building on the matched-data protocol in Part 1 (W1 & Q1)—10K clips each from EgoDex, KAI-Ego, and EgoVerse (30K total), 3,000 steps, and the same random seed—we further compare MM-World and EgoSim with JointControlVideo and Generated Reality (Gen. Reality), all initialized from their respective pretrained backbones. As in Part 1, EgoSim uses MM-World’s scene point-cloud rendering pipeline to align geometric scale, and Gen. Reality is adapted from its 5B base to Wan2.1-Fun-14B-Control so all four methods use 14B pretrained backbones:
| Dataset | Method | PSNR↑ | SSIM↑ | LPIPS↓ | RotErr↓ | ATE↓ | RPE rot↓ | RPE trans↓ | Flow-EPE↓ | FVD↓ | ROI-LPIPS↓ | RA-MPJPE↓ | PA-MPJPE↓ | PA-MPVPE↓ |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| EgoDex | JointControlVideo | 25.9257 | 0.8411 | 0.1517 | 83.8456 | 0.0330 | 0.2345 | 0.0059 | 1.0369 | 103.1116 | 0.2113 | 14.2462 | 6.0150 | 5.6590 |
| Gen. Reality (14B) | 26.3954 | 0.8422 | 0.1482 | 79.2570 | 0.0325 | 0.2301 | 0.0058 | 0.9963 | 94.3107 | 0.1997 | 13.2845 | 5.3012 | 4.9845 | |
| EgoSim | 28.4119 | 0.9005 | 0.0941 | 46.9952 | 0.0246 | 0.1447 | 0.0049 | 0.6545 | 87.3962 | 0.1767 | 12.9828 | 5.3306 | 5.0080 | |
| MM-World | 29.1585 | 0.9073 | 0.0847 | 47.7762 | 0.0239 | 0.1341 | 0.0047 | 0.5991 | 75.0340 | 0.1526 | 11.0520 | 4.7376 | 4.4521 | |
| KAI-Ego | JointControlVideo | 22.0917 | 0.7113 | 0.1778 | 70.9048 | 0.0406 | 0.1668 | 0.0065 | 0.8601 | 147.0186 | 0.1809 | 8.1702 | 3.3412 | 3.1642 |
| Gen. Reality (14B) | 22.3927 | 0.7180 | 0.1753 | 65.6138 | 0.0413 | 0.1662 | 0.0063 | 0.8095 | 138.2329 | 0.1706 | 7.6607 | 2.9833 | 2.8317 | |
| EgoSim | 24.0302 | 0.8155 | 0.1373 | 38.8252 | 0.0293 | 0.1135 | 0.0050 | 0.5338 | 113.2835 | 0.1760 | 8.5179 | 3.1873 | 3.0339 | |
| MM-World | 25.1598 | 0.8317 | 0.1095 | 34.1714 | 0.0288 | 0.1012 | 0.0048 | 0.4726 | 99.0096 | 0.1433 | 7.1048 | 2.8275 | 2.6812 | |
| EgoVerse | JointControlVideo | 16.3605 | 0.5506 | 0.3767 | 101.4370 | 0.1140 | 0.7324 | 0.0202 | 4.1580 | 235.6124 | 0.3335 | 18.1506 | 6.7162 | 6.3299 |
| Gen. Reality (14B) | 16.3315 | 0.5493 | 0.3768 | 89.9413 | 0.1140 | 0.7323 | 0.0201 | 3.9633 | 223.6457 | 0.3162 | 16.2645 | 5.7880 | 5.4624 | |
| EgoSim | 20.4038 | 0.7187 | 0.2084 | 25.8048 | 0.0594 | 0.2931 | 0.0116 | 1.6816 | 152.3877 | 0.2659 | 16.3681 | 5.8540 | 5.5211 | |
| MM-World | 20.9478 | 0.7365 | 0.1936 | 23.0219 | 0.0558 | 0.2400 | 0.0106 | 1.3587 | 144.5398 | 0.2421 | 13.4451 | 5.2693 | 4.9700 |
RA-MPJPE: root-aligned MPJPE; hand errors are in mm.
Among all four methods, MM-World achieves the best results on all 13 metrics on KAI-Ego and EgoVerse, and on 12 of 13 on EgoDex, substantially outperforming JointControlVideo and Gen. Reality (14B) while maintaining clear margins over EgoSim in image/video quality, motion accuracy, and hand-pose accuracy. Thus, MM-World’s gains are not merely a byproduct of in-domain data access, consistent with the system contribution discussed in Part 1 (W1 & Q1).
Response to Reviewer 5cLH (1/6)
W1 & Q1: Contribution, System Comparison, and Component Ablations
Thank you for suggesting that we clarify our contribution and the sources of our gains. We agree that MM-World’s primary contribution is its system design and integration for single-image egocentric mobile manipulation: unified visual conditioning and a shared training interface let one model learn viewpoint changes and hand interactions jointly from existing navigation and manipulation data.
1. Differences from DWM and EgoSim, and system comparison. DWM assumes a static 3D scene and conditions on trajectory scene renders plus hand-mesh renders, so scene appearance and view changes come mainly from rendering existing 3D assets. MM-World instead estimates a partial point cloud from a single initial image at inference, generating both hand interactions and newly revealed regions under camera motion.
EgoSim renders a first-frame point cloud, projected hand keypoints, and a validity mask, encoding scene and hand separately before concatenation. MM-World uses MANO meshes and composites scene and hand renders into one visual condition fed through the backbone’s native visual pathway; navigation samples simply omit the hand render under the same format and objective.
For the system comparison between EgoSim and MM-World, we sampled 10K clips each from EgoDex, KAI-Ego, and EgoVerse (30K total), initialized both from their respective Wan2.1 14B backbones, and trained for 3,000 steps with the same data and seed, adapting EgoSim to MM-World’s scene point-cloud rendering pipeline to align geometric scale:
RA-MPJPE: root-aligned MPJPE; hand errors are in mm.
With matched data and steps, MM-World outperforms EgoSim across all splits: PSNR rises by 0.54–1.13 dB, while LPIPS, FVD, and RA-MPJPE drop by 7.1%–20.2%, 5.1%–14.1%, and 14.9%–17.9%. We will clarify this system contribution relative to prior render conditioning, hand representations, and ray encoding in the revision.
2. Separating system performance from component contributions. Beyond §5.4’s render-coverage study, we remove scene rendering (
w/o scene render) or remove SCoPE (w/o SCoPE), training all variants on the same 30K manipulation + 10K navigation clips for 3,000 steps under identical inference settings:Removing either component worsens PSNR, SSIM, ATE, and Flow-EPE on both tasks. Scene rendering has the larger impact (−3.04 dB PSNR on navigation, −3.27 dB on mobile manipulation), while SCoPE adds gains in PSNR, SSIM, LPIPS, ATE, and Flow-EPE on mobile manipulation. For hand representation, the EgoSim comparison above serves as a practical reference for MANO mesh vs. skeleton conditioning, reflecting the full designs including their scene–hand encoding and fusion pathways.