acceptodds
Withdrawn from ICLR 2027

World2Motion: Turning Video World Models into 3D Human Motion Generators

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We present World2Motion, a framework that generates scene-aware 3D human motion and corresponding video from a single image and a text prompt. While existing 3D motion generators learn from motion datasets, their generalization is constrained by limited coverage of environments. In contrast, video world models such as Cosmos 3 offer broader environmental priors but are not designed for full-body motion generation; recovering motion from their generated videos requires costly two-stage inference. To address these, we turn Cosmos 3 into a single-stage 3D motion generator. This adaptation has two challenges: the scarcity of paired video–motion data and temporal instability in the generated motion. First, we construct a training dataset combining synthetic video–motion pairs with real videos paired with estimated 3D motion. Second, we propose a shift-decoupled noise schedule that assigns different noise levels to video and motion through shared denoising progress. This design accommodates the different denoising requirements of the two modalities, reducing motion jitter. Experiments on a multi-source interaction benchmark show that World2Motion has better motion–text alignment and scene interaction compared with the evaluated 3D motion generators. It also matches the interaction success rate of the two-stage baseline while achieving approximately 3.3 faster inference.

Primary area:
generative models
Keywords:
Motion Generation, Diffusion Model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_duYeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes World2Motion, a joint framework for both scene-aware video and 3D human motion generation. The proposed method adapts the video world model Cosmos 3 by adding 3D motion tokens and the corresponding encoder-decoder. To further reduce the motion jerk, the paper proposes a shift-decoupled denoising strategy for video and motion streams. Experiments on the proposed dataset demonstrate the effectiveness of the proposed model.

Critical strengths
  1. The idea of using video world model prior for scene-aware 3D human motion generation is well-motivated.
  2. The presentation is clear and easy to follow, with helpful figures to explain the ideas.
Critical weaknesses
  1. Novelty is one of my concerns. The proposed model simply extends the video generation model to jointly video and human motion generation. Compared to CoMoVi, although the paper claims to further jointly generate scene-aware 3D human motion and corresponding videos, except training data, the whole model design and losses are not scene-specific.
  2. This paper only trains and evaluates on the proposed dataset, and the test set only has 100 sequences. It would be more convincing if the paper could also evaluate on common human motion generation benchmarks such as Motion-X++ and CoMoVi dataset as in the CoMoVi paper.
  3. No video quality metrics. Some videos in the supplementary materials look not plausible, such as the mop is video “03 · Lift and swing a mop” and human hand in “09 · Hip sways and arm waves”. As the paper is proposed to address video and human motion co-generation, it is important to evaluate the video quality metrics as in CoMoVi. Furthermore, some human motion examples have severe ground penetration such as “17 · Step sideways and sit”, it would be better to further add floating and ground penetration metrics.
  4. Metrics about the contact-F1 and ISR are very unclear: 3D object trajectory is not input or output of the proposed model, then how to render it with 3D human motion to evaluate the contact-F1? Especially for the cases with moving objects such as lifting boxes? Furthermore, is this ISR evaluation sensitive to camera position and rotation? Also it is important to release the text prompt of Qwen3-VL for the evaluation metrics.
  5. The proposed method applies post-processing for main table comparison, which is not convincing enough for comparison.
Questions for authors
  1. Do you have any explanation why human-object interaction methods are so bad in terms of contact metrics such as Contact-F1? From the provided visualization, although the interaction is not good, most of them have contacts on the objects. Moreover, it would be helpful to also provide the ground truth motion performance so that we can better see the gap between each model and the upper bound.
  2. The CoMoVi visualization is strange, as all the SMPL videos of CoMoVi don’t have root translation. Is it the reason why the CoMoVi video looks good, but its SMPL motions are bad?
Reviewer_yNXDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes World2Motion, a framework that jointly generates video and 3D human motion by leveraging the pretrained Cosmos 3 model, which supports joint video-action generation. To this end, the authors construct a hybrid dataset by synthesizing videos from existing 3D motion sequences using MiniMax-H3 and estimating 3D motion from real videos using CameraHMR. They also propose a shift-decoupled noise schedule that assigns different shift values to video and motion while maintaining shared denoising progress. Experiments demonstrate improved motion-text alignment and scene interaction performance compared to existing motion generation methods.

Critical strengths
  • I think adapting a pretrained video world model to directly generate scene-aware 3D human motion is a practical and interesting direction. The framework generates both video and 3D motion from a single image and text prompt without requiring explicit 3D scene geometry.

  • I find the construction of paired video-motion training data using powerful pretrained models, such as MiniMax-H3 and CameraHMR, interesting. This provides a practical approach to obtaining training data for joint video-motion generation.

  • The authors evaluate World2Motion against various motion generation methods and demonstrate promising improvements in motion-text alignment and scene interaction. The ablation studies also provide evidence for the effectiveness of the proposed motion representation and noise schedule.

Critical weaknesses
  • The methodological novelty beyond the existing Cosmos 3 framework is limited. The proposed method mainly extends the existing video-action joint generation architecture of Cosmos 3 to video-motion generation by replacing the action representation with full-body SMPL motion. The MoT architecture and shared attention mechanism are largely inherited from Cosmos 3. Moreover, assigning different noise levels to different modalities has already been explored in multimodal diffusion models, such as UniDiffuser [1] and Unified World Models [2]. The proposed shift-decoupled noise schedule appears to be a specific implementation of modality-specific timestep shifting with shared denoising progress. Although the ablation studies demonstrate improvements over a shared noise schedule, they do not sufficiently justify the advantage of this formulation over existing independent or asynchronous denoising strategies.

  • The contributions of joint video generation and the pretrained world model priors are not sufficiently demonstrated. The authors motivate their approach by leveraging the rich scene understanding of pretrained video world models for 3D human motion generation. However, it is unclear whether joint video generation actually improves motion quality and scene interaction compared to directly generating motion using the same pretrained backbone. Furthermore, most training pairs are synthesized from existing 3D motion datasets, and the paper does not provide an ablation isolating the contribution of pretrained video priors or different training data sources. Therefore, it is difficult to determine whether the reported improvements come from the proposed joint generation framework or primarily from supervised motion learning on the constructed dataset.

[1] Bao et al., One Transformer Fits All Distributions in Multi-Modal Diffusion at Scale, ICML'23
[2] Zhu et al., Unified World Models: Coupling Video and Action Diffusion for Pretraining on Large Robotic Datasets, RSS'25

Questions for authors
  • Could the authors compare the proposed shift-decoupled noise schedule with existing independent or asynchronous noise scheduling strategies? Additional ablations with different video and motion noise shifts would help justify the proposed formulation.

  • Since the proposed method builds on Cosmos 3, which supports forward dynamics (action-conditioned video generation), inverse dynamics (video-conditioned action generation), and joint video-action generation, could the authors evaluate whether World2Motion supports analogous generation modes? Specifically, it would be interesting to evaluate motion-conditioned video generation, video-conditioned 3D motion generation, and joint video-motion generation. Reporting the performance of each mode would help clarify the capabilities of the proposed framework beyond joint generation.

Reviewer_VRhKOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents World2Motion, which adapts the Mixture-of-Transformers backbone of Cosmos 3 to generate video and 3D human motion together, from one image and a text prompt. It trains on a hybrid dataset in two parts. The first is motions from existing interaction datasets paired with videos synthesized by MiniMax-H3. The second is real videos with CameraHMR pseudo-labels. The paper also proposes a shift-decoupled noise schedule that uses different noise levels for video and motion. On a 100-case interaction benchmark, it reports better text alignment than motion generators and matches the interaction success rate of a MiniMax-H3 + CameraHMR two-stage pipeline, at about 3.3× faster inference.

Critical strengths
  • Using video world models as a source of scene priors for 3D motion generation is a promising direction.
  • A single-stage joint generator avoids a separate motion-recovery pass.
  • The noise-schedule idea is good and validated empirically, but I would have liked to see more details and the intuition behind this design.
Critical weaknesses

The choice of pose estimator weakens both the data and the baseline.

  • CameraHMR is a relatively old method and works on individual images.
  • Many recent methods are significantly better, e.g., DuoMo, GENMO, GEM-X, MHR, NLF and CARI4D.
  • The estimator matters a lot for the MiniMax-H3 + pose baseline. Using an image-based method on a temporal task disadvantages that baseline.
  • A fair comparison should use a temporal pose estimation model.

The 3.3× faster inference claim needs clarification.

  • The reported numbers (523.67 s vs 156.59 s) do give about 3.3×.
  • However, the baseline uses a different video model (MiniMax-H3), and no timing breakdown is given. MiniMax-H3 itself is much bigger than Cosmos-Nano used in the paper.
  • It is therefore unclear how much of the speedup comes from removing the motion-recovery stage and how much from the choice of video generator.

Missing comparisons with interaction-focused motion generation methods. Scene and object interaction is the paper’s main motivation, yet it does not compare with methods such as DAViD and CHOIS, and potentially GRAIL.

No quality control for the MiniMax pseudo-videos.

  • It is not clear how the authors make sure the MiniMax-H3 videos accurately follow the source motions.
  • No details about any filtering process are provided.
  • Misaligned video–motion pairs would directly corrupt the paired supervision the method depends on.

The relative root translation design comes from ARDY, so it cannot be considered a major contribution of the paper.

The shift-decoupled noise schedule needs more rationale. I would have liked a clearer explanation of the main intuition behind it, and why this parameterization was chosen over alternatives.

No controllability. Controllability similar to Kimodo, such as keyframe, joint or trajectory constraints, is missing. Such constraints are important for practical use of motion generators.

Motion quality. The motion in the supplementary video looks quite low quality and jittery, and not what I would expect from Minimax-h3 + Good Pose Estimator based on my extensive experience on this topic. Also, for Astra examples, it is not clear if ASTRA is also refining the motion in addition to generating the assets and object motion.

Overall, the paper addresses an interesting direction, but in its current form it has several gaps before it can be accepted to ICLR.

Questions for authors
  • Why choose CameraHMR over recent temporal pose estimation methods? How do the results change for the baseline (and the training labels) with a stronger temporal estimator?
  • Can you give a timing breakdown of the two-stage baseline, separating video generation from pose estimation?
  • How were the MiniMax-H3 videos checked or filtered for accuracy against the source motion?
  • What is the main intuition behind the shift-decoupled noise schedule?
  • How does the method compare with DAViD, CHOIS and GRAIL? Can the model support Kimodo-style controllability?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.