- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes World2Motion, a joint framework for both scene-aware video and 3D human motion generation. The proposed method adapts the video world model Cosmos 3 by adding 3D motion tokens and the corresponding encoder-decoder. To further reduce the motion jerk, the paper proposes a shift-decoupled denoising strategy for video and motion streams. Experiments on the proposed dataset demonstrate the effectiveness of the proposed model.
Critical strengths
- The idea of using video world model prior for scene-aware 3D human motion generation is well-motivated.
- The presentation is clear and easy to follow, with helpful figures to explain the ideas.
Critical weaknesses
- Novelty is one of my concerns. The proposed model simply extends the video generation model to jointly video and human motion generation. Compared to CoMoVi, although the paper claims to further jointly generate scene-aware 3D human motion and corresponding videos, except training data, the whole model design and losses are not scene-specific.
- This paper only trains and evaluates on the proposed dataset, and the test set only has 100 sequences. It would be more convincing if the paper could also evaluate on common human motion generation benchmarks such as Motion-X++ and CoMoVi dataset as in the CoMoVi paper.
- No video quality metrics. Some videos in the supplementary materials look not plausible, such as the mop is video “03 · Lift and swing a mop” and human hand in “09 · Hip sways and arm waves”. As the paper is proposed to address video and human motion co-generation, it is important to evaluate the video quality metrics as in CoMoVi. Furthermore, some human motion examples have severe ground penetration such as “17 · Step sideways and sit”, it would be better to further add floating and ground penetration metrics.
- Metrics about the contact-F1 and ISR are very unclear: 3D object trajectory is not input or output of the proposed model, then how to render it with 3D human motion to evaluate the contact-F1? Especially for the cases with moving objects such as lifting boxes? Furthermore, is this ISR evaluation sensitive to camera position and rotation? Also it is important to release the text prompt of Qwen3-VL for the evaluation metrics.
- The proposed method applies post-processing for main table comparison, which is not convincing enough for comparison.
Questions for authors
- Do you have any explanation why human-object interaction methods are so bad in terms of contact metrics such as Contact-F1? From the provided visualization, although the interaction is not good, most of them have contacts on the objects. Moreover, it would be helpful to also provide the ground truth motion performance so that we can better see the gap between each model and the upper bound.
- The CoMoVi visualization is strange, as all the SMPL videos of CoMoVi don’t have root translation. Is it the reason why the CoMoVi video looks good, but its SMPL motions are bad?