- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
In this paper, the authors propose a ViewWeaver, which is a condition-enhanced multi-view world model. In detail, a View-aware Semantic-Motion Conditioning (VSMC) first incorporates each view’s initial observation and camera information into conditions and aligns the text and action conditions. Then, Condition Modulated Cross-view Attention (CM-CVA) uses view-specific conditions to modulate each view’s features. The proposed method achieve well performance on the LIBERO-90 and AgiBot-World datasets, which demonstrates its effectiveness.
Critical strengths
1/ The paper is well-motivated. Robots are typically equipped with multiple cameras providing different viewpoints, making effective coordination of multi-view information an important research problem.
2/ This paper is well-written and easy to follow.
3/ The proposed ViewWeaver achieves strong performance across multiple views on both simulated and real-world datasets.
Critical weaknesses
1/ In the Introduction, the authors point out that existing methods struggle to establish correspondences between image regions and the same 3D content, as well as to model how shared physical motion manifests across different views. However, it remains unclear how the proposed method effectively addresses these challenges, as it does not explicitly incorporate 3D geometric information like depth maps or physical priors.
2/ Another concern is the technical novelty of the proposed method. Although the paper is well-motivated, both VSMC and CM-CVA appear to primarily rely on existing attention and modulation mechanisms, which have been widely adopted in previous works. The authors should further clarify the technical novelty of these two modules and better distinguish them from existing approaches.
3/ The proposed method is built upon Stable Video Diffusion (SVD), which was introduced in 2023. Why did the authors not consider more recent and advanced video generation models, such as Wan or Cosmos?
4/ A text instruction may correspond to multiple valid motion trajectories. However, the paper provides limited discussion on how the proposed method handles such ambiguity. It would be helpful if the authors could provide further analysis and discussion on this issue.
5/ Since VSMC is a core component of the proposed framework, the authors should provide more details about the camera-aware attention modules and the image-text attention module in the main text rather than relegating them to the appendix. The main paper should be sufficiently self-contained to allow readers to understand the key technical designs without referring to the appendix.
I will consider raising my rating according to the authors’ rebuttal and other reviewers’ comments.
Questions for authors
See Weaknesses
2 comments
Thanks for the response. My concerns have been addressed, but I still don't think the paper meets the acceptance bar. I will keep my score unchanged.
Response to Reviewer vnoC - Thank you for reviewing our paper!
We sincerely appreciate your time and efforts in reviewing our paper! We provide detailed discussions based on your review:We sincerely appreciate your time and efforts in reviewing our paper! We provide detailed discussions based on your review:
Modeling Cross-View Content and Motion Consistency
ViewWeaver uses readily available camera and action information to model semantic–motion alignment and cross-view content interaction.
To enhance the consistency of image regions and 3D content, we integrate camera intrinsics and extrinsics to construct view-specific conditioning representations. CM-CVA further promotes scene and object consistency by modulating cross-view attention with embeddings encoding task intent and motion tendencies, encouraging information exchange about task-relevant objects and interactions.
To address physical motion consistency, we introduce VSMC which incorporates robot action information. Since one instruction can admit multiple trajectories, individual views may depict plausible but incompatible executions. VSMC grounds text in the current scene and aligns it with paired action representations, providing motion cues that encourage a unified execution across views.
Significance for Embodied Multi-View Generation and Comparison with Existing Approaches.
ViewWeaver improves multi-view consistency by exploiting complementary information from language and actions.
VSMC introduces action information specific to embodied scenarios to improve motion consistency. Embodied World Models such as GE and Cosmos Predict use text conditioning for multi-view generation. VSMC additionally aligns scene-grounded text with action representations to supply fine-grained motion cues. This addresses the ambiguity of multiple trajectories under the same instruction and encourages consistent motion across views.
Multi-view generation requires consistent objects and interactions. Existing models often concatenate views and exchange information through joint self-attention, which may distribute attention too diffusely. Robot arms and manipulated objects typically move against static backgrounds, making their cross-view consistency harder to maintain. We introduce condition-modulated cross-view attention. Through the preceding alignment, the conditioning representations already encode task intent and the robot’s motion tendencies. Modulating attention with these representations encourages cross-view information exchange to focus on task-relevant entities, particularly the objects being manipulated, improving multi-view consistency.
More broadly, we hope our work offers a design and training perspective for unified world models across modalities and views, as well as for visual, language, and action representations in vision-language-action models.
Applications on Other Backbone
Following Ctrl-World, we adopt SVD as our backbone. We further incorporate ViewWeaver’s design into Wan2.2 and Cosmos Predict2, training both variants in a single stage due to training-time constraints. The results are shown below.
For Wan2.2, head/hand FVD decreases by 8.9%/20.3%. For Cosmos Predict2, FVD decreases by 50.2%/58.5%. Both backbones also achieve higher PSNR,SSIM and lower LPIPS across views. These results demonstrate that ViewWeaver’s design can be applied across different generative backbones, and raise their performance on multi-view generation.
Resolving Cross-View Motion Ambiguity with VSMC
A text instruction may correspond to multiple valid trajectories, which can lead to inconsistent motion across views. VSMC is designed to address this ambiguity. It grounds the instruction in the current scene and aligns the resulting text representation with the action representation in a shared latent space. This alignment incorporates fine-grained motion information into the text embedding, strengthening motion constraints across views and encouraging them to depict a consistent execution.
In Figure 3, Cosmos Predict follows the ground-truth motion in the Hand view but moves rightward instead of upward in the Third view. Each trajectory may satisfy the instruction individually, yet the views depict incompatible executions. ViewWeaver maintains consistent motion across both views through VSMC’s complementary semantic and motion representations.
Writing About VSMC
Thanks for pointing this out. We have revised the manuscript and added details about VSMC to the main text.