acceptodds
Under review as a conference paper at ICLR 2027

ViewWeaver: A Condition-Enhanced Multi-View Embodied World Model

Abstract

Video world models have been increasingly adopted for future prediction and world simulation in embodied intelligence. As embodied policies increasingly rely on multi-view observations, generating coherent futures across views becomes essential. However, methods that simply concatenate multi-view inputs or rely on attention to implicitly infer cross-view relationships often produce inconsistencies in scene appearance and robot motion. We propose ViewWeaver, a condition-enhanced multi-view world model comprising two key components. First, View-aware Semantic-Motion Conditioning (VSMC) incorporates each view's initial observation and camera information into conditions. It further aligns text and action conditions in a shared latent space to encourage complementarity between task semantics and fine-grained motion cues. Second, Condition-Modulated Cross-View Attention (CM-CVA) uses the view-specific conditions to modulate each view's features before selectively aggregating complementary information from the other views. We evaluate ViewWeaver on the LIBERO-90 dataset and the AgiBot-World dataset under both text- and action-conditioned settings. ViewWeaver achieves the strongest overall performance across views. In particular, it reduces third-view FVD from to on text-conditioned LIBERO-90, a relative reduction of , and reduces left- and right-view FVD by and on action-conditioned AgiBot-World. Extensive ablation studies and qualitative comparisons further demonstrate that ViewWeaver improves cross-view consistency in both scene appearance and robot motion. Code is provided in the supplementary material.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Embodied AI, World Model, Multi-View Generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_vnoCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors propose a ViewWeaver, which is a condition-enhanced multi-view world model. In detail, a View-aware Semantic-Motion Conditioning (VSMC) first incorporates each view’s initial observation and camera information into conditions and aligns the text and action conditions. Then, Condition Modulated Cross-view Attention (CM-CVA) uses view-specific conditions to modulate each view’s features. The proposed method achieve well performance on the LIBERO-90 and AgiBot-World datasets, which demonstrates its effectiveness.

Critical strengths

1/ The paper is well-motivated. Robots are typically equipped with multiple cameras providing different viewpoints, making effective coordination of multi-view information an important research problem.

2/ This paper is well-written and easy to follow.

3/ The proposed ViewWeaver achieves strong performance across multiple views on both simulated and real-world datasets.

Critical weaknesses

1/ In the Introduction, the authors point out that existing methods struggle to establish correspondences between image regions and the same 3D content, as well as to model how shared physical motion manifests across different views. However, it remains unclear how the proposed method effectively addresses these challenges, as it does not explicitly incorporate 3D geometric information like depth maps or physical priors.

2/ Another concern is the technical novelty of the proposed method. Although the paper is well-motivated, both VSMC and CM-CVA appear to primarily rely on existing attention and modulation mechanisms, which have been widely adopted in previous works. The authors should further clarify the technical novelty of these two modules and better distinguish them from existing approaches.

3/ The proposed method is built upon Stable Video Diffusion (SVD), which was introduced in 2023. Why did the authors not consider more recent and advanced video generation models, such as Wan or Cosmos?

4/ A text instruction may correspond to multiple valid motion trajectories. However, the paper provides limited discussion on how the proposed method handles such ambiguity. It would be helpful if the authors could provide further analysis and discussion on this issue.

5/ Since VSMC is a core component of the proposed framework, the authors should provide more details about the camera-aware attention modules and the image-text attention module in the main text rather than relegating them to the appendix. The main paper should be sufficiently self-contained to allow readers to understand the key technical designs without referring to the appendix.

I will consider raising my rating according to the authors’ rebuttal and other reviewers’ comments.

Questions for authors

See Weaknesses

2 comments
AuthorsOpenReview ↗

Response to Reviewer vnoC - Thank you for reviewing our paper!

We sincerely appreciate your time and efforts in reviewing our paper! We provide detailed discussions based on your review:We sincerely appreciate your time and efforts in reviewing our paper! We provide detailed discussions based on your review:

Modeling Cross-View Content and Motion Consistency

ViewWeaver uses readily available camera and action information to model semantic–motion alignment and cross-view content interaction.

To enhance the consistency of image regions and 3D content, we integrate camera intrinsics and extrinsics to construct view-specific conditioning representations. CM-CVA further promotes scene and object consistency by modulating cross-view attention with embeddings encoding task intent and motion tendencies, encouraging information exchange about task-relevant objects and interactions.

To address physical motion consistency, we introduce VSMC which incorporates robot action information. Since one instruction can admit multiple trajectories, individual views may depict plausible but incompatible executions. VSMC grounds text in the current scene and aligns it with paired action representations, providing motion cues that encourage a unified execution across views.

Significance for Embodied Multi-View Generation and Comparison with Existing Approaches.

ViewWeaver improves multi-view consistency by exploiting complementary information from language and actions.

VSMC introduces action information specific to embodied scenarios to improve motion consistency. Embodied World Models such as GE and Cosmos Predict use text conditioning for multi-view generation. VSMC additionally aligns scene-grounded text with action representations to supply fine-grained motion cues. This addresses the ambiguity of multiple trajectories under the same instruction and encourages consistent motion across views.

Multi-view generation requires consistent objects and interactions. Existing models often concatenate views and exchange information through joint self-attention, which may distribute attention too diffusely. Robot arms and manipulated objects typically move against static backgrounds, making their cross-view consistency harder to maintain. We introduce condition-modulated cross-view attention. Through the preceding alignment, the conditioning representations already encode task intent and the robot’s motion tendencies. Modulating attention with these representations encourages cross-view information exchange to focus on task-relevant entities, particularly the objects being manipulated, improving multi-view consistency.

More broadly, we hope our work offers a design and training perspective for unified world models across modalities and views, as well as for visual, language, and action representations in vision-language-action models.

Applications on Other Backbone

Following Ctrl-World, we adopt SVD as our backbone. We further incorporate ViewWeaver’s design into Wan2.2 and Cosmos Predict2, training both variants in a single stage due to training-time constraints. The results are shown below.

For Wan2.2, head/hand FVD decreases by 8.9%/20.3%. For Cosmos Predict2, FVD decreases by 50.2%/58.5%. Both backbones also achieve higher PSNR,SSIM and lower LPIPS across views. These results demonstrate that ViewWeaver’s design can be applied across different generative backbones, and raise their performance on multi-view generation.

headhand
FVD↓PSNR↑SSIM↑LPIPS↓FVD↓PSNR↑SSIM↑LPIPS↓
text
wan48.420.140.8160.12173.517.720.6520.336
cosmos341.5520.550.8020.134144.9218.130.6290.328
viewweaver-wan44.0921.250.8320.09958.6018.450.6650.296
viewweaver-cosmos170.0720.850.8210.11360.1618.330.6670.306
action
viewweaver-wan43.6423.040.8580.07657.4419.610.6960.252
viewweaver-cosmos160.8323.760.8650.07846.7420.410.7200.234

Resolving Cross-View Motion Ambiguity with VSMC

A text instruction may correspond to multiple valid trajectories, which can lead to inconsistent motion across views. VSMC is designed to address this ambiguity. It grounds the instruction in the current scene and aligns the resulting text representation with the action representation in a shared latent space. This alignment incorporates fine-grained motion information into the text embedding, strengthening motion constraints across views and encouraging them to depict a consistent execution.

In Figure 3, Cosmos Predict follows the ground-truth motion in the Hand view but moves rightward instead of upward in the Third view. Each trajectory may satisfy the instruction individually, yet the views depict incompatible executions. ViewWeaver maintains consistent motion across both views through VSMC’s complementary semantic and motion representations.

Writing About VSMC

Thanks for pointing this out. We have revised the manuscript and added details about VSMC to the main text.

Thanks for the response. My concerns have been addressed, but I still don't think the paper meets the acceptance bar. I will keep my score unchanged.

Reviewer_GGqaOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a novel-view generation method on the robot manipulation benchmarks. This method has been built on top of Stable Video Diffusion and finetuned to predict both action-conditioned and text-conditioned novel-view generation. That is the only information that I catch through out the paper.

Critical strengths

S-1. Clear motivation.

I agree with the motivation of this work. It is important to achieve view-consistent generation as part of the world model design. Especially, this paper aims at multi-view world modeling and proposed method details are well aligned with the motivation.

S-2. Overall results are good.

The authors provide two main tables in the manuscript. Overall the proposed method outperforms previous works and it is good.

Critical weaknesses

W-1 Too simple.

I also prefer simple designs with clear motivation. The proposed method also followed this philosophy. However, the method itself is simply viewed as a multiview-finetuned diffusion model. While authors also newly designed both action-conditioned and text-condition generation pipeline, still it is simply a supervised finetuned approarch. There is not that much insights that I can achieve from this submission.

W-2 Missing qualitative results

I also read the supplementary material, but the authors did not include any qualitative results from other methods like recent video diffusion models. So it is highly not guarantee that the proposed method' generated novel-view quality is far better than other methods' results.

W-3 Separate evaluations on action-conditioned results vs. text-conditioned results.

Even though other methods can predict images only from action conditions or text conditions, it is necessary to show the clear differences in terms of qual / quant results. Tables 1 and 2 provide mixed results.

Questions for authors

Q-1. What about evaluation metrics for the robot manipulation task?

I am not sure about this but I hope to get responses from the authors.

While the authors mostly provide PSNR, SSIM, etc which measures the quality of rendered / generated images. However, as the authors also stated in abstract and introduction section, such an "imagination" should be helpful for the downstream task and for this paper the downstream task is the robot manipulation. However, there is no any other results about the robot manipulation task itself. Even LIBERO benchmark is originally designed for the robot manipulation task. So for me, it is quite weird whether the proposed method only focus on multi-view consistent view generation or multiview world modeling.

3 comments
AuthorsOpenReview ↗

Response to Reviewer GGqa - Thank you for reviewing our paper!

We thank the reviewer for their time and feedback. We address the concerns with additional experiments and discussion.

Multi-view Consistency in Embodied Scenarios.

Building on multi-view supervision, ViewWeaver addresses inconsistencies in actions, interactions, and scenes across robot views in embodied scenarios.

First, ViewWeaver investigates how to combine text with the current scene and associate it with a specific motion trajectory, thereby resolving ambiguity in the task description and ensuring consistent generated motions across views. In embodied scenarios, the same text in the same scene may correspond to multiple plausible motion trajectories. Without specific motion constraints, this can lead to ambiguity in the motions generated across different video views. Additional information is therefore needed to constrain generation and maintain consistency. The action modality inherent to embodied scenarios can provide precisely such fine-grained constraints.

Second, beyond motion consistency, consistency in objects and interactions with them across views is also required. In embodied scenarios, the robot arm and the objects being manipulated are typically in motion, while the background and other objects remain stationary. Maintaining consistency across views is more challenging for these moving objects than for stationary ones. Therefore, instead of directly applying cross-view attention, we adopt condition-modulated attention. Through the preceding alignment, these conditions already contain information about task intent and the robot’s motion tendencies, allowing cross-view attention to focus more on task-relevant objects being interacted with and further enhancing multi-view consistency.

Furthermore, we aim to provide an alternative module design and training scheme for a unified multimodal, multi-view world model that supports both text and action conditioning.

More Qualitative Results

Figure 3 provides qualitative comparisons with Cosmos under text conditioning and Ctrl-World under action conditioning. We further include video comparisons with Cosmos Predict, Wan2.2, OpenSora and Ctrl-World in the supplementary material.

In the “push the capsule machine button” example, Cosmos predicts gripper retraction in the head and right-hand views, while the gripper remains extended in the left-hand view. Our model consistently depicts button pressing followed by gripper retraction across views. In the “pick up the alphabet soup and put it in the basket” example, OpenSora produces substantial object deformation in the hand view, while Wan2.2 predicts noticeably faster motion in the hand view than in the head view. Our model maintains synchronized motion across both views. More visualizations appear in our supplements.

Seperate Evaluations

We compare against models that support both text and action conditioning, such as iVideoGPT, as well as models that support only one modality, including text-conditioned models such as GE, Cosmos Predict2.5, and Stable Video Diffusion, and action-conditioned models such as IRASim and Ctrl-World. We also adapt the strongest baseline, Ctrl-World, into a text-conditioned variant, Ctrl-World*, for comparison.

Tables 1 and 2 report independent text-conditioned and action-conditioned evaluations, presented together for comparison using the same datasets and metrics. We additionally evaluate Wan2.2 and OpenSora on LIBERO below and provide video comparisons in the supplementary material.

headhand
FVD↓PSNR↑FVD↓PSNR↑MEt3R↓FVD↓PSNR↑FVD↓
OpenSora24318.620.7460.20834916.690.6100.454
wan2.248.420.140.8160.12173.517.720.6520.336
ViewWeaver16.6022.090.8560.09157.1819.000.7040.279

Downstream Using in Manipulation Tasks

We evaluate downstream manipulation on LIBERO using a lightweight inverse dynamics model (IDM) to translate predictions into robot actions. We evaluate task success rates on four tasks selected from LIBERO. ViewWeaver-Single uses a single view as input to the IDM, whereas ViewWeaver and Cosmos Predict use two views.

As shown in the table, ViewWeaver achieves an average success rate of 75%, exceeding ViewWeaver-Single by 52.5%. This improvement may stem from the additional wrist-camera view, which helps the robot accurately estimate the spatial relationship between the gripper and the target object. These results suggest that multiple views provide complementary information that improves downstream task. Furthermore, ViewWeaver outperforms Cosmos Predict as a planner by 12.5 percentage points, suggesting that more consistent multi-view predictions can lead to better performance on downstream manipulation tasks.

Close bottom drawerClose top drawerOpen MicroWaveTurn on the stoveAverage
ViewWeaver-single40455022.5
Cosmos Predict401001010062.5
ViewWeaver851001510075
Reviewer_GGqaOpenReview ↗

Thank you for the rebuttal.

Overall, the authors well resolve the issues that I addressed in my review. I have no further questions about this. However, I still have my impressions that the work itself is too simple. Also the reviewer vnoC addressed the technical novelty issue. Do the authors can answer to this question?

AuthorsOpenReview ↗

Response to Reviewer GGqa - Thank you for the question!

We sincerely thank the reviewer for acknowledging that our rebuttal has addressed the previous concerns. Regarding the reviewer's further concerns about technical novelty, we would like to elaborate on ViewWeaver's contributions from the following two perspectives.

  1. At the technical level, ViewWeaver is a carefully designed approach rather than a simple combination of existing modules.

    We first explored simple multi-view approaches, such as concatenating multi-view inputs in Ctrl-World (FVD: FVD LIBERO hand 192.71) and cross-view attention in Genie Envisioner (FVD LIBERO hand 172.06). However, simple multi-view concatenation relies on the model to implicitly learn cross-view relationships, while conventional cross-view attention lacks explicit guidance from task and motion information. So these methods substantially underperform our method (FVD LIBERO hand 57.18).

    To address this problem, we first explored conventional Cross-View Attention (FVD: 128.46), hoping to improve cross-view consistency, but the results remained unsatisfactory. We then experimented with Joint Spatiotemporal Attention (153.00) to incorporate temporal information, but it failed to improve performance and even introduced negative effects. We further investigated Deformable Attention (130.00) to enable adaptive spatial attention, yet the improvement was still limited. These explorations led us to recognize the importance of conditioning information in guiding cross-view interactions. We therefore proposed CM-CVA, achieving the best FVD of 116.74.

  2. At the methodological level, ViewWeaver not only improves the multi-view performance, but also generalizes to different video world models.

    ViewWeaver reduces the FVD on LIBERO hand of SVD from 230.33 to 57.18. When transferred to Wan2.2, the hand-view FVD on LIBERO reduced from 73.50 to 58.60. We further transferred to Cosmos Predict2, the FVD reduced from 144.92 to 60.16.

We sincerely thank the reviewer again for their thoughtful feedback and hope this clarification better conveys the novelty and significance of our work.

Reviewer_MrY2OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

A multi-view video world model, initialized from Stable Video Diffusion, that supports both text- and action-conditioned prediction. VSMC adds per-view camera information to the conditions and aligns text and action latents. CM-CVA lets condition-modulated queries attend to the other views. Validation covers LIBERO-90 and an 8-task AgiBot-World subset with FVD, PSNR/SSIM/LPIPS, MEt3R, and trajectory accuracy, plus ablations.

Critical strengths
  • Consistent gains in both modes. It has the best FVD in every view and setting (e.g., LIBERO text third-view 16.60 vs 48.96) and the lowest MEt3R in all four settings.
  • Alignment closes the text-action gap. Alignment alone cuts text-conditioned third-view FVD from 80.20 to 43.68. The LIBERO third-view text-vs-action gap is 3.70 vs 14.39 for Ctrl-World. This is a simple, transferable idea.
Critical weaknesses
  • Cross-view consistency gains are not isolated. The ablation omits MEt3R, CM-CVA is not compared with plain cross-view attention, and MEt3R margins on AgiBot are small (0.607 vs 0.609). CM-CVA’s role in consistency is therefore unclear.
  • Limited evaluation scale:
    • Resolution is low (up to 192x256).
    • There are only 40 real test trajectories.
    • The text baseline (Ctrl-World*) is the authors’ own adaptation.
    • There is no downstream policy or planning use.
Questions for authors
  • Can you report MEt3R for each ablation row and compare CM-CVA with unmodulated cross-view attention?
  • Were all baselines trained on the same data, resolution, and history, and how was Ctrl-World* tuned?
  • What is the variance across seeds on AgiBot-World?
1 comment
AuthorsOpenReview ↗

Response to Reviewer MrY2 - Thank you for reviewing our paper!

We sincerely appreciate your time and effort in reviewing our paper! Following your feedback, we conducted additional experiments:

More Details in Ablation Study.

We present the MEt3R scores of the ablation study in the table below. Adding VSMC makes MEt3R better, and adding CM-CVA lowers it further, indicating that both modules improve cross-view consistency.

MEt3R ↓
textaction
SVD0.7520.722
+alignment0.6930.665
+VSMC0.6650.614
CVA0.6470.594
CM-CVA0.6320.577
all-in-all0.7400.622

We added a comparison with plain CVA on LIBERO. CM-CVA reduces third/hand-view FVD from 35.79/128.46 to 30.33/116.74 under text conditioning, and from 19.11/88.66 to 18.49/85.61 under action conditioning.

headhand
FVD↓PSNR↑FVD↓PSNR↑MEt3R↓
Text-conditioned
CVA35.7921.64128.4616.680.647
CM-CVA30.3321.88116.7418.930.632
Action-conditioned
CVA19.1124.5188.6620.700.594
CM-CVA18.4925.1985.6121.040.577

Variance Across Seeds on Agibot.

We report the performance of our model on AgiBot under four different seeds. The results show that performance is relatively stable across seeds. Specifically, our MEt3R on the text and action conditions is 0.607 ± 0.0005 and 0.607 ± 0.0008, respectively, which shows that our model's MEt3R is stable across repeated runs.

HeadLeftRight
SeedFVDPSNRSSIMLPIPSFVDPSNRSSIMLPIPSFVDPSNRSSIMLPIPSMEt3R
Text
056.7824.9550.8690.10873.1124.2600.8240.16292.4325.7100.8170.1940.607
42 (paper)58.3224.8500.8700.10975.7624.1200.8250.16498.5625.6100.8180.1940.606
6655.5024.9700.8700.10874.7724.1900.8230.16493.1225.7200.8170.1930.607
123456.4724.9800.8700.10874.6724.2600.8240.16394.2025.6900.8170.1930.607
Mean56.7724.9390.8700.10874.5824.2080.8240.16394.5825.6830.8170.1940.607
Std.1.170.0600.00050.00051.100.0670.00080.00102.750.0500.00050.00060.0005
Action
056.7725.0100.8900.10875.9224.2800.8240.16293.1325.6900.8170.1940.606
42 (paper)57.1824.9400.8710.10876.9124.2000.8270.16296.9525.6000.8170.1940.607
6656.0925.0200.8700.10873.6924.3000.8240.16192.0025.6800.8160.1940.607
123456.4125.0000.8700.10875.8224.2600.8240.16192.0925.7600.8170.1930.608
Mean56.6124.9930.8750.10875.5924.2600.8250.16293.5425.6830.8170.1940.607
Std.0.470.0360.00980.00001.360.0430.00150.00062.330.0660.00050.00050.0008

Visual Resolution for Embodied Systems.

Embodied systems often need to respond quickly, so moderate image resolutions are common. For example, π₀ uses 224 × 224 images as input, while DreamZero uses 176 × 320. Embodied world models used to generate training data or serve as policy backbones also operate at comparable resolutions. Ctrl-World and UniSim use 192 × 320, iVideoGPT uses 256 × 256, and Interactive World Simulator uses 128 × 128. Our choice of 192 × 256 for AgiBot-World is therefore consistent with established practice for the manipulation settings.

The Total Size of Evaluation Benchmark.

The videos in AgiBot-World are usually long. Our 40 AgiBot-World test trajectories average 58 seconds each, totaling approximately 2,300 seconds. For context, Ctrl-World evaluates video generation on 256 clips of 10 seconds, totaling 2,560 seconds.

Training recipe for baselines and Ctrl-World*.

Baseline training. We initialized each baseline from its released pretrained checkpoint and fine-tuned it using the released code. While finetuning, all baselines used the same data, image resolution, and history frames.

Ctrl-World*. The original Ctrl-World supports action conditioning only. To obtain a text-conditioned baseline, we replaced its action encoder with a CLIP text encoder that encodes the task instruction.

Downstream Application.

To evaluate whether multi-view predictions improve downstream task execution, we used a simple inverse dynamics model to convert the world model’s predicted future observations into robot actions. We evaluated this approach on four tasks on LIBERO and reported the success rates in the table below.

ViewWeaver-Single uses only a single view as input to the IDM for downstream tasks, whereas ViewWeaver and Cosmos Predict use two views as input. As shown in the table, ViewWeaver achieves an average success rate of 75%, improving by 50% over ViewWeaver-Single. Compared with using Cosmos Predict for planning, ViewWeaver achieves a 12.5% improvement, indicating that more consistent multi-view generation can lead to better downstream task performance.

Close bottom drawerClose top drawerOpen MicroWaveTurn on the stoveAverage
ViewWeaver-single40455022.5
Cosmos Predict401001010062.5
ViewWeaver851001510075

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.