acceptodds
Under review as a conference paper at ICLR 2027

WEAVE-0.5: Weaving Multi-stream Observations into Embodied World Models

Abstract

Visual intelligence provides the perceptual grounding for embodied agents to learn how the physical world evolves, calling for world models that aggregate diverse and complementary visual evidence. However, interfaces built around fixed multi-view layouts or modality-specific branches constrain joint modeling across variable camera configurations, resolutions, modality sets, and observation conditions. We argue that embodied world models should represent heterogeneous observations as visual streams and flexibly predict requested streams from the available visual context. We present , an exploration of multi-stream embodied visual world modeling. Each stream corresponds to one visual modality from one viewpoint and is independently encoded at its native resolution. Streams are organized on a shared virtual canvas and jointly processed with a video diffusion transformer. Canvas coordinates support multi-view and multi-resolution composition while retaining the pretrained spatiotemporal rotary positional encoding (RoPE) structure, whereas diffusion timesteps specify observed and target content. This formulation unifies future prediction, cross-view completion, modality conversion, and joint multi-view generation within a shared architecture without task-specific prediction heads. Built on Wan2.2-TI2V-5B, WEAVE-0.5 is jointly trained on real-robot, simulated, egocentric, and general videos. Component interventions support a contribution of cross-stream attention to RGB prediction of views without their own initial observations. Benchmark-specific WEAVE-0.5 checkpoints rank first in the reported RBench leaderboard snapshot with an average score of 0.642 and achieve 82.9 overall on PAI-Bench-G. These results support multi-stream visual modeling as an effective approach to integrating heterogeneous visual observations for embodied world modeling.

Primary area:
generative models
Keywords:
Embodied World Model, Multi-View video Generation, Multi-Stream Generation, Video Diffusion Models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_8G2sOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

WEAVE-0.5 proposes a unified multi-stream visual world model that represents different visual information, including RGB, multi-view observations, Depth, and Segmentation, as composable streams. It leverages a Virtual Canvas and a shared diffusion transformer to enable future prediction, cross-view completion, and cross-modal generation. The method achieves strong performance on RBench and PAI-Bench-G, demonstrating the potential of multi-stream visual modeling.

Critical strengths
  1. Strong experimental performance and effective validation WEAVE-0.5 achieves strong performance on embodied video generation benchmarks such as RBench and PAI-Bench-G. The ablation studies further validate the effectiveness of key components, such as cross-stream attention, for multi-view information fusion.

  2. Innovative Virtual Canvas design The paper introduces Virtual Canvas, which organizes different visual streams through a unified spatial coordinate system. By preserving the RoPE structure of the pretrained video model, it enables efficient fusion of multi-view, multi-resolution, and multi-modal information, improving the flexibility and scalability of multi-stream visual modeling.

Critical weaknesses

1.The necessity of Virtual Canvas is not sufficiently validated. Virtual Canvas is one of the core components of the paper, but there is currently a lack of direct ablation studies to evaluate its effectiveness, such as comparisons with direct token concatenation, standard view embeddings, or alternative spatial layout strategies. Therefore, it is difficult to fully determine whether the performance improvements mainly come from the Virtual Canvas design itself, rather than other factors such as larger-scale training data, a stronger pretrained backbone, or other model components.

2.The practical value of auxiliary modalities is insufficiently demonstrated. The paper introduces additional visual streams, including Depth, Segmentation, and Edge, to achieve unified multi-modal visual modeling. However, the current experiments mainly focus on RGB video generation and do not sufficiently demonstrate whether these auxiliary modalities improve world understanding, physical reasoning, or downstream robotic tasks. Moreover, the results show that generating Depth provides limited improvement for RGB prediction, suggesting that the actual contribution of auxiliary modalities to model capability still requires further investigation.

Questions for authors

1.Lack of ablation study for Virtual Canvas Virtual Canvas is a core component of the proposed unified multi-stream modeling framework, but there is currently a lack of direct ablation studies evaluating its contribution. Could the authors further demonstrate the performance gains brought by Virtual Canvas itself? For example, compared with direct token concatenation, view embeddings, or other multi-view spatial encoding strategies, does Virtual Canvas provide clear advantages in terms of generation quality, multi-view consistency, or model scalability?

2.On the practical contribution of auxiliary modalities The paper introduces auxiliary visual streams, including Depth, Segmentation, and Edge. However, these modalities are mainly generated from RGB data using additional models, which may introduce errors from upstream models. Could the authors further demonstrate that these auxiliary modalities provide benefits beyond expanding the model’s output capabilities, such as improving RGB prediction quality or downstream robotic task performance? Furthermore, the experimental results show that generating Depth provides limited improvement for RGB prediction, suggesting that the practical contribution of auxiliary modalities to the core task remains unclear. Could the authors further analyze the specific benefits brought by incorporating these modalities?

3.On the advantages of unified multi-stream modeling over specialized models The paper proposes a unified multi-stream framework to jointly model RGB, Depth, Segmentation, and other visual information. However, it remains unclear whether such a unified approach can outperform specialized models optimized for individual tasks. For example, dedicated Depth models such as VGGT may achieve better prediction quality and inference efficiency for depth estimation. Could the authors further demonstrate the practical advantages of WEAVE-0.5 over specialized models in terms of generation quality, inference efficiency, or downstream embodied task performance?

Reviewer_ubnCOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

WEAVE-0.5 is a multi-stream embodied video world model on Wan2.2-TI2V-5B. Each camera stream is encoded at native resolution and packed into a single virtual-canvas sequence with per-view offsets and scale factors, so the backbone's attention fuses views without architectural changes. In addition, a frozen Qwen3.5-9B provides textual memory via a trainable adapter, and per-frame diffusion timesteps handle clean, prefix, and diffuse roles. Training follows a five-stage curriculum on 87,700 hours of data.

Critical strengths
  1. Packing native-resolution streams into one sequence gives cross-view attention at a quantified cost. At three views, the design uses 18,040 tokens versus 36,900 for pixel-stitching. At seven views, 29,520 versus 86,100, the difference between fitting the 50K per-rank budget and not. The scale factor with a virtual offset handles heterogeneous intrinsics without resampling, and Proposition 1's offset-invariance result makes the construction well-defined rather than ad hoc.

  2. The attention ablation is targeted. Tables 10 and 11 vary the cross-stream attention mask at a fixed checkpoint. Table 11 gives the per-view decomposition: the two wrist views score 14.12 and 14.29 dB with attention on and 10.83 and 11.17 with it off, while the main view is essentially unchanged. That is exactly the pattern a cross-view mechanism should produce, since the main view has its own observation and the wrists do not, and the identity-table ablation shows the view embeddings carry signal.

Critical weaknesses
  1. The paper presentation is poor. I do not think this paper is well-formed. The experiments section only includes two main quantitative results and one qualitative result. I understand that the authors include many other experiments in the appendix, and I do check them. However, such a presentation is clearly not ready for publication in ICLR.

  2. The checkpoints used in different experimental sections are different. In the ablation study, the authors use the 5,000-step R0 checkpoint on two non-public sets. Neither the final model's multi-view scores nor the benchmark checkpoints' component behavior is reported, so I can only see that a certain WEAVE-0.5 (not sure how many steps) ranks first and that a short-budget one predicts wrists better with attention, but I’m not sure the architecture is the biggest contribution.

  3. The depth doesn’t show any advantage. Adding generated depth changes nothing measurable. For example, in the ablation study, C−B is 0.0212, and D−A is 0.0129, both quite marginal. Therefore, I do not think the depth-related contribution is well established.

Questions for authors
  1. Can you report the attention on/off ablation at the final checkpoint rather than R0?

  2. In Tables 12 and 13, the ablation study B−A provides the wrists with their first-frame observation, so it measures the value of the observation as much as the value of cross-stream attention. Can you provide an additional study that holds the wrist's own observation fixed and varies only whether other views are visible?

  3. The paper reports that adding generated depth does not improve RGB quality, with all three intervals including zero. What is the case for the modality part of the multi-stream claim?

  4. Imaging Quality drops from 69.6 to 62.6. However, the PAI-Bench-G Quality average drops only 0.9. Can you tell which sub-metrics compensate, and is a seven-point loss on the sub-metric most tied to per-frame fidelity acceptable for the intended use?

  5. Table 10's identity-table zeroing costs 1.09 dB. On which views does that loss fall, and is it consistent with the view embeddings encoding camera identity rather than content?

Reviewer_8N5XOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

WEAVE-0.5 extends Wan2.2-TI2V-5B into a unified multi-view, multi-modal video predictor. It independently encodes visual streams, organizes their tokens on a virtual canvas preserving pretrained RoPE, and uses diffusion timesteps to specify observations and targets. This enables future prediction, cross-view completion, modality conversion, and joint generation within one architecture. Trained on massive robot, simulated, egocentric, and general videos. Qualitative examples and internal ablations support cross-stream information sharing, but closed-loop robot control is not evaluated.

Critical strengths
  • The virtual-canvas RoPE design is a natural approach to supporting multi-view inputs and outputs.
  • The model is trained on large-scale data with substantial compute and shows promising potential for scaling multimodal output prediction, making this a valuable direction for future research.
Critical weaknesses
  • The paper lacks clear ablation evidence demonstrating the benefits of replacing T5 with a VLM, particularly for instruction understanding and physical understanding.
  • No video demonstrations are provided. I consider this the paper’s most significant weakness, given that its central task is video generation
  • The metrics used in the ablations are not fully appropriate for this task. The PSNR ablations demonstrate fidelity to recorded futures, but because generation is instruction-conditioned and action trajectories are unspecified, they do not adequately establish improvements in general generative or world-model quality. The authors should include metrics such as 3D consistency to assess the quality of the generated videos.
Questions for authors

How many distinct valid input–output combinations does the current model support beyond the examples shown in the paper? What failure cases or OOD cases have been observed?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revised manuscript: controlled ablations and evaluation details

Thank you for the helpful feedback. We have uploaded a revised manuscript. New and substantially revised passages and table captions are marked in light blue; table entries remain black for readability.

  • Section 4.2, Table 3 (p. 9): Key controlled ablations now appear in the main text. Panel A compares layouts under matched training from the end-of-S2 checkpoint, before multi-stream training. Panel B reports attention and identity interventions at the same final S5 multi-stream checkpoint, without retraining.
  • Appendices 6.4.2–6.4.5, Tables 11–14 (pp. 26–27): We provide the common evaluation protocol, full training and inference comparisons, per-view results, and joint-prediction and structure-conditioning results.
  • The Reproducibility Statement (p. 11) now points to these updated sections.

These revisions incorporate the key results presented in our shared comment, Controlled ablations at S2 and S5. The overview video remains available under Supplementary Material.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.