- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
WEAVE-0.5 proposes a unified multi-stream visual world model that represents different visual information, including RGB, multi-view observations, Depth, and Segmentation, as composable streams. It leverages a Virtual Canvas and a shared diffusion transformer to enable future prediction, cross-view completion, and cross-modal generation. The method achieves strong performance on RBench and PAI-Bench-G, demonstrating the potential of multi-stream visual modeling.
Critical strengths
-
Strong experimental performance and effective validation WEAVE-0.5 achieves strong performance on embodied video generation benchmarks such as RBench and PAI-Bench-G. The ablation studies further validate the effectiveness of key components, such as cross-stream attention, for multi-view information fusion.
-
Innovative Virtual Canvas design The paper introduces Virtual Canvas, which organizes different visual streams through a unified spatial coordinate system. By preserving the RoPE structure of the pretrained video model, it enables efficient fusion of multi-view, multi-resolution, and multi-modal information, improving the flexibility and scalability of multi-stream visual modeling.
Critical weaknesses
1.The necessity of Virtual Canvas is not sufficiently validated. Virtual Canvas is one of the core components of the paper, but there is currently a lack of direct ablation studies to evaluate its effectiveness, such as comparisons with direct token concatenation, standard view embeddings, or alternative spatial layout strategies. Therefore, it is difficult to fully determine whether the performance improvements mainly come from the Virtual Canvas design itself, rather than other factors such as larger-scale training data, a stronger pretrained backbone, or other model components.
2.The practical value of auxiliary modalities is insufficiently demonstrated. The paper introduces additional visual streams, including Depth, Segmentation, and Edge, to achieve unified multi-modal visual modeling. However, the current experiments mainly focus on RGB video generation and do not sufficiently demonstrate whether these auxiliary modalities improve world understanding, physical reasoning, or downstream robotic tasks. Moreover, the results show that generating Depth provides limited improvement for RGB prediction, suggesting that the actual contribution of auxiliary modalities to model capability still requires further investigation.
Questions for authors
1.Lack of ablation study for Virtual Canvas Virtual Canvas is a core component of the proposed unified multi-stream modeling framework, but there is currently a lack of direct ablation studies evaluating its contribution. Could the authors further demonstrate the performance gains brought by Virtual Canvas itself? For example, compared with direct token concatenation, view embeddings, or other multi-view spatial encoding strategies, does Virtual Canvas provide clear advantages in terms of generation quality, multi-view consistency, or model scalability?
2.On the practical contribution of auxiliary modalities The paper introduces auxiliary visual streams, including Depth, Segmentation, and Edge. However, these modalities are mainly generated from RGB data using additional models, which may introduce errors from upstream models. Could the authors further demonstrate that these auxiliary modalities provide benefits beyond expanding the model’s output capabilities, such as improving RGB prediction quality or downstream robotic task performance? Furthermore, the experimental results show that generating Depth provides limited improvement for RGB prediction, suggesting that the practical contribution of auxiliary modalities to the core task remains unclear. Could the authors further analyze the specific benefits brought by incorporating these modalities?
3.On the advantages of unified multi-stream modeling over specialized models The paper proposes a unified multi-stream framework to jointly model RGB, Depth, Segmentation, and other visual information. However, it remains unclear whether such a unified approach can outperform specialized models optimized for individual tasks. For example, dedicated Depth models such as VGGT may achieve better prediction quality and inference efficiency for depth estimation. Could the authors further demonstrate the practical advantages of WEAVE-0.5 over specialized models in terms of generation quality, inference efficiency, or downstream embodied task performance?
Revised manuscript: controlled ablations and evaluation details
Thank you for the helpful feedback. We have uploaded a revised manuscript. New and substantially revised passages and table captions are marked in light blue; table entries remain black for readability.
These revisions incorporate the key results presented in our shared comment, Controlled ablations at S2 and S5. The overview video remains available under Supplementary Material.