- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
World in World proposes a training-free method to make full use of pretrained world models’ visual and motion prior, extending its controllability. It introduces a unified visual evidence interface to guide generation through native self-attention, with additional mechanisms to improve spatial alignment and control the influence of different evidence.
Critical strengths
WiW solved a very interesting problem. For world models, visual evidence comes from diverse sources. Most existing methods inject new visual evidence in a modular way, but this is obviously not unified enough and not elegant enough. WiW proposes a clever idea: the causal self-attention of the existing video world model itself can read the historical clean visual states. Then, other visual evidence can also be transformed into a similar form and injected into the model through the existing attention mechanism. In this way, there is no need to design and train new control modules for different visual evidence, and at the same time, the existing prior knowledge of the pre-trained model can be fully utilized. I think this idea is very concise and elegant, and it's also training-free. This is my favorite part of this paper.
In addition, the author also conducted relatively thorough experiments. On DAVIS and OpenVid-1M, WiW has achieved good results in both video quality and camera control accuracy, especially in the case of significant changes in viewing angles, it can still generate relatively consistent videos. The article also showcases different applications such as bullet time and video editing, which to some extent demonstrates the flexibility of this unified interface. I think these experiments have demonstrated the potential of this idea quite well.
Critical weaknesses
In my opinion, one of the main problems with WIW is its reliance on external geometric estimates. While the idea of injecting visual evidence with self-attention is elegant, practical implementation still requires additional depth estimation, camera pose estimation, point tracking, and even human geometric reconstruction. This weakens the simplicity of the method to some extent, and also makes the whole pipeline heavy. More importantly, the performance of the method seems to depend heavily on this geometric information. As you can see from Table 2, there is a significant performance degradation with target-view warping removed, but a relatively small performance degradation with CGAR removed. This makes me wonder how much of the improvement comes from the author's attention mechanism and how much from the strong constraints provided by the external geometric information?
In addition, I am concerned about the robustness of the method to geometric estimation errors. Geometric estimation may be inaccurate under large-scale view changes or complex dynamic scenarios, and these erroneous geometric information will be injected into the model as visual evidence, which may affect the subsequent generation and long-range consistency. Although the authors compare different depth estimators in the Appendix and also present a 220-frame long-horizon revisit, it is mainly a qualitative experiment. It would be more convincing if some quantitative experiments on geometric estimation errors and the stability of long-range rollout could be added.
Questions for authors
I wonder whether the errors introduced by the external geometric estimation module will gradually accumulate in the long-term rollout, thereby leading to increasingly obvious scene drift. Although the author presented the qualitative results of 220 frames, I would prefer to see some systematic quantitative analyses, such as how the geometric and visual consistencies change as the rollout length increases, especially in cases where there are estimation errors in depth, camera pose, or point tracking.
3 comments
Thank you for your detailed reply. The new results have resolved most of my concerns. I will keep my score.
Thank You for Your Constructive Feedback
We greatly appreciate the time and effort you have invested in carefully reviewing our work. We sincerely appreciate your recognition of the novelty of our work and your constructive suggestions, which have helped us improve the paper. We are glad that our additional results have addressed most of your concerns. Please feel free to let us know if you have any further questions or concerns.
Response to Reviewer DkGE
We sincerely thank Reviewer DkGE for recognizing our unified evidence interface.
1. External geometry and overhead. We agree WiW uses external geometry, common in rerendering: geometry provides the spatial reference needed to project observed source content and its time-varying motion into new camera views. Notably, WiW supports diverse visual evidence via the same frozen attention interface, selectively used by task rather than requiring all geometry at once (e.g., human geometry is mainly for 180°/360° rotations to guide unseen surfaces). For efficiency, Appendix A reports that for matched frames, depth/pose estimation (0.084 s/frame) and point tracking (0.098 s/frame) total just 13.1% of WiW's inference time (1.389 s/frame, Table A2). Our newly measured LHM++ and SMPL-X take 0.108 and 0.049 s/frame/person (by subject count). Preprocessing runs once per source video and is reused across target paths, adding only modest one-time overhead.
2. Contribution of geometry and attention. Target-view warping is indeed vital, providing pixel-level references for how the scene evolves over time under the target camera; without it, generation becomes unconstrained (Fig. A3(a)). Besides providing the interface for warped videos to enter the frozen model, WiW improves how the model reads and uses this evidence. With identical geometry, EWA boosts PSNR from 21.52 to 22.98 dB on DAVIS (Table 2) and from 19.59 to 20.60 dB under large rotations (Table A1). CGAR routes attention to corresponding unwarped source features so generated subjects follow source appearance and pose at that event time. To isolate this, we added ablations with identical source K/V against ordinary attention and routing-only (Table R1 in Common Response), and tested 50 high-motion pairs (Table R2 in Common Response), showing larger CGAR gains. These results separate geometry's spatial constraints from the extra gains of our attention mechanisms.
3. Robustness to geometric errors and long-rollout stability. To test sensitivity to depth estimators, we first compared DepthCrafter, ViGeo, and VDA on the 200 DAVIS pairs with fixed references (Table R7), verifying WiW's robustness. For long rollouts, depth, poses, and tracks are preprocessed once over the full source video rather than re-estimated chunk by chunk; thus, while these priors may have estimation errors, this largely avoids autoregressive error accumulation in the control signals. To test robustness to imperfect geometry and if errors still propagate via autoregressive generation, we built 50 long sequences (180-220 frames: rotation, translation, orbit) from 25 OpenVid-1M videos. We perturbed depth (randomly dropping 5%/10% of values), camera poses (randomly picking 5%/10% of poses for a 10% multiplicative rotation/translation perturbation), and point tracks (randomly dropping 5%/10% of tracks) and compared WiW with streaming 4D InSpatio-World (ISW) under matched settings. Table R8 shows WiW degrades more gradually as errors increase and outperforms ISW in visual fidelity and camera control. We also measured fidelity and camera errors at early (T₁), mid (T₂), and late (T₃) rollout stages with clean and perturbed geometry against ISW (Table R9). In Tables R8 and R9, even with input errors, WiW keeps more stable visual and camera consistency over long sequences, resisting autoregressive error accumulation. We hope these new results fully address your concerns.
Table R7. Depth estimators on 200 DAVIS pairs.
Table R8. Robustness under geometric perturbations.
Table R9. Long-rollout consistency across stages T₁-T₃.