- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes WorldNoise, a method that aims to add camera control to video diffusion models and improve scene consistency via structured 3D noise. The motivation is that most generative video models start from entirely random noise that is uncorrelated across frames even when frames view the same content. The paper's solution is to instead construct a 3D noise space that is shared across all views.
The method first looks at each camera frustum along a predefined trajectory to determine the 3D area that needs to be filled with noise. To keep the number of noise specks manageable, the method scatters a fixed number from each camera viewpoint, sampling uniformly in image coordinates and logarithmic depth. The specks are then sized according to the expected local spacing derived from the sampling density. That density drops with the cube of depth, such that specks are small and capture fine detail up close and are larger far away. Each speck is given a random Gaussian value that is fixed across views.
The 3D noise cloud is then converted into 2D frame noise via EWA splatting (similar to 3DGS but without alpha blending). This leads to blotchiness that is different from the iid Gaussian noise diffusion models typically see, as each speck's Gaussian value is spread across the pixels it occupies. A noise residual is therefore added that cancels this within-frame correlation, variance is then normalized, and the resulting noise is spatially iid within a frame. As the residuals are drawn independently for each view, cross-view correlation remains intact.
WorldNoise can add camera control capabilities to a pre-trained video model by LoRA-finetuning it to align with the structured noise distribution (the architecture and training objectives remain unchanged). The paper also proposes a long-horizon generation extension that partitions noise into "temporary" specks (the 3D noise previously described) and "history" specks that reside on surfaces that the camera has already observed (as determined by DA3). The initial history comes from the first frame's depth and later surfaces are added from generated keyframes if they pass a confidence and consistency check. To generate a new chunk, temporary specks fill the upcoming camera views and are filtered by geometric carving rules. The generation is also conditioned on the first frame, 4 retrieved past frames with good view overlap, and the most recent 16 frames. After a frame chunk is generated, the temporary noise is discarded.
The method is integrated into Wan 2.1-14B and LTX-2.5-22B and LoRA-finetuned on DL3DV-10K. The paper evaluates 81-frame sequences from 100 RealEstate10K and 100 DL3DV-10K scenes by measuring camera control (via GLOMAP reconstruction compared to the requested trajectory), geometric consistency (TSED), and visual quality. The main baselines are camera-control methods (SCoPE, Gen3C, UCM), other noise methods (CameraNoise and UniCaMo Wan 2.1-14B reimplementations trained on the same data as WorldNoise), and the base Wan and LTX models. Compared to the camera-control methods, the method generally has the lowest rotation error but higher translation error than SCoPE (both datasets) and UCM (RealEstate10K) and worse TSED score than UCM. Compared to the noise-based methods and the base models, it does much better. Long-horizon generation is evaluated on sequences from DL3DV-10K longer than 300 frames and compared to SANA-WM and Lyra 2.0. WorldNoise performs best on most camera and geometry metrics (the margins are small) with comparable visual quality. The point sampling budget is ablated (more points generally gives better metrics) as is the log-depth sampling (which matters more for translation error and less for rotation error).
Critical strengths
I find the core idea of the paper (how to structure noise in 3D in a manner that improves video diffusion) to be an interesting and neat topic. Although there is prior work that discusses how to use 3D consistent noise generally and for camera-control specifically, this is the first that I'm aware of that does so across multiple depths without needing the underlying 3D scene geometry (at least in short-horizon settings), which is a valuable contribution. More generally, how the noise is constructed seems principled and well-defined. The idea is also relatively simple to implement and doesn't require model architecture changes, which is practical.
The evaluation section does a convincing job of showing that the method works well in comparison to some other noise-based schemes and baselines, validating the need for principled 3D consistent noise, along with evidence of generalization (training on DL3DV-10K and evaluating on RealEstate10K).
Critical weaknesses
Some of the novelty in the paper is overstated such as the abstract's claim to reveal "a previously overlooked role of diffusion noise as a latent carrier of 3D structure." ConsistDreamer [1], CFD [2], and MultiDiff [3] (none of these are cited) structure noise via 3D scene geometry, with MultiDiff doing so to enhance camera control (although it also uses warped-image conditioning). The core novelty of this submission then becomes deriving 3D noise across multiple depths without known scene geometry (and the novelty of the long-horizon approach, which does require it, diminishes). This is still valuable, but the authors should take care to properly attribute prior work and contextualize their claims accordingly (How I Warped Your Noise [4] should also be cited as its Eq. 6 and Eq. 2 are very similar to how this paper derives its noise residual). In the same vein, the "conclusion and limitations" section does not contain any discussion of limitations. Several immediately come to mind (the camera trajectory has to be known beforehand and the 3D noise construction assumes the scene is static). The runtime overhead of the noise construction strategy should also be discussed.
The evaluation protocol has some gaps that are important for the authors to address:
- Table 2 is missing a crucial baseline - the depth-warped noise construction strategy used by MultiDiff is essential to compare against.
- Several important metrics are missing. Standard video (FVD/FID) and reconstruction metrics (PSNR/SSIM/LPIPS) are not provided. TSED also only compares consecutive frames and nothing measures whether a revisited area looks the same. VMem [5], WorldMem [6], and Context-as-Memory [7] discuss protocols for evaluating revisiting, and this paper should do something similar.
- Key components of the method aren't ablated, such as the impact of the noise residual and normalization step and the LoRA-finetuning (it's a shame that the method isn't training-free, but it would still be an interesting ablation). As the long-horizon generation is more of a systems-level contribution as opposed to something theoretically novel, it is especially important to ablate its components (such as the carving rules and the memory frames).
The writing is generally ok, although terminology like "world-space stochastic support" could be made simpler. The figure captions should be expanded to describe what the figures are actually conveying. For example, Figure 2 doesn't explicitly say which noise strategy the paper is actually using and why it's better than the other approaches. Figure 3's caption is quite jargon-heavy ("noise projection with covariance completion and normalization") and is a missed opportunity to allow readers to better understand the method.
References:
- [1] Jun-Kun Chen et al. ConsistDreamer: 3D-Consistent 2D Diffusion for High-Fidelity Scene Editing. CVPR 2024.
- [2] Runjie Yan et al. Consistent Flow Distillation for Text-to-3D Generation. ICLR 2025.
- [3] Norman Müller et al. MultiDiff: Consistent Novel View Synthesis from a Single Image. CVPR 2024.
- [4] Pascal Chang et al. How I Warped Your Noise: a Temporally-Correlated Noise Prior for Diffusion Models. ICLR 2024.
- [5] Runjia Li et al. VMem: Consistent Interactive Video Scene Generation with Surfel-Indexed View Memory. ICCV 2025.
- [6] Zeqi Xiao et al. WORLDMEM: Long-term Consistent World Simulation with Memory. NeurIPS 2025.
- [7] Jiwen Yu et al. Context as Memory: Scene-Consistent Interactive Long Video Generation with Memory Retrieval. SIGGRAPH Asia 2025.
Questions for authors
My tentative positive rating reflects that I find the noise construction method to be valuable, but is contingent on the authors addressing my evaluation concerns, especially the depth-warped baseline for Table 2 and revisit experiments that go beyond TSED.
Also:
- It would be helpful for the authors to re-articulate their own claims of novelty in the context of the prior work I mentioned.
- What is the runtime and memory overhead of the 3D noise construction?
- Can you confirm that there is no overlap between the DL3DV-10K train and test scenes?
5 comments
Q4: Compare with MultiDiff.
A4:
We thank the reviewer for this valuable suggestion. We have added the depth-warped noise construction strategy from MultiDiff as an additional baseline. The result are as below. The corresponding quantitative results and qualitative visualizations have also been updated into the revised manuscript.
Comparison of noise construction strategies.
RealEstate10K
| Method | Camera: RotErr ↓ | Camera: TransErr ↓ | Camera: RPE-t ↓ | Camera: RPE-r ↓ | Geometry: TSED ↑ | Geometry: Reg. ↑ | VBench: MS ↑ | VBench: DD ↑ | VBench: AQ ↑ | VBench: IQ ↑ |
|---|---|---|---|---|---|---|---|---|---|---|
| MultiDiff | 0.3736 | 0.418 | 0.9960 | 0.9903 | 0.5199 | 0.7310 | ||||
| CameraNoise | 7.249 | 0.2381 | 0.0247 | 0.9045 | 0.9903 | 0.6800 | 0.5173 | |||
| UniCaMo | 7.316 | 0.4825 | 0.0300 | 0.388 | 0.9286 | 0.9953 | 0.5500 | 0.7330 | ||
| WorldNoise | 1.940 | 0.0216 | 0.128 | 0.9779 | 0.9996 | 0.9907 | 0.6800 | 0.5242 | 0.7448 |
DL3DV
| Method | Camera: RotErr ↓ | Camera: TransErr ↓ | Camera: RPE-t ↓ | Camera: RPE-r ↓ | Geometry: TSED ↑ | Geometry: Reg. ↑ | VBench: MS ↑ | VBench: DD ↑ | VBench: AQ ↑ | VBench: IQ ↑ |
|---|---|---|---|---|---|---|---|---|---|---|
| MultiDiff | 0.2371 | 0.9621 | 1.000 | 0.4793 | 0.6633 | |||||
| CameraNoise | 36.353 | 0.3426 | 0.1465 | 2.706 | 0.1648 | 0.9279 | 0.9510 | 0.4784 | 0.6756 | |
| UniCaMo | 37.701 | 0.3712 | 0.1346 | 2.486 | 0.9192 | 0.9531 | 0.4926 | 0.6962 | ||
| WorldNoise | 7.664 | 0.1806 | 0.0778 | 1.033 | 0.4865 | 0.9666 | 0.9636 | 1.0000 |
The relatively limited performance of this strategy may be attributed to its reference-view-based noise construction. Specifically, MultiDiff warps noise from the first view to subsequent views using depth information and fills uncovered regions with independent Gaussian noise. Under large camera rotations, the overlap with the reference view becomes limited, leaving large regions filled with independent noise and weakening cross-view correlations. In contrast, our method constructs a shared 3D noise field covering the camera trajectory, allowing different views to share consistent noise even in regions not visible from the first view. This provides stronger geometric correlations across views, particularly under challenging camera motions.
--
Q5: Additional Metric.
A5:
We thank the reviewer for this valuable suggestion. We have added the missing evaluation metrics in the revised manuscript, including PSNR, SSIM, and LPIPS for reconstruction quality, as well as FVD for overall video quality. These additional results provide a more comprehensive evaluation of both video generation quality and scene consistency. All these results have also been updated into the revised manuscript.
Additional evaluation results on RealEstate10K and DL3DV using reconstruction metrics and standard video quality.
RealEstate10K
| Method | PSNR ↑ | SSIM ↑ | LPIPS ↓ | FVD ↓ |
|---|---|---|---|---|
| SCoPE | 16.8130 | 0.5454 | 0.3121 | 156.466 |
| Gen3C | 16.8361 | 0.5623 | 0.2931 | 149.856 |
| UCM | 15.7508 | 0.5219 | 0.3382 | 197.988 |
| Wan | 12.7913 | 0.4482 | 0.5283 | 435.162 |
| Wan + Ours | 16.3363 | 0.5417 | 0.3100 | 220.588 |
| LTX | 16.0521 | 0.5222 | 0.4001 | 276.290 |
| LTX + Ours | 17.2733 | 0.5678 | 0.2800 | 141.315 |
DL3DV
| Method | PSNR ↑ | SSIM ↑ | LPIPS ↓ | FVD ↓ |
|---|---|---|---|---|
| SCoPE | 12.6969 | 0.3798 | 0.5232 | 233.823 |
| Gen3C | 13.3487 | 0.3942 | 0.5387 | 279.080 |
| UCM | 12.8588 | 0.3927 | 0.5308 | 435.646 |
| Wan | 10.5532 | 0.3146 | 0.6175 | 574.857 |
| Wan + Ours | 13.7145 | 0.3886 | 0.5272 | 195.275 |
| LTX | 12.3637 | 0.3825 | 0.5781 | 531.800 |
| LTX + Ours | 14.1521 | 0.4349 | 0.5187 | 203.681 |
Q6: Revisitation experiments.
A6:
We thank the reviewer for this valuable suggestion. Following this recommendation, we have added a revisitation evaluation in the revised manuscript (Sec.4.4). Specifically, we follow the evaluation protocol of Context-as-Memory[7] and consider both ground-truth and generated historical observations as memory.
We evaluate the revisit results against the corresponding ground-truth or previously generated observations at identical camera poses using PSNR, SSIM, and LPIPS. We further compare WorldNoise with SANA-WM and Lyra 2.0 under the same evaluation protocol. Since the released SANA-WM inference pipeline does not support ground-truth history conditioning, its ground-truth comparison is marked as N/A. The results are as below:
Quantitative comparison of revisitation consistency on DL3DV.
| Method | GT: PSNR ↑ | GT: SSIM ↑ | GT: LPIPS ↓ | History: PSNR ↑ | History: SSIM ↑ | History: LPIPS ↓ |
|---|---|---|---|---|---|---|
| SANA-WM | N/A | N/A | N/A | 11.6251 | 0.2327 | 0.6326 |
| Lyra 2.0 | 0.5310 | 0.3478 | ||||
| WorldNoise | 13.4526 | 0.3967 | 12.9905 | 0.5803 |
The quantitative results have been added to Figure 8, and more cases are provided in the Appendix E.4 of the revised manuscript. We kindly refer the reviewer to Sec.4.4 and the corresponding results in the revised manuscript for a detailed evaluation of long-term scene consistency.
Q7: More Ablations.
A7:
We thank the reviewer for this helpful suggestion. We add more detailed Ablations studies i
Noise Construction. We first conduct ablation about noise construction as below:
Ablation of the noise residual and normalization on DL3DV.
| Noise Residual | Normalization | RotErr ↓ | TransErr ↓ | TSED ↑ | Reg. ↑ |
|---|---|---|---|---|---|
| -- | ✓ | 7.688 | 0.1813 | 0.4837 | 0.9645 |
| ✓ | -- | 7.701 | 0.1821 | 0.4809 | 0.9587 |
| ✓ | ✓ | 7.664 | 0.1806 | 0.4865 | 0.9666 |
Lora. In our experiments, directly injecting our constructed noise into a pretrained diffusion model fails to produce coherent images, instead yielding drifting, blob-like color patterns. This suggests that the diffusion model needs to be adapted to the noise distribution induced by our construction strategy to generate meaningful visual content. We further investigate the impact of LoRA rank by fine-tuning the baseline model with ranks of 32, 256, and 512, respectively. Increasing the LoRA rank brings modest improvements in generation performance, but also incurs substantially higher computational costs. We therefore adopt rank 32 as a practical trade-off between performance and computational efficiency. Detailed quantitative results are provided in Appendix F.3, and we kindly refer the reviewer to this section for further details.
long-horizon generation. We have added a progressive ablation of the long-horizon generation components, including visual memory, persistent noise identities, and geometry updates and carving. Notably, visual memory provides substantial improvements: compared with the baseline, it reduces RotErr from 19.214 to 15.181 (a 21.0% reduction) and TransErr from 0.2528 to 0.2116 (a 16.3% reduction), while improving PSNR from 12.7331 to 13.5230 and LPIPS from 0.5633 to 0.5521. The additional ablation results and detailed analysis are provided in Appendix F.4. We kindly refer the reviewer to this section for further details.
Q8: Writing.
A8: We thank the reviewer for this helpful suggestion. We have replaced the term "world-space stochastic support" with the simpler term "world-space noise field" throughout the revised manuscript to improve readability. We revised the captions of Figures 2 and 3 to make them more informative and accessible. Specifically, Figure 2 now explicitly identifies (d) as our proposed WorldNoise and explains its advantages over other noise construction strategies. Figure 3 now provides a clearer step-by-step explanation of the noise construction and projection process, including how the residual noise and normalization preserve the Gaussian noise distribution while maintaining cross-view correlations. We have also revised the corresponding terminology in the manuscript to improve clarity.
Q9: Dataset.
A9: Yes, we confirm that there is no overlap between the DL3DV-10K train and test scenes.
Q1: Claims and Novelty.
A1:
We thank the reviewer for highlighting these relevant works. Accordingly, we have toned down the original claim in the revised abstract. The revised statement now reads: "Extensive experiments demonstrate the effectiveness of WorldNoise in improving camera controllability and cross-view consistency, highlighting the potential of shared 3D noise construction for generative world models."
Besides, we would like to emphasize that our key novelty lies in constructing a shared 3D noise space throughout the observable world region, rather than restricting noise to scene surfaces. Specifically, ConsistDreamer [1], CFD [2], and MultiDiff [3] establish noise correspondences based on scene surfaces, where noise is effectively supported on surface geometry rather than distributed throughout the 3D space. In contrast, WorldNoise distributes world-space noise primitives throughout the region jointly covered by the camera frusta along the trajectory. This volumetric construction naturally supports multiple depths without requiring prior knowledge of the geometry of the scene to be generated.
We appreciate the reviewer for bringing these additional references to our attention. We have accordingly refined our novelty claims and expanded the Related Work section to properly acknowledge and discuss these studies.
Q2: Limitations.
A2: We thank the reviewer for the constructive suggestion. We have further clarified the following limitations in the revised Conclusion and Limitations section. (1) Camera Trajectory Requirement. WorldNoise requires a predefined camera trajectory to determine the observable world region for noise construction. (2) Dynamic Scene Modeling. Our current formulation focuses on static scenes, while extending the stochastic world representation to dynamic scenes remains an interesting direction for future work. (3) Limitations of Video Generation Models. Despite recent progress, video generation models still face challenges in maintaining physical consistency and long-term temporal coherence, particularly in complex scenes and extended generation.
Q3: Computational Overhead of Noise Construction.
A3: We measured the runtime and memory overhead of 3D noise construction on a single NVIDIA A100-SXM4-80GB GPU, using a budget of 4M points for 81 RGB frames (21 latent frames) at 480×832 resolution. Across the RealEstate10K and DL3DV datasets, the average noise construction time is 0.533 s, with approximately 0.95 GiB of incremental peak GPU memory allocation. Noise construction accounts for only 1.01% of one training step (forward, backward, and optimizer update) and 0.162% of one inference run (15 denoising steps).