- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper presents UniWorld, a framework for depth-conditioned video generation. It combines existing pretrained models with two inference-time mechanisms: noise-aligned overlapping continuation and rolling depth latent state. The framework supports long-horizon generation from depth trajectories and reference images without additional model training.
Critical strengths
- Using depth as a unified interface for different simulators is a practical idea.
- The framework is evaluated across multiple scene categories and supports continuous video generation.
Critical weaknesses
- Limited technical novelty. The paper mainly integrates existing pretrained models without additional training. Noise-aligned overlapping continuation closely resembles methods in diffusion inpainting such as RePaint, while rolling depth latent state is essentially a sliding-window caching strategy. The technical contributions appear rather incremental.
- Missing comparisons with relevant controllable video generation methods. Since the core task is depth-conditioned video generation, the paper should include comparisons with more relevant methods such as VDOT. The current baselines, primarily H3 and LongVie variants, do not sufficiently establish the advantages of UniWorld over existing controllable video generation approaches.
- Unconvincing experimental results and unclear presentation. The qualitative results in Figure 4 are not particularly impressive. In particular, the simulation examples exhibit noticeable visual drifting, raising concerns about the claimed long-horizon consistency. Moreover, Table 3 is difficult to interpret, as the ablation settings and reported differences are not clearly explained. The results also suggest only marginal improvements from noise-aligned continuation, making its practical effectiveness questionable.
Ref:
- Wang, Yutong, et al. "Vdot: Efficient unified video creation via optimal transport distillation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026.
- Lugmayr, Andreas, et al. "Repaint: Inpainting using denoising diffusion probabilistic models." 2022 IEEE/CVF conference on computer vision and pattern recognition (CVPR). IEEE, 2022.
Questions for authors
- What distinguishes noise-aligned overlapping continuation from existing methods?
- Why are more relevant depth-conditioned video generation methods, such as VDOT, not included in the comparisons?
3 comments
Thanks for the additional experiments and clarifications. I have two remaining questions:
- Although the longer history length can improve on quality, but it also affect the efficiency. What is the trade-off between generation quality and inference efficiency?
- The comparison with VDOT uses different sampling budgets, Could the authors provide a direct comparison of inference latency or throughput?
Thank you for the follow-up questions. We provide further clarification below.
1. Quality–efficiency trade-off.
Longer histories provide additional temporal context, but their computational cost depends on the windowing strategy: enlarging the input window increases computation, whereas keeping the window size fixed reduces the number of newly generated frames. In our four-scene evaluation, increasing the history length from 2 to 12 latents yields only a marginal improvement in common-continuation PSNR, from 16.261 to 16.281 dB, with no consistent improvement across intermediate history lengths. Our default configuration uses a fixed 39-frame overlap within a 345-frame window, allowing 306 new frames to be generated per continuation. At a resolution of 960 × 544 on eight H100 GPUs, the DiT forward pass takes 18.173 s, corresponding to a throughput of 16.838 newly generated frames/s. These results illustrate the trade-off between temporal context and generation efficiency, while showing that increasing history length does not necessarily translate into substantial quality gains.
2. Direct efficiency comparison with VDOT.
We agree that comparing 8 versus 4 sampling steps alone is insufficient to establish actual inference efficiency. For reference, UniWorld takes 84.561 s to deliver a 30-second (720-frame) video at 960 × 544 on eight H100 GPUs, corresponding to 8.515 delivered frames/s. This end-to-end measurement includes sampling, VAE operations, and video export, while excluding model loading and online geometry acquisition. Our existing VDOT measurements were not conducted under a matched evaluation protocol covering resolution, output duration, warmup procedure, and timing scope. Therefore, we refrain from claiming an inference-efficiency advantage over VDOT. The reported quality comparison uses each method's native sampling configuration and should be interpreted as a comparison under those configurations, rather than evidence of superior quality at matched computational cost.
W1. Method and contribution
We do not claim known-region replacement or overlapping windows as new. Our contribution is a synchronized continuation procedure for depth-controlled video: at each of eight model evaluations, the inherited 12-latent prefix follows its flow state,
tH + (1 - t) * epsilon, while overlapping depth remains clean.His generated history andepsilonis the fixed global noise slice at the same temporal positions. Only the new suffix is committed on the global VAE timeline. This couples history, geometry, noise, and output across requests, state transitions unspecified by a finite-window generator.On 24 sources with fixed first window, depth, suffix noise, boundaries, commitment points, and sampling budget, boundary-only and endpoint-only conditioning produce identical final latents. Per-step projection keeps prefix error at zero and changes the suffix. DINO boundary jump is 23.6% lower than boundary-only in the primary run and 35.2% lower in a different-noise rerun (22/24 and 23/24 sources improved). This supports a boundary-continuity benefit. Rolling and direct depth reads also match in all 24 cases; active depth storage is 19.050 MiB at 960 × 544.
We additionally refined the evaluated eight-step DMD student for 250 generator updates with the DMD2 no-GAN objective. On 30 separate 14-second clips, paired PSNR improves by 0.240 dB over the Stage-B teacher (95% CI [0.106, 0.391]), with gains in SSIM and LPIPS. We present this as checkpoint refinement, not a new training algorithm.
W2. Comparison with VDOT
We added the official VDOT-14B model to a matched 24-source comparison using the same reference image, prompt, canonical depth, and 832 × 480 output.
The PSNR interval favors UniWorld; LPIPS includes zero. Native sampling differs, so the comparison does not establish universal perceptual superiority. It shows competitive conditional rendering alongside multi-frame history continuation and aligned depth. In four 30-second VDOT runs, each released-interface request uses one static RGB reference and current depth window. Side-by-side examples and protocol are on the anonymous project page and in the revised supplement.
W3. Long-horizon results and ablations
We added long-video examples, before/after DMD training and VDOT comparisons, to the anonymous project page. On 17 held-out 60-second sources, PSNR falls from 20.911 dB in the first 10-second block to 16.302 dB in the last; fixed-scale depth NMAE rises from 0.0463 to 0.1100. Committed-history preservation does not prevent new-content drift.
Main Table 3 and Appendix C.7 (Table S7) now explain the nine matched ablations and subtraction direction (alternative minus full); continuation PSNR covers frames [345,720), and LPIPS/temporal error use fixed seam neighborhoods. Table S9 gives 22-source sensitivity. The boundary test isolates history at fixed compute; paired reduction intervals are above zero (10,000 category-stratified source-group bootstrap resamples, marginal and unadjusted). Fidelity remains close, so we limit the claim to improved feature continuity at evaluated joins. VDOT details are in Appendix E.3 (Table S26).