acceptodds
Desk rejected at ICLR 2027

UniWorld: A Unified World Model for Real-Time Visual World Simulation

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Continuous visual world simulation requires coupling evolving scene geometry with temporally persistent appearance. We present UniWorld, a unified framework that integrates depth trajectories and reference appearance within a shared video generation model. The central design maintains visual and geometric histories on a common timeline. Noise-aligned overlapping continuation places generated history at the prescribed sampling noise level and commits only newly generated latents. Rolling depth latent state preserves the corresponding geometric controls, incrementally reads newly available geometry, and keeps the conditioning fixed within each sampling window. Together, these mechanisms ensure exact preservation of latent history while bounding the active depth state by the window size. We evaluate continuous generation, paired history interventions, and long-horizon behavior across diverse scene categories. The same generator supports estimated, native, and on-demand simulator depth, while cached and incremental execution paths produce bitwise-identical outputs. Appearance changes and geometric control branches operate through the same interface without modifying committed history. The measured steady-state DiT execution path achieves real-time generation throughput, demonstrating the efficiency of UniWorld for continuous visual world simulation.

Primary area:
generative models
Keywords:
Visual World Models, Depth-Guided Video Generation, Few-Step Distillation, Long-Horizon Generation, Controllable Video Generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hQQaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents UniWorld, a framework for depth-conditioned video generation. It combines existing pretrained models with two inference-time mechanisms: noise-aligned overlapping continuation and rolling depth latent state. The framework supports long-horizon generation from depth trajectories and reference images without additional model training.

Critical strengths
  1. Using depth as a unified interface for different simulators is a practical idea.
  2. The framework is evaluated across multiple scene categories and supports continuous video generation.
Critical weaknesses
  1. Limited technical novelty. The paper mainly integrates existing pretrained models without additional training. Noise-aligned overlapping continuation closely resembles methods in diffusion inpainting such as RePaint, while rolling depth latent state is essentially a sliding-window caching strategy. The technical contributions appear rather incremental.
  2. Missing comparisons with relevant controllable video generation methods. Since the core task is depth-conditioned video generation, the paper should include comparisons with more relevant methods such as VDOT. The current baselines, primarily H3 and LongVie variants, do not sufficiently establish the advantages of UniWorld over existing controllable video generation approaches.
  3. Unconvincing experimental results and unclear presentation. The qualitative results in Figure 4 are not particularly impressive. In particular, the simulation examples exhibit noticeable visual drifting, raising concerns about the claimed long-horizon consistency. Moreover, Table 3 is difficult to interpret, as the ablation settings and reported differences are not clearly explained. The results also suggest only marginal improvements from noise-aligned continuation, making its practical effectiveness questionable.

Ref:

  1. Wang, Yutong, et al. "Vdot: Efficient unified video creation via optimal transport distillation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026.
  2. Lugmayr, Andreas, et al. "Repaint: Inpainting using denoising diffusion probabilistic models." 2022 IEEE/CVF conference on computer vision and pattern recognition (CVPR). IEEE, 2022.
Questions for authors
  1. What distinguishes noise-aligned overlapping continuation from existing methods?
  2. Why are more relevant depth-conditioned video generation methods, such as VDOT, not included in the comparisons?
3 comments

W1. Method and contribution

We do not claim known-region replacement or overlapping windows as new. Our contribution is a synchronized continuation procedure for depth-controlled video: at each of eight model evaluations, the inherited 12-latent prefix follows its flow state, tH + (1 - t) * epsilon, while overlapping depth remains clean. H is generated history and epsilon is the fixed global noise slice at the same temporal positions. Only the new suffix is committed on the global VAE timeline. This couples history, geometry, noise, and output across requests, state transitions unspecified by a finite-window generator.

On 24 sources with fixed first window, depth, suffix noise, boundaries, commitment points, and sampling budget, boundary-only and endpoint-only conditioning produce identical final latents. Per-step projection keeps prefix error at zero and changes the suffix. DINO boundary jump is 23.6% lower than boundary-only in the primary run and 35.2% lower in a different-noise rerun (22/24 and 23/24 sources improved). This supports a boundary-continuity benefit. Rolling and direct depth reads also match in all 24 cases; active depth storage is 19.050 MiB at 960 × 544.

We additionally refined the evaluated eight-step DMD student for 250 generator updates with the DMD2 no-GAN objective. On 30 separate 14-second clips, paired PSNR improves by 0.240 dB over the Stage-B teacher (95% CI [0.106, 0.391]), with gains in SSIM and LPIPS. We present this as checkpoint refinement, not a new training algorithm.

W2. Comparison with VDOT

We added the official VDOT-14B model to a matched 24-source comparison using the same reference image, prompt, canonical depth, and 832 × 480 output.

MethodNative samplingPSNR ↑LPIPS ↓
VDOT-14B4 NFE, 16 fps15.5510.3566
UniWorld8 NFE, 24 fps16.9030.3538
UniWorld − VDOTNative settings+1.353 [0.303, 2.509] dB−0.0028 [−0.0417, 0.0335]

The PSNR interval favors UniWorld; LPIPS includes zero. Native sampling differs, so the comparison does not establish universal perceptual superiority. It shows competitive conditional rendering alongside multi-frame history continuation and aligned depth. In four 30-second VDOT runs, each released-interface request uses one static RGB reference and current depth window. Side-by-side examples and protocol are on the anonymous project page and in the revised supplement.

W3. Long-horizon results and ablations

We added long-video examples, before/after DMD training and VDOT comparisons, to the anonymous project page. On 17 held-out 60-second sources, PSNR falls from 20.911 dB in the first 10-second block to 16.302 dB in the last; fixed-scale depth NMAE rises from 0.0463 to 0.1100. Committed-history preservation does not prevent new-content drift.

Main Table 3 and Appendix C.7 (Table S7) now explain the nine matched ablations and subtraction direction (alternative minus full); continuation PSNR covers frames [345,720), and LPIPS/temporal error use fixed seam neighborhoods. Table S9 gives 22-source sensitivity. The boundary test isolates history at fixed compute; paired reduction intervals are above zero (10,000 category-stratified source-group bootstrap resamples, marginal and unadjusted). Fidelity remains close, so we limit the claim to improved feature continuity at evaluated joins. VDOT details are in Appendix E.3 (Table S26).

Thanks for the additional experiments and clarifications. I have two remaining questions:

  1. Although the longer history length can improve on quality, but it also affect the efficiency. What is the trade-off between generation quality and inference efficiency?
  2. The comparison with VDOT uses different sampling budgets, Could the authors provide a direct comparison of inference latency or throughput?

Thank you for the follow-up questions. We provide further clarification below.

1. Quality–efficiency trade-off.

Longer histories provide additional temporal context, but their computational cost depends on the windowing strategy: enlarging the input window increases computation, whereas keeping the window size fixed reduces the number of newly generated frames. In our four-scene evaluation, increasing the history length from 2 to 12 latents yields only a marginal improvement in common-continuation PSNR, from 16.261 to 16.281 dB, with no consistent improvement across intermediate history lengths. Our default configuration uses a fixed 39-frame overlap within a 345-frame window, allowing 306 new frames to be generated per continuation. At a resolution of 960 × 544 on eight H100 GPUs, the DiT forward pass takes 18.173 s, corresponding to a throughput of 16.838 newly generated frames/s. These results illustrate the trade-off between temporal context and generation efficiency, while showing that increasing history length does not necessarily translate into substantial quality gains.

2. Direct efficiency comparison with VDOT.

We agree that comparing 8 versus 4 sampling steps alone is insufficient to establish actual inference efficiency. For reference, UniWorld takes 84.561 s to deliver a 30-second (720-frame) video at 960 × 544 on eight H100 GPUs, corresponding to 8.515 delivered frames/s. This end-to-end measurement includes sampling, VAE operations, and video export, while excluding model loading and online geometry acquisition. Our existing VDOT measurements were not conducted under a matched evaluation protocol covering resolution, output duration, warmup procedure, and timing scope. Therefore, we refrain from claiming an inference-efficiency advantage over VDOT. The reported quality comparison uses each method's native sampling configuration and should be interpreted as a comparison under those configurations, rather than evidence of superior quality at matched computational cost.

Reviewer_FKbqOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents UniWorld, a video renderer that takes depth sequences and a reference image as input. It uses a bidirectional video generator within each window and continues across windows using generated history. The main contributions are noise-aligned history sampling and rolling depth state management. These keep past video latents and their depth conditions unchanged while generating new frames. The paper tests video quality, history consistency, long-video generation, and runtime across several scene types.

Critical strengths

The design choices are reasonable and useful in practice.

The evaluation is broad. The paper tests several scene types, depth sources, history settings, and video lengths.

Critical weaknesses

My main concern is limited novelty. The framework largely builds on depth-conditioned image-to-video generation. Its main changes are cross-window sampling and state management, using existing models and familiar sampling and caching ideas. These changes are useful, but the paper does not clearly show a large enough advance over existing works.

The experiments are broad, but the gains are relatively limited.

Questions for authors

This work adopte a bidirectional generator with autoregressive window continuation. Have the authors considered using a causal autoregressive model, like the approaches discussed in the related work?

1 comment

W1. Motivation and novelty

We connect controllable 3D world evolution to continuous visual observations. The renderer accepts different geometry sources, carries visual/geometric state across requests, and applies new controls only to future output.

UniWorld addresses three challenges: (i) a shared image-aligned depth interface; (ii) per-step use of multi-frame history while preserving committed output; and (iii) incremental geometry input and frame commitment with bounded state. Appendix E (Tables S24–S26) reports:

ChallengeEvidence
Shared interfaceOne checkpoint renders 100 thirty-second clips across four categories (300 windows) and 25 native-depth simulation inputs.
Updated controlA second-window geometry branch preserves initial latents bitwise and changes only the suffix (latent MAE 0.036161); full history projection has zero prefix error.
Long horizonRolling/direct: 24/24 bitwise matches; online/cached: 3/3. Active depth rows are bounded at 19.050 MiB (960 × 544); tests include 60-second trajectories and 120-second stress tests.

Depth-conditioned image-to-video generation produces a clip but does not define how generated multi-frame history constrains the next request. A boundary image provides single-frame context; stitching preserves exported frames but does not constrain subsequent sampling. In our VDOT-14B rolling-reference test, each request uses one static RGB reference and its depth window; all 28 regenerated overlaps differ from the preceding request at matching timestamps (Table S26). This is specific to the tested interface. Our contribution is synchronized continuation, not depth conditioning or known-region projection alone.

We compare VDOT-14B on 24 matched sources, using the same reference, prompt, canonical depth frames, and 832 × 480 output:

MethodNative samplingPSNR ↑LPIPS ↓
VDOT-14B4 NFE, 16 fps15.5510.3566
UniWorld8 NFE, 24 fps16.9030.3538
DifferenceNative configurations+1.353 [0.303, 2.509] dB−0.0028 [−0.0417, 0.0335]

PSNR favors UniWorld; the LPIPS interval includes zero. Native configurations are not equal-compute and do not establish universal superiority, but show competitive fidelity with depth-aligned multi-frame continuation.

W2. Limited gains

We added a controlled boundary-continuity experiment on 24 sources (six per category), fixing the checkpoint, first-window output, conditions, suffix noise, boundaries, commitment points, and eight evaluations per continuation window. DINO boundary jump is mean cosine distance between L2-normalized DINO ViT-B/16 features at frame pairs (344, 345) and (650, 651); lower is better. Videos: anonymous project page.

Table 1. DINO boundary jump (24 sources per arm).

History treatmentNFEPrimary ↓Rerun ↓
Boundary image only80.0502560.046655
Endpoint restoration only80.0502560.046655
Per-step projection, independent history noise80.0473720.039240
UniWorld: per-step projection, shared global noise80.0384010.030235

Table 2. Paired reductions in boundary jump (positive favors UniWorld).

ComparatorRunAbsolute reduction [95% interval]RelativeImproved
Boundary image onlyPrimary0.011855 [0.007040, 0.016883]23.59%22/24
Boundary image onlyRerun0.016420 [0.010192, 0.023775]35.19%23/24
Independent history noisePrimary0.008971 [0.005946, 0.011997]18.94%22/24
Independent history noiseRerun0.009005 [0.004657, 0.014537]22.95%18/24

Intervals are marginal, unadjusted 95% category-stratified source-group bootstrap intervals (10,000 resamples), computed separately per run. Relative reduction is the mean difference divided by the comparator mean.

At the same budget, shared-global-noise projection reduces boundary jump by 23.6%/35.2% versus boundary-image-only continuation (22/24 and 23/24 sources), and 18.9%/22.9% versus independent history noise. Continuation PSNR and LPIPS remain numerically close; boundary jump measures join continuity. Quality ablations remain in Table 3 and Appendix C.7 (Table S7), sensitivity in Table S9, and state/output checks in Appendix E.2 (Table S25).

Q1. Why not use a causal autoregressive generator?

  • UniWorld is autoregressive across windows but bidirectional within each, allowing H3 to use the full depth interval.
  • H3 was not pretrained causally. An inference-time causal mask changes its training attention context, so it is not a like-for-like baseline; we have not tested it.
  • A causal counterpart requires a causal depth-conditioned backbone trained with generated-history inputs, plus distillation/acceleration evaluation. We use the released bidirectional backbone and leave a causal backend to future work.
Reviewer_WTr7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

UniWorld is a continuous generative rendering framework conditioned on depth trajectories and reference appearance. It combines released H3 ControlNet-Union and VDN-H3 eight-step components without additional joint training. Its main contributions are noise-aligned projection of overlapping video history during sampling and synchronized rolling depth conditioning. Evaluation covers multiple scene categories, paired ablations, long-horizon generation, simulator demonstrations, and stage-specific throughput measurements.

Critical strengths

Practical integration across geometry sources. The framework provides a common rendering interface for estimated, native, and on-demand depth. Its explicit treatment of temporal alignment and output commitment is useful for connecting existing video generators to simulation systems.

Careful implementation validation. Paired interventions, bootstrap intervals, and exact-equivalence checks provide useful evidence that the implementation preserves committed history and maintains consistent conditioning across execution paths.

Critical weaknesses

The methodological advance is insufficiently established. The history projection closely follows known-region conditioning during diffusion sampling, while rolling depth mainly retains overlapping conditions and reads new positions. Its output equivalence to direct slicing confirms correct state management. However, the paper does not sufficiently establish the substantive advance over existing inpainting and overlapping-window continuation approaches.

The experiments provide limited direct support for the proposed continuation mechanisms. Table 1 evaluates 5- and 14-second outputs, primarily assessing the underlying generator rather than cross-window continuation. In Table 3, full history projection improves seam LPIPS by only 0.00195 over boundary-image conditioning, while continuation PSNR and temporal-error intervals include zero.

Questions for authors

What specific methodological capability distinguishes the proposed history projection from existing noise-aligned inpainting and overlapping-window continuation? A direct comparison would materially clarify the contribution.

Beyond the reported hot DiT/Union forward throughput on eight H100 GPUs, what are the end-to-end generation rate, time to first delivered frame, and control-response latency? These measurements are important for assessing the practical real-time claim.

1 comment

W1 / Q1. Methodological distinction

Finite-window rendering must carry generated observations and geometry into each new window while accepting new geometry. UniWorld couples: (i) history projection before every model evaluation, replacing the inherited multi-frame prefix with its corresponding flow-noise state using the current noise level and a fixed global-noise slice aligned to the same latent positions; (ii) rolling depth, retaining clean overlaps, fetching only new geometry, and aligning depth, history, and noise on one timeline with window-bounded storage; and (iii) output commitment, appending only new latents and maintaining global VAE phase. Thus, history guides every sampling step, while new controls affect only uncommitted output. The contribution is synchronized continuation, not inpainting itself.

Controlled history test. On 24 sources (six/category), we fix checkpoint, first-window output, depth, conditions, suffix noise, boundaries, commitment points, and eight evaluations/window; each arm commits only its suffix. DINO boundary jump is mean cosine distance between L2-normalized DINO ViT-B/16 features at the 345-frame joins (344,345) and (650,651), lower being better. It is an exploratory diagnostic (not an official VBench metric) using VBench's feature pipeline; source scores are equally averaged. Rerun: different noise, same cohort.

Table 1. DINO boundary jump (↓).

History treatmentNFEPrimaryRerun
Boundary image80.0502560.046655
Endpoint restoration80.0502560.046655
Per-step, independent history noise80.0473720.039240
UniWorld: per-step, shared global noise80.0384010.030235

Table 2. Paired reductions vs. comparators (positive favors UniWorld).

ComparatorRunAbsolute reduction [95% CI]Relative
Boundary imagePrimary0.011855 [0.007040, 0.016883]23.59%
Boundary imageRerun0.016420 [0.010192, 0.023775]35.19%
Independent noisePrimary0.008971 [0.005946, 0.011997]18.94%
Independent noiseRerun0.009005 [0.004657, 0.014537]22.95%

Intervals are marginal, unadjusted 95% category-stratified source-group bootstrap CIs (10,000 resamples), per run. Shared-noise projection reduces boundary jump by 23.6%/35.2% vs. boundary-image conditioning and 18.9%/22.9% vs. independent noise. Boundary-only and endpoint-only outputs coincide in all 24 cases; per-step projection has zero prefix error. All three online/cached simulator pairs have identical final latents; control changes only the suffix.

W2. Long-horizon evaluation

To further evaluate the continuation mechanism, we added matched 5-, 14-, and 30-second trajectory evaluations using the same official seven-dimension VBench Quality Score, 24 sources, DMD250, and four controlled history interventions.

Table 3. VBench Quality Score (×100, ↑).

History treatmentNoise run5 s14 s30 s
Boundary image onlyPrimary80.61179.73879.414
Endpoint restoration onlyPrimary80.61179.73879.414
Independent history noisePrimary80.61179.73879.450
UniWorldPrimary80.61179.73879.430
Boundary image onlyRerun80.24779.99579.686
Endpoint restoration onlyRerun80.24779.99579.686
Independent history noiseRerun80.24779.99579.675
UniWorldRerun80.24779.99579.683

All durations use identical VBench settings and lossless 30-second prefixes (120/336/720 frames at 24 fps, 960 × 544); both runs share 24 sources. The 30-second evaluation spans three windows; 5- and 14-second prefixes remain in the first. VBench scores align well across durations.

In the primary run, UniWorld scores 79.430 at 30 seconds, improving by 0.017 points over boundary-image-only continuation (95% CI: [0.003, 0.030]); the 14-to-30-second change is also 0.017 points. Shared-noise projection reduces DINO boundary jump by 23.6%/35.2% versus boundary-image-only continuation and 18.9%/22.9% versus independent history noise, supporting improved continuity at window transitions.

Q2. End-to-end delivery and control latency

For 30-s/720-frame requests on eight H100 GPUs:

ResolutionRequest (s)Delivered FPSFirst RGB (s)
960×54484.5618.51550.390
672×38454.53813.20236.436

Each row averages 12 requests (four sources × three repeats) after warmup. Timing starts after condition loading and includes sampling, state handoff, boundary VAE, RGB decoding, and H.264 output; model loading and online geometry acquisition are excluded. Live PyBullet control-to-RGB latency at 672×384 is 64.823 s (including geometry acquisition/delivery), versus 14.859 s with cached geometry. See revised Supplementary Material §§D.1/D.4 for timing protocol, incremental delivery, and control-latency measurement.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.