acceptodds
Withdrawn from ICLR 2027

Decoupled Drift-Diffusion World Models for Consistent Future Frame Prediction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

World models of visual information predict future image frames conditioned on given action sequences. This predicted scene evolution is driven by deterministic changes such as controlled motion on a robot platform, as well as stochastic changes such as rendering of previously occluded regions. Our key insight is that the spatial location of these stochastic changes is itself learnable. We leverage this insight by jointly regressing both the deterministic next-state prediction and the spatial uncertainty map, then using the uncertainty map to guide a conditional diffusion process. We propose Decoupled Drift-Diffusion World Models (D3WM), which uses a transformer trained with a Gaussian negative-log-likelihood objective to produce both the deterministic next-state prediction and a spatial map of heteroscedastic aleatoric uncertainty. The learned uncertainty map then provides a structured spatial gate for a conditional diffusion model: deterministic regions are anchored to the predicted state, while uncertain regions, such as disocclusions or dynamic textures, are sampled by the diffusion process. We frame this decomposition under a variance-gated stochastic differential equation that separates deterministic drift from stochastic diffusion. D3WM shows improved results over competitive baselines in two distinct types of action spaces: 1) camera movement in long-horizon novel view synthesis and 2) trajectory planning for robotic manipulation from generated images.

Primary area:
generative models
Keywords:
action-conditioned world models, generative models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_JZzqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes D3WM for action-conditioned future frame prediction, aiming to disentangle deterministic scene transformations from stochastic generation. A Transformer predicts the future latent mean and spatial uncertainty using Gaussian NLL, while a conditional diffusion model uses the uncertainty map to guide generation. Experiments on robotic manipulation and novel view synthesis demonstrate improvements over several baselines.

Critical strengths
  1. The paper introduces a conceptually interesting drift-diffusion framework that separates deterministic prediction from stochastic generation, aiming to balance geometric consistency with generative flexibility through uncertainty-guided diffusion.

  2. The experiments cover both NVS and robotic manipulation. In particular, evaluating downstream trajectory prediction and action candidate selection using generated future observations provides meaningful evidence of the model's practical utility beyond image quality. The paper provides relatively comprehensive ablations on variance conditioning, inference-time gating, and alternative blending strategies, helping identify the contribution of different components to the final performance.

Critical weaknesses
  1. Insufficient justification of the deterministic-stochastic decomposition. The Drift Network directly regresses future latent states without explicit geometric projection or physical constraints. While its outputs are deterministic, this does not necessarily mean the model captures deterministic physical transformations. Therefore, the proposed architecture does not fully substantiate the central motivation of disentangling deterministic dynamics from stochastic generation.

  2. About the deterministic component. The variance map is learned through Gaussian NLL and may reflect prediction errors or limited model capacity rather than inherently stochastic regions. The paper lacks quantitative evidence demonstrating that high-uncertainty regions correspond to disocclusions or other genuinely ambiguous observations. This weakens the main assumption underlying variance-guided generation. Given that the motivation explicitly contrasts geometric projection with generative models, comparisons with geometry-guided diffusion or geometric warping followed by diffusion inpainting would be valuable (e.g. SplatDiff). Although DepthSplat is included, this comparison does not isolate the advantage of learned variance gating over explicit geometry-based masks. Moreover, the evaluation primarily relies on image-quality metrics and indirect policy consistency rather than direct geometric or physical consistency measurements.

Reference work: High-Fidelity Novel View Synthesis via Splatting-Guided Diffusion

Questions for authors
  1. How do the authors verify that the learned Drift Network captures deterministic geometric or physical transformations rather than simply predicting the conditional mean of future observations?

  2. Can the authors quantitatively compare the learned uncertainty map with geometric disocclusion masks or analyze whether it primarily reflects prediction errors?

  3. Would geometric warping combined with diffusion-based inpainting, using the same diffusion backbone, achieve comparable results?

Reviewer_kBHSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Decoupled Drift-Diffusion World Model (D3WM), an action-conditioned visual world model for future frame prediction. The key idea is to separate deterministic scene evolution from stochastic generation: a transformer-based drift network predicts the deterministic next state together with a spatial map of heteroscedastic aleatoric uncertainty, and this uncertainty map is then used to guide a variance-gated diffusion process. The paper argues that the spatial locations of stochastic changes are learnable, rather than needing to be handled by unconstrained free-form diffusion.

Critical strengths
  • The decomposition between deterministic dynamics and stochastic rendering is well motivated for action-conditioned future frame prediction. In many robotic or embodied settings, some changes are predictable from actions, while others arise from occlusions, newly revealed regions, or inherently uncertain visual details. Explicitly modeling this distinction is a compelling design choice.
  • The variance-gated diffusion mechanism provides a principled way to anchor generation to deterministic predictions while allowing diffusion to focus on uncertain regions. This is preferable to applying diffusion uniformly over the whole image, which can introduce unnecessary structural distortions.
Critical weaknesses
  • The idea of combining deterministic prediction with generative refinement is natural and has appeared in related forms. The paper would benefit from a sharper comparison to prior methods that use residual diffusion, uncertainty-aware generation, or deterministic backbones followed by stochastic refinement. The authors should make clearer whether the main novelty lies in the learned spatial variance map, the specific gating formulation, the SDE interpretation, or the empirical integration of these components.
  • The drift network is described as a vision transformer with approximately 450M parameters, in addition to relying on a diffusion backbone. This raises questions about computational cost, training efficiency, inference latency, and accessibility compared with baselines. The paper would be stronger with a clearer cost-performance analysis.
  • The proposed method targets consistent future frame prediction, but it is not fully clear how performance evolves under repeated rollout. A method may perform well at fixed prediction horizons while still accumulating errors during autoregressive multi-step prediction. More long-horizon evaluation would strengthen the paper.
Questions for authors
  • How well calibrated is the predicted spatial aleatoric uncertainty map? Can the authors report quantitative uncertainty calibration results or uncertainty-error correlation?
  • What is the computational overhead of D3WM compared with NWM and other baselines, both during training and inference?
  • How does the method perform under long autoregressive rollouts, beyond the evaluated fixed time shifts?
  • Which component contributes most to the final improvement: the deterministic drift predictor, the variance map, the gating mechanism, or the diffusion backbone?
  • How sensitive is the method to the choice of diffusion backbone, given that the implementation uses the VP regime aligned with Stable Diffusion 1.5?
  • Does the learned variance map correspond to intuitive uncertain regions such as disocclusions, object boundaries, or newly visible areas? More visualizations would help.
Reviewer_pdaROpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes D3WM for predicting future images given reference images and an action sequence. A deterministic Transformer first learns the mean and spatially heteroscedastic variance of future latent states using a Gaussian negative log-likelihood objective. The predicted mean, variance map, and reference images then condition a diffusion model. During reverse denoising, the variance controls the extent to which each spatial location is anchored to the noised mean prediction. The goal is to preserve structure in predictable regions while generating plausible content in uncertain regions, such as disocclusions. The authors evaluate image prediction, the consistency of a frozen policy's outputs, and candidate-action selection on Real PushT and BridgeData V2, as well as camera-action-conditioned novel-view extrapolation on DL3DV and RealEstate10K. The results show advantages on some image metrics at longer horizons and on downstream proxy metrics, alongside a trade-off between pixel accuracy and perceptual quality.

Critical strengths
  1. The design directly addresses the tension between generating detail and preserving action-dependent structure.
  2. The experiments cover different action spaces and include useful downstream proxy evaluations, which proves author's ideas.
Critical weaknesses

1. The theoretical justification for uncertainty-based gating is incomplete. Page 6 and Appendix E.3 argue that blending two latents at the same noise level preserves that noise level. This is not generally true: averaging two independent noise terms with equal variance reduces the variance by half. The authors should clarify how the noise terms are coupled and when their argument holds. This does not mean the sampler is ineffective, but its probabilistic justification needs improvement.

  1. The ablations support the usefulness of variance-guided refinement, but do not clearly establish whether the predicted variance captures ambiguity in future outcomes or errors in the deterministic predictor. More direct validation would strengthen the interpretation of the variance map as aleatoric uncertainty.

  2. It remains unclear whether better visual predictions lead to better decisions. The control evaluations rely on offline measures of policy consistency and agreement with demonstrated actions. These provide useful evidence, but do not directly show whether planning with D3WM leads to more successful task execution. Visually accurate predictions may still miss details that matter for choosing actions. Closed-loop evaluations measuring task success would provide stronger support for the claimed planning and control benefits.

Questions for authors

See Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.