- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes D3WM for action-conditioned future frame prediction, aiming to disentangle deterministic scene transformations from stochastic generation. A Transformer predicts the future latent mean and spatial uncertainty using Gaussian NLL, while a conditional diffusion model uses the uncertainty map to guide generation. Experiments on robotic manipulation and novel view synthesis demonstrate improvements over several baselines.
Critical strengths
-
The paper introduces a conceptually interesting drift-diffusion framework that separates deterministic prediction from stochastic generation, aiming to balance geometric consistency with generative flexibility through uncertainty-guided diffusion.
-
The experiments cover both NVS and robotic manipulation. In particular, evaluating downstream trajectory prediction and action candidate selection using generated future observations provides meaningful evidence of the model's practical utility beyond image quality. The paper provides relatively comprehensive ablations on variance conditioning, inference-time gating, and alternative blending strategies, helping identify the contribution of different components to the final performance.
Critical weaknesses
-
Insufficient justification of the deterministic-stochastic decomposition. The Drift Network directly regresses future latent states without explicit geometric projection or physical constraints. While its outputs are deterministic, this does not necessarily mean the model captures deterministic physical transformations. Therefore, the proposed architecture does not fully substantiate the central motivation of disentangling deterministic dynamics from stochastic generation.
-
About the deterministic component. The variance map is learned through Gaussian NLL and may reflect prediction errors or limited model capacity rather than inherently stochastic regions. The paper lacks quantitative evidence demonstrating that high-uncertainty regions correspond to disocclusions or other genuinely ambiguous observations. This weakens the main assumption underlying variance-guided generation. Given that the motivation explicitly contrasts geometric projection with generative models, comparisons with geometry-guided diffusion or geometric warping followed by diffusion inpainting would be valuable (e.g. SplatDiff). Although DepthSplat is included, this comparison does not isolate the advantage of learned variance gating over explicit geometry-based masks. Moreover, the evaluation primarily relies on image-quality metrics and indirect policy consistency rather than direct geometric or physical consistency measurements.
Reference work: High-Fidelity Novel View Synthesis via Splatting-Guided Diffusion
Questions for authors
-
How do the authors verify that the learned Drift Network captures deterministic geometric or physical transformations rather than simply predicting the conditional mean of future observations?
-
Can the authors quantitatively compare the learned uncertainty map with geometric disocclusion masks or analyze whether it primarily reflects prediction errors?
-
Would geometric warping combined with diffusion-based inpainting, using the same diffusion backbone, achieve comparable results?