- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces Neural Voxel Dynamics, a framework for predicting future scene dynamics in a lifted 3D feature space. Frozen V-JEPA features are unprojected into a voxel grid using depth estimates and augmented with occupancy and unseen-space channels. An action-conditioned flow-matching model predicts future voxel states, which can be projected to image features, decoded to RGB, or converted into trajectories for motion-guided video generation. Experiments report improvements over a matched 2D latent predictor and several video-generation and neural-physics baselines on feature prediction, motion, and occupancy metrics.
Critical strengths
- Learning transitions in an explicit 3D feature grid provides a promising inductive bias for spatially consistent dynamics prediction, without requiring material-specific solvers or simulator-state supervision.
- Experiments across multiple datasets demonstrate improvements over representative 2D and 3D baselines, particularly in latent prediction and geometric consistency.
Critical weaknesses
- Comparison fairness. The proposed method observes four initial RGB frames and incorporates depth-derived geometry, using estimated/ground-truth depth. However, Figure 3 describes CogVideoX as receiving only the first frame. Consequently, the reported improvements may partly reflect differences in conditioning rather than the proposed dynamics model.
- Insufficient specification of baseline configurations. Important evaluation details remain unclear, including the CogVideoX prompts, the physical parameters and scene initialization used by physics-based baselines, and whether/how generated videos are resampled to the evaluation timestamps. Suboptimal prompts or physical parameters could substantially degrade these baselines’ performance, potentially compromising the fairness of the comparison.
Questions for authors
- Could the authors provide complete baseline configurations, including CogVideoX prompts and force-to-text mappings, physical-parameter settings, 3D scene initialization, and output fps?
- Physics-IQ provides useful measurements of agreement with reference videos through motion-overlap and pixel-error metrics, but offers limited direct evidence of physical consistency. Could the authors complement it with metrics such as trajectory error and physical commonsense assessments in PhysCtrl?
- In addition to the Wan-Move without tracks, could the authors include the underlying Wan2.1 I2V model under matched conditioning and generation settings to better isolate the contributions of the proposed method?
- The main evaluation predicts 16 future RGB frames. Could the authors evaluate longer prediction horizons, particularly beyond the training horizon, and report how errors accumulate with rollout duration?
- Section 4 states that all video baselines receive the same initial frames, whereas Figure 3 describes CogVideoX as using only the first frame.
- Minor: Appendix B.4.1 is missing.
Thank you for your response. I would retain my original score.