acceptodds
Withdrawn from ICLR 2027

Neural Voxel Dynamics: Learning Latent 3D Physics via Volumetric Feature Advection

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We present , a self-supervised framework for learning 3D latent dynamics from monocular video. While generative video models now produce visually compelling motion, their predominantly 2D representations provide limited geometric structure for modeling and controlling physical interactions. Instead, we learn dynamics in a lifted volumetric latent space. Specifically, we unproject semantic Video Joint-Embedding Predictive Architecture (V-JEPA) features into a voxel grid using monocular depth priors, producing a geometrically grounded representation that retains rich video features. We then introduce , an action-conditioned transition model that predicts the evolution of these latent features directly in 3D. Unlike hybrid approaches that assume access to privileged physics-engine states or explicit simulation during training and/or inference, we learn only using video-derived supervision and actions, allowing a single latent dynamics model to represent heterogeneous material-dependent phenomena, unifying rigid-body and fluid dynamics. We evaluate across multiple datasets (synthetic and real) and unseen scenarios (e.g., unseen boundary conditions, OOD generalization), measuring both predictive fidelity and 3D geometric consistency. We further evaluate physical plausibility with Physics-IQ using videos decoded from predicted latent states and videos generated from predicted optical flow. Our results show that geometrically lifting pretrained video latent representations provides a scalable route toward dynamic world models that learn structured 3D physical dynamics without access to privileged simulator state.

Primary area:
generative models
Keywords:
3D, latent space, world models, JEPA, self-supervised, simulation, video generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_XDSCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Neural Voxel Dynamics, a framework for predicting future scene dynamics in a lifted 3D feature space. Frozen V-JEPA features are unprojected into a voxel grid using depth estimates and augmented with occupancy and unseen-space channels. An action-conditioned flow-matching model predicts future voxel states, which can be projected to image features, decoded to RGB, or converted into trajectories for motion-guided video generation. Experiments report improvements over a matched 2D latent predictor and several video-generation and neural-physics baselines on feature prediction, motion, and occupancy metrics.

Critical strengths
  1. Learning transitions in an explicit 3D feature grid provides a promising inductive bias for spatially consistent dynamics prediction, without requiring material-specific solvers or simulator-state supervision.
  2. Experiments across multiple datasets demonstrate improvements over representative 2D and 3D baselines, particularly in latent prediction and geometric consistency.
Critical weaknesses
  1. Comparison fairness. The proposed method observes four initial RGB frames and incorporates depth-derived geometry, using estimated/ground-truth depth. However, Figure 3 describes CogVideoX as receiving only the first frame. Consequently, the reported improvements may partly reflect differences in conditioning rather than the proposed dynamics model.
  2. Insufficient specification of baseline configurations. Important evaluation details remain unclear, including the CogVideoX prompts, the physical parameters and scene initialization used by physics-based baselines, and whether/how generated videos are resampled to the evaluation timestamps. Suboptimal prompts or physical parameters could substantially degrade these baselines’ performance, potentially compromising the fairness of the comparison.
Questions for authors
  1. Could the authors provide complete baseline configurations, including CogVideoX prompts and force-to-text mappings, physical-parameter settings, 3D scene initialization, and output fps?
  2. Physics-IQ provides useful measurements of agreement with reference videos through motion-overlap and pixel-error metrics, but offers limited direct evidence of physical consistency. Could the authors complement it with metrics such as trajectory error and physical commonsense assessments in PhysCtrl?
  3. In addition to the Wan-Move without tracks, could the authors include the underlying Wan2.1 I2V model under matched conditioning and generation settings to better isolate the contributions of the proposed method?
  4. The main evaluation predicts 16 future RGB frames. Could the authors evaluate longer prediction horizons, particularly beyond the training horizon, and report how errors accumulate with rollout duration?
  5. Section 4 states that all video baselines receive the same initial frames, whereas Figure 3 describes CogVideoX as using only the first frame.
  6. Minor: Appendix B.4.1 is missing.
Reviewer_oDGkOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Neural Voxel Dynamics (NVD), a framework for learning geometrically structured 3D latent dynamics from monocular video. The method lifts frozen V-JEPA features into a sparse voxel representation with feature, occupancy, and occlusion channels using monocular depth and camera estimates. A flow-matching spatiotemporal Transformer then predicts future voxel states and can optionally take an explicit force input. The paper evaluates latent prediction, optical flow, occupancy consistency, and Physics-IQ across CLEVRER-style data, PhysInOne, PhysGaia, and Kubric MOVI-C, against a 2D latent baseline, CogVideoX, PhysGen, PhysGaussian, and PhysCtrl.

Critical strengths
  1. Lifting pretrained video representations into an explicit 3D voxel space and learning latent feature transport is a clear and meaningful design. The occupancy and occlusion channels are also well motivated for incomplete monocular observations.

  2. The attempt to model rigid bodies, fluids, and smoke with one model, without physics-engine states or material parameters, is an appealing problem formulation.

Critical weaknesses
  1. The data splits and sample-selection procedure require clarification.

Appendix A.5 states that 5% of each dataset is used for testing and that samples involving fluid and smoke interactions are hand-picked. However, it remains unclear when this selection is performed, how many samples belong to each split, and whether scenes and trajectories are strictly separated across splits. These details are necessary to assess the representativeness of the evaluation and the generalization claims.

  1. Baseline fairness and reproducibility are unclear.

PhysGaussian receives all camera views, while most other methods receive monocular input. (The original PhysGaussian takes a pre-optimized 3D Gaussian Splatting representation as input; PhysGM, a related work in the same line of research, accepts monocular input only and could be considered for adaptation.) CogVideoX receives forces through text, while PhysGen receives projected 2D forces. The conditioning information is therefore not equivalent. The paper should state baseline tuning budgets, training data, model versions, retraining details, and the precise configurations used for fluids, smoke, and multi-object scenes.

  1. Evidence for physical and controllable dynamics is weaker than the paper’s framing.

Latent errors and optical-flow consistency measure prediction quality but do not directly verify momentum conservation, collision responses, or material-dependent physical laws. Occupied volume is also not equivalent to physical mass. Action annotations are available only for the custom rigid-body data, and controllable generalization is mainly demonstrated through qualitative ball-force examples. Evidence for force-based control across materials is therefore limited.

Questions for authors
  1. How many trajectories, scenes, and camera views are used for training, validation, and testing on each dataset? What exactly are the “hand-picked” criteria, and are scenes, object configurations, and views strictly separated across splits?

  2. Could the authors report 3D motion or trajectory errors independent of V-JEPA feature space on synthetic data with ground-truth geometry, to verify the benefits of 3D lifting for dynamics modeling?

2 comments

Thank you for your response. I would retain my original score.

Thanks again for your comments and suggestions.

Reviewer_n38GOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies augmenting V-JEPA features with 3D information from 3D foundation models. It modifies the V-JEPA process by adding an intermediate 3D representation - a sparse feature volume that is obtained by lifting V-JEPA features to the 3D with depth predicted from an off-the-shelf model. The dynamic model is an autoregressive model in the 3D feature space. The authors argue that this brings better awareness with the geometric information for the dynamic model. In evaluation, the paper compares mainly the latent prediction with a family of baselines that use physical simulation to predict dynamics, and show some advantages.

Critical strengths
  • The idea of maintaining a 3D intermediate representation for dynamics prediction is interesting.
  • The paper is well-written and easy to follow.
Critical weaknesses
  • The evaluation is weak and cannot really defend the claims made in this paper. The comparison is not apple-to-apple: other methods are (1) training-free, and (2) designed for predicting pixels with an explicit 3D simulator, while this method is predicting latent dynamics with finetuning. Baselines like CogVideoX is not finetuned on the tested dataset. It would make more sense if the approach is compared to other 2D dynamics models, including SotA video prediction models (Wan, Minimax H3) and a finetuned V-JEPA. The metrics are also designed by this paper itself, which is the same as the training target, while other models are not trained for this target.
  • The advantage is mostly shown on a very limited setting - the camera is not moved, which, is not realistic for in-the-wild videos. The compared benchmarks are also relatively toy-ish. How would the model perform on more complicated datasets like DROID? The action conditioning is also very limited, which is only shown on one of the datasets.
Questions for authors
  • How would the method estimate depth for smoke and fluid, which doesn't have a definition of "depth" because of transparency?
  • How would the method perform on more complicated datasets?
  • How does the method compare against other baselines including a well-trained video prediction model?
1 comment
AuthorsOpenReview ↗

Any comments/feedback?

Dear n38G,

Sorry for another message. As the system closes to the authors soon, we would like to know if we can provide further clarifications and/or if your concerns have been addressed.

We understand that it is a very busy time of the year. Thanks for your attention.

Authors.

Comments on the paper

1 comment

Dear reviewers,

With only a couple more days to go, we would like to know your thoughts on our response. If needed, we can clarify or add to it during the remaining days of the discussion period. We would appreciate if you can let us know your comments now so we have time to prepare our follow-ups for discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.