acceptodds
Under review as a conference paper at ICLR 2027

PhysLDM: Latent Diffusion for High-Fidelity Deformable Simulation

13 views

Abstract

Neural simulation of high-fidelity deformable bodies is a foundational challenge in computer graphics and physical AI. However, predicting long-horizon dynamics for 3D volumetric meshes presents multifaceted and tightly coupled challenges. Step-by-step autoregressive methods lack the global temporal context necessary for long-horizon consistency. Conversely, direct multi-frame one-shot prediction is computationally prohibitive at native resolutions. Thus, compressing these dynamics into a spatiotemporal latent space becomes necessary, yet it remains largely unexplored for mesh-based volumetric physics. Furthermore, the optimal predictive paradigm for this task remains an open question. Specifically, it is unclear whether deterministic regression or generative diffusion is better suited for complex physical simulation. To address these coupled challenges, we introduce PhysLDM, a unified latent-diffusion paradigm for one-shot volumetric deformable simulation. The foundation of PhysLDM is a novel holistic spatiotemporal VAE (ST-VAE). In our explorations, we find that standard temporal compression (e.g., video VAEs) loses high-frequency information and causes severe “staircase” artifacts. Our holistic ST-VAE effectively resolves this issue, achieving 2.48 mm precision on meter-scale scenes at up to 78 token compression. Based on this reliable latent space, we systematically compare regression and diffusion paradigms. Our experiments uncover a key modeling insight: complex deformable dynamics are inherently chaotic, and in this regime deterministic regression tends to produce damped, non-physical averages, whereas diffusion is better suited. Consequently, we employ a Latent Diffusion Model (LDM) that naturally respects this stochasticity to generate physically plausible trajectories. By training one single unified model on an Objaverse-scale simulation dataset in a constitutive-model-agnostic manner, PhysLDM achieves strong zero-shot generalization to unseen, out-of-distribution object datasets (GSO, Toys4K). To our knowledge, PhysLDM yields the first high-fidelity spatiotemporal autoencoder and latent-diffusion paradigm for volumetric deformable dynamics, offering a scalable and robust approach to neural simulation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Neural Simulation, Latent Diffusion, Spatiotemporal VAE, World Models, Physical AI

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_FrPyOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PhysLDM proposes a latent-diffusion paradigm for one-shot prediction of high-fidelity volumetric deformable trajectories. It introduces a holistic spatiotemporal VAE (ST-VAE) that decouples rigid motion from local deformation and removes the “staircase” artifact of standard temporal compression. A flow-matching DiT then predicts the full 64-frame latent trajectory in one pass, conditioned on the initial state. The paper also provides a systematic comparison between deterministic regression and generative diffusion under chaotic deformable dynamics, demonstrates zero-shot generalization to OOD datasets, and showcases differentiable applications including material estimation and second-order throw design.

Critical strengths

1.Strong technical contribution. The ST-VAE is well motivated: holistic temporal compression with explicit rigid/deformation decoupling is a clean solution to a real artifact in video-VAE-style compression. The reported 78× token compression with ~2.48 mm reconstruction error is impressive.

2.Insightful chaos analysis. The paper convincingly shows that GPU IPC solvers are not bit-reproducible and that chaotic deformable dynamics amplify tiny perturbations into macroscopically different outcomes. This explains why deterministic regression collapses to damped averages and why diffusion is better suited. The analysis in Appendix D is thorough and scientifically valuable.

3.Convincing empirical results. PhysLDM outperforms a capacity-matched regression model and an MGN-style AR baseline across Objaverse, GSO, and Toys4K on physics-consistency metrics and world-space error. Zero-shot ST-VAE generalization to OOD datasets and meshes beyond the training cap is solid.

Critical weaknesses

1.Main experiments are single-body. The main tables use n=1. Multi-body scenes are relegated to Appendix J and are only qualitatively shown. Real deformable simulation often involves multiple interacting objects, so this limits the immediate impact.

2.Real captured data requires fine-tuning. Zero-shot LDM prediction on real point clouds fails badly (875 mm error). The ST-VAE transfers zero-shot, but the full pipeline does not. The paper is honest about this, but it weakens the “zero-shot generalization” claim for real-world use.

3.Concurrent-work comparison is at low resolution. The comparison with Chen et al. (2026) is performed at ≤1,000 vertices because the concurrent model OOMs above that. This is fair given the memory constraint, but it does not establish superiority at native high resolution.

4.No direct comparison to the IPC solver. The training data come from StiffGIPC, but inference is compared only to neural baselines. A direct accuracy-vs-speed comparison against the ground-truth solver would strengthen the practical case.

Questions for authors

Please see weakness

Reviewer_V8VxOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PhysLDM predicts 64-frame trajectories of volumetric deformable bodies from their initial state and material stiffness. It combines rigid-motion and deformation factorization with a holistic spatiotemporal VAE to compress trajectories while reducing temporal staircase artifacts. A conditional latent diffusion model generates the full trajectory jointly and is compared against deterministic regression and autoregressive prediction. Experiments on Objaverse, GSO, and Toys4K demonstrate accurate reconstruction, improved prediction metrics, and zero-shot transfer, complemented by material-estimation and throw-design applications.

Critical strengths

The combination of rigid/deformation factorization and holistic temporal compression addresses a concrete bottleneck in volumetric trajectory modeling. This representation is consequential because it makes the latent prediction cost independent of the vertex count.

Critical weaknesses

-The evaluation does not establish conditional distribution fidelity. Appendix D.6 provides useful multi-sample results, but best-of-eight error and the correlation between sample spread and solver divergence do not establish calibrated outcome probabilities. Repeated solver samples on a representative subset, clearly defined coverage, and proper distributional scores would better substantiate the claimed modeling of multiple plausible futures.

-Physical fidelity remains insufficiently characterized. Table 8 reports volume-violation rates of 8.5% on Objaverse and 10.6% on Toys4K even for ST-VAE reconstruction, indicating that low positional error can coexist with substantial geometric defects. Further evaluation should include ground penetration, tetrahedral inversion, and upper-tail failures, while calibrating reference-relative velocity and acceleration checks against variability between valid solver runs.

Questions for authors

How much of the reported velocity/acceleration violation rate also occurs when comparing two valid solver runs? Please report ground-penetration and signed-volume/inversion statistics, including upper-tail scene results, and explain the sizeable violations already introduced by ST-VAE reconstruction in Table 8.

Reviewer_3idpOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes PhysLDM, a latent-diffusion framework that predicts the whole trajectory of a high-resolution volumetric deformable body in one pass. It first compresses each frame into a fixed set of latent tokens (spatial VAE), then compresses time with a spatiotemporal VAE (ST-VAE). The ST-VAE separates per-frame rigid motion from canonical deformation and replaces video-VAE-style chunked compression with a "holistic" temporal design, which removes "staircase" artifacts. A rectified-flow DiT then generates the trajectory latent from the initial state and stiffness. The model is trained purely kinematically on about 80k IPC simulations of Objaverse objects. The ST-VAE reaches 2.48 mm reconstruction error at up to 78× compression. PhysLDM beats a matched regression model and an autoregressive MeshGraphNets baseline on physics-consistency metrics, and it generalizes zero-shot to GSO and Toys4K.

Critical strengths
  1. Thorough pipeline from VAE to DiT. The paper builds and trains the full pipeline (spatial VAE, ST-VAE and latent DiT) on about 80k IPC trajectories, with ablations for each component. The experiments also demonstrate that the proposed pipeline generalizes zero-shot to GSO and Toys4K.
  2. Efficiency and differentiability. Inference cost does not depend on mesh resolution and runs at 40–80 fps. The model is fully differentiable and supports inverse tasks such as material estimation, where it is much faster than existing differentiable simulators.
Critical weaknesses
  1. Missing discussion of related 4D generation and physics works. The related work omits three groups of relevant methods:

    • Feed-forward 4D generation [1–3]. These methods share the core design with this paper. They compress dynamic mesh or Gaussian sequences into a compact latent with a VAE, then generate the latent trajectory with a diffusion or rectified-flow transformer conditioned on the initial shape. The authors should clearly state the advantage of the proposed pipeline over previous works.
    • Feed-forward generation with physics [4–5]. PhysCtrl [4] trains a diffusion model on large-scale simulated data to generate 3D point trajectories conditioned on physical parameters such as Young's modulus, which is very similar to PhysLDM's setting. Rozet et al. [5] already compare deterministic and diffusion emulators in a latent space for physics, which weakens the claim that the two paradigms "have never been systematically compared."
    • Non-feed-forward physics-based methods [6–11]. These methods combine physical simulators with per-scene optimization, often for material estimation. The paper compares only with DiffIPC and Newton/Warp, and mainly on speed and stability. The authors should also compare with these methods on accuracy, including physical fidelity and quality of the recovered materials.
  2. Simple physical modeling. Each object is assigned a single Young's modulus, and all other properties appear to be fixed, e.g., Poisson's ratio, density, and friction. This means every object is homogeneous. All scenes also share one setup: one to three objects dropped onto a ground plane. Because of this, the inverse task only recovers one global scalar from a simulated trajectory, which limits its practical value. Real objects are often heterogeneous, and prior work (e.g., PhysDreamer [7]) estimates spatially varying material fields.

  3. Limited visualization. The paper shows results only as static frames, which makes the dynamic results hard to judge. This matters most for temporal effects such as the staircase artifact, jitter and chaotic branching. I suggest that the authors provide some video comparisons in the supplementary material.

[1] Wu et al. AnimateAnyMesh: A Feed-Forward 4D Foundation Model for Text-Driven Universal Mesh Animation. ICCV 2025. arXiv:2506.09982.

[2] Zhang et al. Gaussian Variation Field Diffusion for High-fidelity Video-to-4D Synthesis. ICCV 2025. arXiv:2507.23785.

[3] Shi et al. Drive Any Mesh: 4D Latent Diffusion for Mesh Deformation from Video. arXiv:2506.07489.

[4] Wang et al. PhysCtrl: Generative Physics for Controllable and Physics-Grounded Video Generation. NeurIPS 2025. arXiv:2509.20358.

[5] Rozet et al. Lost in Latent Space: An Empirical Study of Latent Diffusion Models for Physics Emulation. NeurIPS 2025. arXiv:2507.02608.

[6] Xie et al. PhysGaussian: Physics-Integrated 3D Gaussians for Generative Dynamics. CVPR 2024.

[7] Zhang et al. PhysDreamer: Physics-Based Interaction with 3D Objects via Video Generation. ECCV 2024. arXiv:2404.13026.

[8] Huang et al. DreamPhysics: Learning Physical Properties of Dynamic 3D Gaussians with Video Diffusion Priors. arXiv:2406.01476.

[9] Liu et al. Physics3D: Learning Physical Properties of 3D Gaussians via Video Diffusion. arXiv:2406.04338.

[10] Lin et al. OmniPhysGS: 3D Constitutive Gaussians for General Physics-Based Dynamics Generation. ICLR 2025. arXiv:2501.18982.

[11] Feng et al. ElastoGen: 4D Generative Elastodynamics. arXiv:2405.15056.

Questions for authors
  1. What are the key technical differences between PhysLDM and feed-forward 4D generation methods such as AnimateAnyMesh [1], beyond the use of simulated volumetric data? Could the authors compare the ST-VAE with DyMeshVAE [1] on the same data?

  2. How does PhysLDM differ from PhysCtrl [4], which also generates physics-conditioned trajectories with diffusion? How do the authors' regression-vs-diffusion findings relate to those of Rozet et al. [5]?

  3. How does PhysLDM compare with non-feed-forward physics-based methods [6–11] in terms of physical fidelity and the accuracy of the estimated material, not only speed?

  4. Are all parts of an object assigned the same Young's modulus? Can the framework handle heterogeneous materials, other varying parameters (e.g., Poisson's ratio, density, friction) or scenarios beyond dropping objects onto the ground? How would material estimation work on real objects with spatially varying properties?

  5. Could the authors provide video comparisons with the baselines in the supplementary material, especially for the staircase artifact and chaotic scenes?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.