- Rating
- 4
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes GPWM, a generative world model that predicts 3D physical dynamics from particle trajectories. It handles rigid, deformable, cloth, and granular materials with one architecture. The model is an object-aware, multi-resolution spatio-temporal transformer. It is trained with flow matching and diffusion forcing, which allows streaming generation well beyond the 16-frame training window. The authors also build ParticleWorlds, a 25.5M-frame simulated corpus over six domains. They report 29–53% lower prediction error than the strongest applicable baseline in five of six domains, a single Unified model that covers all domains with a gap to the specialists, generalization to unseen shapes, object counts and stiffness, and stable 1,000-frame cloth rollouts. They also use latent noise optimization to turn the same model into a contact-rich planner for pushing, pile manipulation, and picking. Everything is evaluated in simulation, plus qualitative demos from single images.
Critical strengths
- Broad, carefully executed evidence. The paper covers six physical domains, four baselines retrained under the same splits and context frames, generalization tests, long-horizon rollouts, and planning. The writing is also candid about limitations: the Unified model's specialization gap, simulation-only data, quality filtering that does not guarantee solver convergence, and per-clip wins and losses.
- A useful recipe for long-horizon physical prediction and planning. Combining flow matching with diffusion forcing gives rollouts about 60× the training horizon, and reward backpropagation through the same generative model produces plans that the baselines fail to find (e.g., Picking 21/25 vs 0/25 for PointWorld and GNS). That the same model supports both prediction and planning is a nice result.
Critical weaknesses
- The contribution is mostly a well-engineered combination, and the main text barely isolates its parts. The multi-resolution DiT, factorized attention, flow matching with diffusion forcing, endpoint parameterization, and latent noise optimization are each existing ideas. The only ablation in the main paper removes diffusion forcing. Without ablations of the multi-resolution hierarchy, the factorized attention, or the rigid-body tokens, it is hard to tell which design choices explain the 29–53% gains over baselines that also differ in architecture and conditioning.
- Some headline claims are weaker than they first appear:
- The Unified model has about 5× the parameters of a Specialist (502M vs 103M) but is clearly worse on four of six domains (e.g., Free Fall 73.2 vs 44.4, Garment Folding 64.0 vs 34.8) and loses to PhysCtrl on Free Fall, Pushing and Dexterous Hands.
- Planning is compared against baselines with different candidate counts and budgets, there is no comparison to a standard simulator-based planner (such as MPPI/CEM with the true simulator) or to a trained policy, and it uses task-specific models rather than the Unified one.
- Evaluation uses one sample per method against one recorded future even though the paper stresses multimodality, and the cloth column uses a different metric (Chamfer at frame 1,000 on 8 garments).
- Everything is simulated with perfect, tracked particles.
Questions for authors
- Could you add ablations for the multi-resolution hierarchy, the factorized attention, and the rigid-body pose tokens, or explain which of these you believe matter most?
- Given the multimodality argument, could you report best-of-N error or a distribution-level metric, rather than a single sample against one recorded future?
- For planning, how do results change when baselines get a compute-matched budget, and how does GPWM compare with MPPI or CEM using the true simulator, or with a trained policy?
- Can the Unified model be used for planning, and what causes its specialization gap: capacity, data balance, or interference between domains?
- How sensitive is the model to noisy or incomplete particle tracking, which a real deployment would face?
7 comments
Q2. Best-of-N error and multimodality
"could you report best-of-N error or a distribution-level metric?"
Based on your suggestion, we now report best-of-N error on the held-out Free Fall test set (all 400 episodes). Drawing 10 samples per episode from the same observed context, a typical sample has 38.0 mm error and the best of 10 has 29.9 mm (−21%). Samples within an episode differ by 5.5 mm (one standard deviation), while the dataset-level error is stable across sampling seeds (38.0 ± 0.4 mm). The single sample reported in the paper is a typical draw, on average 8.0 mm worse than the best of the 10 samples and 9.3 mm better than the worst.
To emphasize why multimodality matters for dynamics prediction, we also visualize outputs from a deterministically rolled out free-fall model. While the average error is actually lower than a typical sample (36.1 vs. 38.0 mm), the predicted dynamics are often physically implausible. Specifically, in deterministic rollouts, objects are flattened and shrunken in the places where multi-modal rollouts would land samples in different places. Samples from the multi-modal rollouts look more realistic, and the best of 10 samples from the model is also more accurate (29.9 mm vs. 36.1 mm). Error against one recorded future therefore rewards hedging toward the mean, while drawing several samples shows the range of outcomes a planner should anticipate. We have included visualizations of this multi-modality experiment on a separate rebuttal tab on our anonymized website (https://gpwm.github.io/rebuttal/#samples).
We also note that effective planning depends on the generative formulation. GPWM learns the joint multi-modal distribution of actuator and object motion, and this is what makes planning in the latent noise space of the model meaningful. We reframe portions of our planning section to emphasize this more directly. We run additional experiments where we hold the GPWM checkpoint, the optimizer (CEM) and the compute budget fixed, and change only where the search happens. Searching GPWM's latent noise moves the object 48%, 21% and 40% of the way to its target on Pushing, Pile Manipulation and Picking. Searching raw actions in the same model makes almost no progress (0%, 1% and −3%).
Q3. Compute-matched planning and stronger baselines
"how do results change when baselines get a compute-matched budget, and how does GPWM compare with MPPI or CEM using the true simulator, or with a trained policy?"
We agree that the results in our original planning section were difficult to compare across methods. Some of this is hard to avoid when the models differ in size, backbone and inference procedure, but we have reformulated the planning experiment to present all methods in as unified a way as possible. We believe planning time is the best common measure of compute for these models. They take different numbers of forward passes per trajectory, and latent noise optimization also backpropagates through its rollouts. Wall-clock time on the same hardware accounts for all of these differences. We have extended the evaluation so that every planner is swept over its budget on one L40S GPU, up to 100 s for Pushing and 200 s for Pile Manipulation and Picking. Instead of reporting success at a single planning budget, we show the full curve of task progress against planning time. The curves are interactive at https://gpwm.github.io/rebuttal/#planning.
To unify the comparison further, we use the same search algorithm, the cross-entropy method (CEM), with every world model: over GPWM's latent noise, over raw actions in the same GPWM checkpoint, and over raw actions in PointWorld. Because every row searches the same way, the number of trajectories each method evaluates within the budget is a second, directly comparable measure of compute, and we report it alongside each result. This helps isolate the effect of the model and the search space from that of the search algorithm. We keep latent noise optimization as a gradient-based search over the same noise. We also add the two baselines you suggested: CEM over the true simulator, on the same GPU and time budget, and a closed loop goal-conditioned DP3 policy trained on the same data as the world models with hindsight relabeling. All plans except DP3's, which runs closed loop, are executed open loop in simulation, and all methods are scored with one metric, task progress = 1 − d_T / d_0: the fraction of the initial object-to-target distance closed at the final frame.
Table R4. Task progress (↑), with the number of trajectories searched in parentheses. Bold marks the best result in each column and underline the second best.
| Method | Pushing · 100 s | Pile Manipulation · 200 s | Picking · 200 s |
|---|---|---|---|
| GPWM, latent noise optimization | (~260) | 0.23 (~60) | 0.56 (~100) |
| GPWM, noise-space CEM | 0.48 (~570) | (~120) | (~250) |
| GPWM, action-space CEM | 0.00 (~570) | 0.01 (~130) | −0.03 (~250) |
| PointWorld, action-space CEM | 0.01 (~3k) | 0.19 (~1k) | 0.00 (~800) |
| GNS | 0.03 | 0.02 | 0.00 |
| CEM with the true simulator | 0.18 (~320k) | 0.08 (~430) | 0.30 (~720k) |
| Goal-conditioned DP3 (closed loop) | 0.76 | 0.16 | 0.18 |
Thank you for your constructive feedback and helpful suggestions. We have worked to address your main concerns below, and are continuing to work on the remaining points before the end of the discussion period. Changes in the revised PDF are marked in blue, and videos and interactive views of the new results are at https://gpwm.github.io/rebuttal/, organized by reviewer question.
Q1. Ablations of the hierarchy, factorized attention and pose tokens
Based on your suggestions, we added controlled ablations of the multi-resolution hierarchy, the relative position encoding, the factorized attention, and the rigid-body pose tokens. Since the hierarchy and the factorized attention were chosen primarily for efficiency, variants are trained from scratch under the same compute budget of 24 L40S GPU-hours and evaluated on the held-out Free Fall test set (400 episodes, mean per-particle error; errors are higher than in Table 1 of the paper because of the shorter training). The no-hierarchy variant runs every layer at full point resolution, the no-RoPE variant replaces the relative position encoding with absolute coordinates as input, and the joint-attention variant attends over space and time jointly instead of factorizing the two. The submission showed stable garment rollouts with diffusion forcing over about 60 times the training window. Since then we have retrained all GPWM models under slightly modified settings (see Q4), and we now ablate diffusion forcing directly with the retrained Garment Folding specialist. Rigid pose tokens are not applicable for the free fall model, so we ablate their inclusion on Pick and Place.
Table R1. Architecture ablations on Free Fall, 24 GPU-hours each.
Table R2. Diffusion forcing on Garment Folding (retrained specialist, Chamfer distance over 898 frames on the 20 held-out folds with the most motion).
Table R3. Pose tokens on Pick and Place (200 held-out episodes).
These results show that the particle hierarchy is essential for learning efficiently from large scenes. The non-hierarchical model trains more than five times slower than GPWM, so under the same compute budget removing the hierarchy raises error by 21% (51.9 to 62.6 mm). Replacing factorized attention with joint space-time attention has a similar effect. It also trains more slowly and raises error by 9% (56.8 mm). We placed explicit emphasis on making the architecture invariant to absolute 3D position, and the ablation shows that this pays off. With nearly the same number of training steps (54.6k vs. 54.2k), enforcing this invariance through 3D RoPE and relative point features reduces error by 9% compared with absolute-position input features (56.8 to 51.9 mm). Diffusion forcing remains essential for long-horizon rollouts: with the retrained specialist, it lowers the error over 898 frames of garment folding from 12.2 to 8.9 mm. Finally, pose tokens matter less for average error than for physical validity: plain particles distort rigid objects by 4.0 mm on average, while pose tokens preserve rigid shape exactly. These ablations are also shown at https://gpwm.github.io/rebuttal/#ablations.