acceptodds
Under review as a conference paper at ICLR 2027

GPWM: Generative Particle World Models for 3D Prediction and Planning

Abstract

We introduce GPWM a generative world model that learns 3D physical dynamics directly from particle trajectories. Its shared representation and architecture covers rigid, deformable, cloth, and granular materials, in both passive physics and robotic interaction. GPWM is an object-aware multi-resolution transformer that combines flow matching with diffusion forcing to enable streaming generation far beyond the training horizon. We train and evaluate GPWM on ParticleWorlds, a 25.5-million-frame corpus spanning diverse geometries, physical properties, interactions, and robot embodiments. When trained within individual domains, GPWM reduces prediction error by 29-53% relative to the strongest applicable prior learned-dynamics models. Going beyond the domain-specific setting of prior methods, a single Unified GPWM jointly models heterogeneous physical regimes within one set of weights. GPWM further generalizes to unseen geometries, physical properties, object counts, and rollout horizons. Beyond prediction, GPWM enables contact-rich planning. An object-space goal reward is backpropagated through the learned interaction dynamics to the latent noise, steering actuator motion and the contacts it establishes. Executed open-loop in simulation, the resulting plans outperform unguided generation and planning with prior learned dynamics models on pushing, picking, and granular manipulation.

Primary area:
generative models
Keywords:
World Models, Dynamics Prediction

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_1hZxOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes GPWM, a generative world model that predicts 3D physical dynamics from particle trajectories. It handles rigid, deformable, cloth, and granular materials with one architecture. The model is an object-aware, multi-resolution spatio-temporal transformer. It is trained with flow matching and diffusion forcing, which allows streaming generation well beyond the 16-frame training window. The authors also build ParticleWorlds, a 25.5M-frame simulated corpus over six domains. They report 29–53% lower prediction error than the strongest applicable baseline in five of six domains, a single Unified model that covers all domains with a gap to the specialists, generalization to unseen shapes, object counts and stiffness, and stable 1,000-frame cloth rollouts. They also use latent noise optimization to turn the same model into a contact-rich planner for pushing, pile manipulation, and picking. Everything is evaluated in simulation, plus qualitative demos from single images.

Critical strengths
  • Broad, carefully executed evidence. The paper covers six physical domains, four baselines retrained under the same splits and context frames, generalization tests, long-horizon rollouts, and planning. The writing is also candid about limitations: the Unified model's specialization gap, simulation-only data, quality filtering that does not guarantee solver convergence, and per-clip wins and losses.
  • A useful recipe for long-horizon physical prediction and planning. Combining flow matching with diffusion forcing gives rollouts about 60× the training horizon, and reward backpropagation through the same generative model produces plans that the baselines fail to find (e.g., Picking 21/25 vs 0/25 for PointWorld and GNS). That the same model supports both prediction and planning is a nice result.
Critical weaknesses
  • The contribution is mostly a well-engineered combination, and the main text barely isolates its parts. The multi-resolution DiT, factorized attention, flow matching with diffusion forcing, endpoint parameterization, and latent noise optimization are each existing ideas. The only ablation in the main paper removes diffusion forcing. Without ablations of the multi-resolution hierarchy, the factorized attention, or the rigid-body tokens, it is hard to tell which design choices explain the 29–53% gains over baselines that also differ in architecture and conditioning.
  • Some headline claims are weaker than they first appear:
  1. The Unified model has about 5× the parameters of a Specialist (502M vs 103M) but is clearly worse on four of six domains (e.g., Free Fall 73.2 vs 44.4, Garment Folding 64.0 vs 34.8) and loses to PhysCtrl on Free Fall, Pushing and Dexterous Hands.
  2. Planning is compared against baselines with different candidate counts and budgets, there is no comparison to a standard simulator-based planner (such as MPPI/CEM with the true simulator) or to a trained policy, and it uses task-specific models rather than the Unified one.
  3. Evaluation uses one sample per method against one recorded future even though the paper stresses multimodality, and the cloth column uses a different metric (Chamfer at frame 1,000 on 8 garments).
  4. Everything is simulated with perfect, tracked particles.
Questions for authors
  • Could you add ablations for the multi-resolution hierarchy, the factorized attention, and the rigid-body pose tokens, or explain which of these you believe matter most?
  • Given the multimodality argument, could you report best-of-N error or a distribution-level metric, rather than a single sample against one recorded future?
  • For planning, how do results change when baselines get a compute-matched budget, and how does GPWM compare with MPPI or CEM using the true simulator, or with a trained policy?
  • Can the Unified model be used for planning, and what causes its specialization gap: capacity, data balance, or interference between domains?
  • How sensitive is the model to noisy or incomplete particle tracking, which a real deployment would face?
7 comments

Thank you for your constructive feedback and helpful suggestions. We have worked to address your main concerns below, and are continuing to work on the remaining points before the end of the discussion period. Changes in the revised PDF are marked in blue, and videos and interactive views of the new results are at https://gpwm.github.io/rebuttal/, organized by reviewer question.

Q1. Ablations of the hierarchy, factorized attention and pose tokens

"Could you add ablations for the multi-resolution hierarchy, the factorized attention, and the rigid-body pose tokens"

Based on your suggestions, we added controlled ablations of the multi-resolution hierarchy, the relative position encoding, the factorized attention, and the rigid-body pose tokens. Since the hierarchy and the factorized attention were chosen primarily for efficiency, variants are trained from scratch under the same compute budget of 24 L40S GPU-hours and evaluated on the held-out Free Fall test set (400 episodes, mean per-particle error; errors are higher than in Table 1 of the paper because of the shorter training). The no-hierarchy variant runs every layer at full point resolution, the no-RoPE variant replaces the relative position encoding with absolute coordinates as input, and the joint-attention variant attends over space and time jointly instead of factorizing the two. The submission showed stable garment rollouts with diffusion forcing over about 60 times the training window. Since then we have retrained all GPWM models under slightly modified settings (see Q4), and we now ablate diffusion forcing directly with the retrained Garment Folding specialist. Rigid pose tokens are not applicable for the free fall model, so we ablate their inclusion on Pick and Place.

Table R1. Architecture ablations on Free Fall, 24 GPU-hours each.

VariantErrorTraining steps
GPWM51.9 mm55k
No hierarchy62.6 mm (+21%)10k
No RoPE56.8 mm (+9%)54k
Joint space-time attention56.8 mm (+9%)35k

Table R2. Diffusion forcing on Garment Folding (retrained specialist, Chamfer distance over 898 frames on the 20 held-out folds with the most motion).

VariantError
GPWM8.9 mm
No diffusion forcing12.2 mm (+37%)

Table R3. Pose tokens on Pick and Place (200 held-out episodes).

Pose tokens (ours)Plain particles
Object error12.5 mm13.3 mm
Rigidity error (mean / final frame)0.0 / 0.0 mm4.0 / 6.9 mm

These results show that the particle hierarchy is essential for learning efficiently from large scenes. The non-hierarchical model trains more than five times slower than GPWM, so under the same compute budget removing the hierarchy raises error by 21% (51.9 to 62.6 mm). Replacing factorized attention with joint space-time attention has a similar effect. It also trains more slowly and raises error by 9% (56.8 mm). We placed explicit emphasis on making the architecture invariant to absolute 3D position, and the ablation shows that this pays off. With nearly the same number of training steps (54.6k vs. 54.2k), enforcing this invariance through 3D RoPE and relative point features reduces error by 9% compared with absolute-position input features (56.8 to 51.9 mm). Diffusion forcing remains essential for long-horizon rollouts: with the retrained specialist, it lowers the error over 898 frames of garment folding from 12.2 to 8.9 mm. Finally, pose tokens matter less for average error than for physical validity: plain particles distort rigid objects by 4.0 mm on average, while pose tokens preserve rigid shape exactly. These ablations are also shown at https://gpwm.github.io/rebuttal/#ablations.

Q2. Best-of-N error and multimodality

"could you report best-of-N error or a distribution-level metric?"

Based on your suggestion, we now report best-of-N error on the held-out Free Fall test set (all 400 episodes). Drawing 10 samples per episode from the same observed context, a typical sample has 38.0 mm error and the best of 10 has 29.9 mm (−21%). Samples within an episode differ by 5.5 mm (one standard deviation), while the dataset-level error is stable across sampling seeds (38.0 ± 0.4 mm). The single sample reported in the paper is a typical draw, on average 8.0 mm worse than the best of the 10 samples and 9.3 mm better than the worst.

To emphasize why multimodality matters for dynamics prediction, we also visualize outputs from a deterministically rolled out free-fall model. While the average error is actually lower than a typical sample (36.1 vs. 38.0 mm), the predicted dynamics are often physically implausible. Specifically, in deterministic rollouts, objects are flattened and shrunken in the places where multi-modal rollouts would land samples in different places. Samples from the multi-modal rollouts look more realistic, and the best of 10 samples from the model is also more accurate (29.9 mm vs. 36.1 mm). Error against one recorded future therefore rewards hedging toward the mean, while drawing several samples shows the range of outcomes a planner should anticipate. We have included visualizations of this multi-modality experiment on a separate rebuttal tab on our anonymized website (https://gpwm.github.io/rebuttal/#samples).

We also note that effective planning depends on the generative formulation. GPWM learns the joint multi-modal distribution of actuator and object motion, and this is what makes planning in the latent noise space of the model meaningful. We reframe portions of our planning section to emphasize this more directly. We run additional experiments where we hold the GPWM checkpoint, the optimizer (CEM) and the compute budget fixed, and change only where the search happens. Searching GPWM's latent noise moves the object 48%, 21% and 40% of the way to its target on Pushing, Pile Manipulation and Picking. Searching raw actions in the same model makes almost no progress (0%, 1% and −3%).

Q3. Compute-matched planning and stronger baselines

"how do results change when baselines get a compute-matched budget, and how does GPWM compare with MPPI or CEM using the true simulator, or with a trained policy?"

We agree that the results in our original planning section were difficult to compare across methods. Some of this is hard to avoid when the models differ in size, backbone and inference procedure, but we have reformulated the planning experiment to present all methods in as unified a way as possible. We believe planning time is the best common measure of compute for these models. They take different numbers of forward passes per trajectory, and latent noise optimization also backpropagates through its rollouts. Wall-clock time on the same hardware accounts for all of these differences. We have extended the evaluation so that every planner is swept over its budget on one L40S GPU, up to 100 s for Pushing and 200 s for Pile Manipulation and Picking. Instead of reporting success at a single planning budget, we show the full curve of task progress against planning time. The curves are interactive at https://gpwm.github.io/rebuttal/#planning.

To unify the comparison further, we use the same search algorithm, the cross-entropy method (CEM), with every world model: over GPWM's latent noise, over raw actions in the same GPWM checkpoint, and over raw actions in PointWorld. Because every row searches the same way, the number of trajectories each method evaluates within the budget is a second, directly comparable measure of compute, and we report it alongside each result. This helps isolate the effect of the model and the search space from that of the search algorithm. We keep latent noise optimization as a gradient-based search over the same noise. We also add the two baselines you suggested: CEM over the true simulator, on the same GPU and time budget, and a closed loop goal-conditioned DP3 policy trained on the same data as the world models with hindsight relabeling. All plans except DP3's, which runs closed loop, are executed open loop in simulation, and all methods are scored with one metric, task progress = 1 − d_T / d_0: the fraction of the initial object-to-target distance closed at the final frame.

Table R4. Task progress (↑), with the number of trajectories searched in parentheses. Bold marks the best result in each column and underline the second best.

MethodPushing · 100 sPile Manipulation · 200 sPicking · 200 s
GPWM, latent noise optimization0.57‾\underline{0.57} (~260)0.23 (~60)0.56 (~100)
GPWM, noise-space CEM0.48 (~570)0.21‾\underline{0.21} (~120)0.40‾\underline{0.40} (~250)
GPWM, action-space CEM0.00 (~570)0.01 (~130)−0.03 (~250)
PointWorld, action-space CEM0.01 (~3k)0.19 (~1k)0.00 (~800)
GNS0.030.020.00
CEM with the true simulator0.18 (~320k)0.08 (~430)0.30 (~720k)
Goal-conditioned DP3 (closed loop)0.760.160.18
Reviewer_wZV8OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces GPWM, a generative particle world model for 3D physical prediction and planning. It combines an object-aware multi-resolution transformer, flow matching, diffusion forcing, and reward-guided trajectory generation. The authors also introduce ParticleWorlds, a 25.5-million-frame dataset spanning six physical domains, and evaluate prediction, generalization, and simulator-based planning.

Critical strengths
  1. The dataset covers diverse materials and robot interactions, providing a useful resource for studying general-purpose physical dynamics.
  2. The reported prediction results are encouraging, particularly the long-horizon cloth rollouts and generalization to unseen geometries and object counts.
  3. The generated plans are executed in the underlying simulator, providing evidence of practical utility than evaluation solely within the learned model.
Critical weaknesses
  1. The methodological contribution appears mainly incremental. The framework integrates established particle hierarchies, generative objectives, diffusion forcing, and noise optimization.
  2. Comparison with the closely related PhysiFormer, which is discussed but not evaluated, would clarify the advance.
  3. The Unified model is substantially larger than the specialists yet is less accurate on five of six domains, and planning uses task-specific models. Moreover, complete tracked 3D states are assumed, while image-based demonstrations use assigned materials and constructed histories. Unified-model planning and tests under imperfect observations would better support broader applicability.
Questions for authors

Please see Weaknesses.

4 comments

Thank you for your constructive feedback and helpful suggestions. We have worked to address your main concerns below, and are continuing to work on the remaining points before the end of the discussion period. Changes in the revised PDF are marked in blue, and videos and interactive views of the new results are at https://gpwm.github.io/rebuttal/, organized by reviewer question.

Q1. Methodological contribution

"The methodological contribution appears mainly incremental"

We are glad the reviewer finds the long-horizon cloth rollouts and the generalization to unseen geometries and object counts encouraging. We agree that each ingredient the reviewer lists builds on prior work, but we respectfully disagree that their integration is incremental. The results the reviewer highlights come from this integration, and so does GPWM's accuracy more broadly: it predicts deformable, granular, cloth and rigid dynamics better than every baseline in all six domains. The integration also unlocks a new capability, diffusion planning over 3D particle states from nothing more than a target position for an object. To our knowledge, this has not been shown before, because prior learned dynamics models lack the combination of components that latent noise optimization over 3D particle states depends on. Below we emphasize the role of each ingredient the reviewer lists, and why GPWM needs them together for both accurate prediction and planning:

  1. Particle hierarchies. The hierarchy moves most interaction modeling to coarse tokens, which lets GPWM handle scenes of up to 12,000 particles. Because it sits inside a denoiser of noisy particle trajectories, anchors are real particles chosen once and kept fixed across the window, so temporal attention can follow them, and pooling neighborhoods are recomputed from the trajectory being denoised. In our new compute-matched ablation, removing the hierarchy raises Free Fall error by 21%.
  2. Generative objectives. Contact makes several futures plausible, and a deterministic model averages them into implausible shapes (https://gpwm.github.io/rebuttal/#samples). GPWM instead samples distinct, plausible futures, which also gives a planner a distribution over actuator and object motion to search. Rigid bodies are generated as pose tokens alongside the particles, so their shapes stay exact beside deformable, granular and cloth material.
  3. Diffusion forcing. To our knowledge, GPWM is the first particle model trained with diffusion forcing, which teaches it to continue from imperfect history, including its own predictions. This keeps long rollouts stable: action-conditioned rollouts, including the cloth rollouts the reviewer highlights, reach nearly 60 times the 16-frame training window, and our planning horizons are 3 to 7.5 times that window.
  4. Noise optimization. Prior diffusion planners work on low-dimensional states (Janner et al., 2022) or plan in pixels and need a separate inverse-dynamics model to recover actions (Du et al., 2023). GPWM brings this to 3D particle states. Because it generates the actuator and the scene together, optimizing its input noise searches over coordinated actuator and object motion instead of raw actions. This depends on the other three ingredients: a generative model to search, stable rollouts over the planning horizon, and a hierarchy that scales to full scenes.

Other design decisions matter as well. For example, GPWM is invariant to absolute xy position, and replacing its relative 3D encoding with absolute coordinates raises Free Fall error by 9% (see https://gpwm.github.io/rebuttal/#ablations).

Q2. Comparison with PhysiFormer

"Comparison with the closely related PhysiFormer … would clarify the advance"

We thank the reviewer for this suggestion and now include PhysiFormer on our Free Fall benchmark, per your request. We run the authors' code unchanged, scaled to about 100M parameters like our model and PhysCtrl, and train and test it exactly as the other methods. It reaches 86.0 mm, against 53.0 mm for PhysCtrl and 40.0 mm for our Free Fall specialist. Side-by-side rollouts on three typical test clips are at https://gpwm.github.io/rebuttal/#physiformer.

We believe the main reason is its representation. PhysiFormer denoises absolute vertex positions, normalized by the size of the scene, so the motion it must predict is a small fraction of the quantity it generates. GPWM and PhysCtrl instead predict motion relative to the observed frames, so their output is the motion itself. PhysCtrl is a diffusion transformer of the same size trained the same way, so its much lower error points to the representation rather than to model size or training. Our own ablation shows similar effects on the input side. Replacing GPWM's relative position encoding with absolute coordinates raises Free Fall error by 9% at almost the same number of training steps (see our response to Reviewer 1hZx).

Q3. The Unified model is larger but less accurate

"substantially larger than the specialists yet is less accurate on five of six domains"

Since the submission, we retrained all GPWM models with an improved optimizer (Muon on the weight matrices) and fixed a bug where register tokens were not being carried across multiple blocks of the transformer. We also train the Unified model for longer, about 585k steps instead of 300k, so it now receives about 97k updates per domain instead of 50k. With these revisions, the Unified model is now better than per-domain baselines in 5 of 6 settings, better than our own specialist models in 2 of 6, and significantly closes the gap to our specialist models in the other four.

Table R1. Revised Table 1, prediction error in mm (↓). Bold marks the best result in each column and underline the second best.

MethodFree FallDraggingGarment Folding†PushingPick and PlaceDex. Hands
Graph/grid-based97.230.420.523.968.226.7
PointWorld––10.6‾\underline{10.6}28.739.827.4
PhysCtrl53.07.810.724.534.124.8
GPWM (Specialist)40.06.5‾\underline{6.5}8.913.312.5‾\underline{12.5}18.7
GPWM (Unified)41.5‾\underline{41.5}6.110.818.0‾\underline{18.0}11.122.6‾\underline{22.6}

†Garment Folding now reports symmetric Chamfer distance averaged over all 898 predicted frames on 20 held-out folds, instead of at the final frame on 8 garments, so that it reflects the whole rollout.

Regarding size: the Unified model's parameters are spread across six domains, leaving it with slightly less capacity per domain than a single specialist. We believe its lower accuracy in the submission came mainly from under-training and difficult training dynamics. The submitted Unified model received only about a third of the per-domain updates of each specialist. Training across domains with AdamW also produced large loss spikes, which Muon's orthogonalized updates reduced. After doubling its training steps and switching the optimizer, the revised Unified model narrows the gap to the specialists, surpassing them on two domains, and outperforms every baseline on five of six.

Q4. Imperfect observations

"tests under imperfect observations would better support broader applicability."

Per your request we added a study of robustness to imperfect observations. It covers two errors a real perception pipeline is likely to introduce:

  1. Position noise. We add Gaussian noise with standard deviation σ to every observed particle position, as from depth sensing.
  2. Tracking errors. We select a fraction of the particles and swap each one's history with that of a neighboring particle of the same object within 3 cm, as when a point tracker confuses nearby points. The swapped particles then carry another particle's motion.

We apply each corruption to the four observed frames of the Free Fall model and score its predictions against the clean ground truth on 400 held-out episodes:

Table R2. Free Fall error (mm) with corrupted observed frames.

Corruption of the observed contextFree Fall error
None (clean)38.0 mm
Position noise, σ = 2 mm40.2 mm (+6%)
Position noise, σ = 5 mm48.7 mm (+28%)
Tracking errors: 5% of particles swapped with a neighbor within 3 cm38.5 mm (+1%)
Tracking errors: 10% of particles39.3 mm (+3%)
Tracking errors: 25% of particles40.9 mm (+8%)

GPWM is robust to small position noise and to tracking errors: with a quarter of the particles' tracks corrupted, error rises by only 8%. Larger position noise (σ = 5 mm) costs more (+28%), which is expected because it materially changes object surfaces and therefore contacts. Rollouts from clean and corrupted inputs are at https://gpwm.github.io/rebuttal/#observations. GPWM does assume complete particle sets, and handling occlusion directly remains future work, as noted in our limitations.

Thank you for the authors' responses. My concerns have been largely resolved. Furthermore, I think Specialist and Unified models should be evaluated under the same training settings. In the revised version, the blank line between lines 486 and 488 needs to be removed.

Overall, I think this work is acceptable due to its relatively complete design and efficient results. I am improving my rating from Weak rejection to Weak acceptance.

Reviewer_xb8QOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a systematic effort in building a particle-based world model with several features: (1) multi-modal prediction with a DiT architecture (2) long-range streaming with diffusion forcing (3) cross-category training using a new synthetic dataset (with a limited set of categories) (4) hierarchical design to encode a large amount of particles. Each of the components is not new, and the combined effort doesn't really unlock new capability compared to existing models. The models are trained using synthetic datasets, yet the rollouts demonstrate significant artifacts in the videos. The contributed dataset, although claiming a large amount of frames, yet only contains 6 categories, which strongly limits the generalizability of the model. The planning experiment using the proposed world model is only conducted in the simulator, with limited tasks and baselines.

Critical strengths
  • The built system seems coherent and complete. The paper demonstrates a good systematic effort: from collecting a synthetic dataset, to having an aggregated design of different components, to having a complete story and experiments. The efforts, as a whole, can be a valuable contribution to the community.
Critical weaknesses
  • The novelty of the proposed method is limited. Each of the proposed components comes from existing literature. For example, the hierarchical design has roots in [Mrowca et al 2018], yet those literatures are not cited or discussed. Having a unified design of particle simulator for diverse phenomena is also not new - which could be found in [Wang et al. 2026] or PointWorld. Therefore, I consider this paper more like a good combination of existing modules, rather than a significant novel contribution.
  • The planning experiment is limited. It only conducts 3 tasks, and no sim2real experiment is shown. Only two baselines are compared in this experiment, and it's not clear whether the model can surpass a classical simulator (maybe crafted with an agentic system) in planning.
  • Although claiming the model to be "generative", it doesn't show the advantage of being "generative". Most metrics are evaluated in prediction error, while for a generative model, this may not necessarily indicate the right goal to pursue.

[Mrowca et al. 2018] Flexible Neural Representation for Physics Prediction; NeurIPS 2018. [Wang et al. 2026] WorldParticle: Unified World Simulation of Lagrangian Particle Dynamics via Transformer; arXiv 2026.

Questions for authors
  • Can this model be used in more complicated planning tasks, like "garment folding", "dragging a deformable", or "dexterous manipulation"? (all within the training domains)? Moreover, can those planning be transferred to real world? What's the success rate?
6 comments
AuthorsOpenReview ↗

Q1. Novelty and new capabilities

Thank you for your constructive feedback and helpful suggestions. We have worked to address your main concerns below, and are continuing to work on the remaining points before the end of the discussion period. Changes in the revised PDF are marked in blue, and videos and interactive views of the new results are at https://gpwm.github.io/rebuttal/, organized by reviewer question.

Q1. Novelty and new capabilities

"the combined effort doesn't really unlock new capability compared to existing models … more like a good combination of existing modules"

First, we thank the reviewer for recognizing GPWM as a coherent and complete system, and for noting that the effort, as a whole, "can be a valuable contribution to the community." We agree that GPWM's individual ingredients build on prior work, and we have expanded our citations following the reviewer's suggestions. However, we respectfully disagree that their combination unlocks no new capabilities. GPWM predicts deformable, granular, cloth and rigid dynamics, driven by rods, grippers and articulated hands, more accurately than every baseline in all six domains. This relies on deliberate design decisions. For example, GPWM is invariant to absolute xy position, and in our new ablation, replacing its relative 3D encoding with absolute coordinates raises Free Fall error by 9%. GPWM also makes diffusion planning possible in diverse 3D scenes, from nothing more than a target position for an object. Diffusion planning has been shown in low-dimensional states, pose sequences and pixels, but to our knowledge not in 3D particle states. Prior learned dynamics models do not support it. Latent noise optimization over 3D particle states is not simply a matter of swapping in a new state space, it needs a generative world model designed specifically with this planning in mind. GPWM brings together the following properties, which support both accurate prediction and planning and which, to our knowledge, no prior particle model combines:

  1. Generative. Contact, and properties that a short history cannot reveal, make several futures plausible. A deterministic model averages them into implausible shapes (see the multi-modality examples at https://gpwm.github.io/rebuttal/#samples). GPWM samples distinct, plausible futures, which gives a planner a distribution to search.
  2. Stable long rollouts. To our knowledge, GPWM is the first particle model trained for recurrent generation, and the first to use diffusion forcing to handle error accumulation across long rollouts. Diffusion forcing teaches it to continue from imperfect history, including its own earlier predictions, which keeps its streaming rollouts stable far beyond the 16-frame training window. Action-conditioned rollouts reach nearly 60 times the training window (Garment Folding and Pick/Place), and planning horizons 3 to 7.5 times.
  3. Exact rigid bodies. Rigid bodies are pose tokens denoised jointly with the deformable particles, so they keep their exact shape over any rollout. This also applies to rigid actuators: a generated rod or gripper stays rigid, so its predicted poses can be executed directly as commands.
  4. Any actuator. Actuators use the same particle and rigid-body representation as every other object in the scene, so no embodiment-specific action space is needed and one model handles rods, grippers, articulated hands and dragged material. An actuator's motion can be given as input, for action-conditioned prediction, or generated together with the rest of the scene, for planning, as naturally as any other object's.
  5. Planning in the latent space. Because GPWM generates the actuator together with the scene, its latent space captures common modes of coordinated contact. GPWM plans by searching this space rather than raw actions.

Q1 (continued). Novelty and new capabilities

Table R1 summarizes how these properties compare across GPWM and the prior particle models closest to it, including the two works the reviewer mentions.

Table R1. Properties of GPWM and prior particle models.

ModelGenerativeRolloutRigid bodiesRobot or actuatorPlanning
HRN (Mrowca et al., 2018)NoOne step, fed backParticles with a distance-preserving lossExternal forces as inputNone
WorldParticle (Wang et al., 2026)NoAutoregressiveStatic boundary particlesExternal forces as input, no robotInverse design of a friction value
PointWorld (Huang et al., 2026)NoOne chunk of about 1 s, no recurrenceFree pointsRobot points as inputMPPI over actions
PhysCtrl (Wang et al., 2025)YesOne fixed windowFree pointsA force and a drag point, no robotNone
PhysiFormer (Chen et al., 2026)YesOne fixed windowPer-vertex trajectoriesNoneNone
UniClothDiff (Tian et al., 2025)Yes5-frame windowCloth onlyEnd-effector motion as inputMPPI over actions
GPWM (ours)YesStreaming with diffusion forcingExact, as SE(3) pose tokensRods, grippers, hands; given or generatedSearch in the noise of our model, which is only possible because we jointly generate actuator and multi-object motion

Q2. Hierarchical design and HRN

"The hierarchical design has roots in [Mrowca et al 2018], yet those literatures are not cited"

We thank the reviewer for pointing us to the Hierarchical Relation Network, which the revision now cites and discusses in Section 2, along with the broader line of hierarchical point processing from PointNet++ (Qi et al., 2017) to Point Transformer V3 (Wu et al., 2024). Hierarchies have been so consistently successful in these models that we treated building one into GPWM as a necessity, rather than as a contribution of our own. Doing so in a diffusion model over particle trajectories, however, required a few design decisions that standard point-cloud models do not need to make.

  1. Downsampling that preserves trajectories. Temporal attention needs each coarse token to follow the same material across the window. We therefore pool around anchor particles, as in PointNet++, rather than with a voxel grid. These anchors are chosen in the first frame and kept fixed across the window, so each coarse token is a real particle with its own trajectory.
  2. Pooling neighbors from noisy positions. During denoising the future particle positions are not yet known. At every frame we compute the positions implied by the current noisy trajectory and choose pooling neighbors from those. The hierarchy then follows the scene as noise is removed.

Together these let attention span the whole scene at coarse resolution and scale GPWM to 12,000 particles. Our new ablations show that without the hierarchy, training is five times slower, and at matched compute the error rises by 21% (see https://gpwm.github.io/rebuttal/#ablations).

Q3. A unified particle simulator is not new

"Having a unified design of particle simulator for diverse phenomena is also not new - which could be found in [Wang et al. 2026] or PointWorld"

These two works are indeed related, and the difference lies in what is unified.

  1. GPWM's Unified model is one set of generative weights across passive, dragged and robot-actuated domains, with material conditioning and actuators from rods to articulated hands. It outperforms every baseline on five of the six domains and our own specialists on two (revised Table 1). Qualitatively, this unification also lets GPWM compose behaviors such as predicting a garment being folded by a gripper it never saw in that domain (all 10 held-out folds are shown at https://gpwm.github.io/rebuttal/#gripper-swap).
  2. WorldParticle shares one architecture across six categories of dynamics but trains a separate network for each and names a shared-weight model as future work. It is deterministic, has no robot actuation, and treats rigid geometry only as static boundaries.
  3. PointWorld is a single checkpoint, built only for robot action conditioned prediction. It is deterministic, takes no material or physical parameters, operates over partial camera inputs, and predicts a fixed one second per forward pass (no recurrence). It is not naturally trained on full point clouds or rolled out auto-regressively. We had to make these modifications so that this baseline could be used for long horizon prediction and planning.

Q4. The advantage of being generative

"it doesn't show the advantage of being 'generative'"

We respectfully disagree that the submission shows no advantage of being generative, but we agree that prediction error alone does not show it, and we have added experiments that test it directly.

Effective planning depends on the generative formulation. GPWM learns the joint multi-modal distribution of actuator and object motion, and this is what makes planning in the latent noise space of the model meaningful. We reframe portions of our planning section to emphasize this more directly. We run additional experiments where we hold the GPWM checkpoint, the optimizer (CEM) and the compute budget fixed, and change only where the search happens. Searching GPWM's latent noise moves the object 48%, 21% and 40% of the way to its target on Pushing, Pile Manipulation and Picking. Searching raw actions in the same model makes almost no progress (0%, 1% and −3%).

Best-of-N error. To specifically analyze the importance of having a generative model for dynamics prediction, we now report best-of-N error on the held-out Free Fall test set (all 400 episodes). Drawing 10 samples per episode from the same observed context, a typical sample has 38.0 mm error and the best of 10 has 29.9 mm (−21%). Samples within an episode differ by 5.5 mm (one standard deviation), while the dataset-level error is stable across sampling seeds (38.0 ± 0.4 mm). The single sample reported in the paper is a typical draw, on average 8.0 mm worse than the best of the 10 samples and 9.3 mm better than the worst.

Deterministic prediction. We also visualize the output of a deterministic rollout from the free-fall model. While the average error is actually lower than a typical sample (36.1 vs. 38.0 mm), the predicted dynamics are often physically implausible. Specifically, in deterministic rollouts, objects are flattened and shrunken in the places where multi-modal rollouts would land samples in different places. Samples from the multi-modal model look more realistic, and the best of 10 samples from the model is also more accurate (29.9 mm vs. 36.1 mm). So error against one recorded future rewards hedging toward the mean, while drawing several samples shows the range of outcomes a planner should anticipate. We have included visualizations of this multi-modality experiment on a separate rebuttal tab on our anonymized website (https://gpwm.github.io/rebuttal/#samples).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.