- Rating
- 2
- Confidence
- 4(+2)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies long-horizon physical dynamics prediction under particle representations, with a focus on exploiting the final-state structure of dissipative or settling systems. ReStep first uses a conditional VAE composed of a GNN encoder and decoder to predict possible final states from the initial state; it then generates the full trajectory with a Double-UNet diffusion model conditioned on both the initial state and the predicted final state. Its “reverse-consequence” formulation should be understood as a conditional decomposition that predicts the outcome first and then generates the process, rather than reverse integration along physical time. The authors further connect particle prediction to 3D Gaussian Splatting for visual dynamics generation and physical parameter estimation based on initial and final multi-view observations. Experiments cover FlexiFall and Physion, and include component ablations, final-state conditioning replacement, comparisons of final-state predictors, computational overhead, and failure case analysis.
Critical strengths
-
The main strength of this paper is its approach to long-horizon dynamics prediction. For physical systems that eventually settle into stable configurations, predicting the final state first and using it to guide trajectory generation is a well-motivated idea. This provides a useful way to incorporate long-term constraints into the prediction process and reduces the reliance on sequential rollouts. The combination of stochastic endpoint prediction and anchored diffusion is particularly interesting for physical systems where multiple outcomes are possible.
-
The experimental evaluation is also a strong aspect of the paper. The authors consider a range of rigid and deformable scenarios and provide ablation studies that help isolate the contributions of endpoint prediction and diffusion-based trajectory generation. I particularly appreciate the analysis of different endpoint conditions and the relationship between endpoint errors and trajectory errors, as these experiments offer useful insight into the behavior of the proposed framework. The integration with 3D Gaussian Splatting further demonstrates the potential of the approach for visually grounded physical reasoning.
Critical weaknesses
-
Some important implementation details are missing from the method description. The paper treats physical properties such as friction, restitution, and mass as inputs to the steady-state predictor and uses this predictor for parameter estimation. However, the architecture in Table 5 only specifies position and velocity features, leaving it unclear how these physical parameters enter the network and how gradients with respect to them are computed. The diffusion model also applies convolutions along the particle dimension without explaining how particles are ordered. Since these operations can depend on particle ordering, this raises questions about the model's behavior across different particle configurations and scene compositions.
-
The experimental comparisons would also benefit from a more detailed description of the evaluation setup. In particular, the paper provides limited information about how the generative baselines are adapted to the 3D particle prediction task, including their conditioning inputs, architectural changes, and training configurations. There are also differences between the results reported in Table 2 and Table 8 for seemingly identical settings that need clarification. These details are important for understanding how much of the improvement comes from the proposed modeling approach. More broadly, since endpoint-conditioned trajectory generation has been explored in prior work, the paper could better distinguish its specific technical contributions from existing approaches.
Questions for authors
Could the authors clarify how physical parameters such as friction, restitution, and mass are incorporated into the steady-state predictor? Table 5 only lists position and velocity as input features, so it is unclear how these parameters influence the prediction and how gradients are computed during inverse parameter estimation. I would also like to know how the stochastic latent variable is handled during this optimization.
How are particles ordered in the second UNet, which applies convolutions along the particle dimension? Does the model rely on a fixed ordering, and how is this ordering maintained across scenes with different numbers of objects or particles? Have the authors examined whether consistently permuting the particle indices affects the predictions?
Could the authors provide more details on how LatentFlow and STFlow are adapted to the 3D particle prediction task, particularly regarding their conditioning inputs, architectural modifications, and training configurations? I also noticed some differences between the results in Table 2 and Table 8 for apparently identical settings. Could the authors explain the source of these differences and clarify whether the reported standard deviations are computed across test samples, random samples, or independent training runs?
Finally, how is the final state defined across the different physical scenarios? Is it determined by a convergence criterion or simply taken as the last frame of each sequence? Could the authors also clarify the training horizons used for the dynamics model and the rendering experiments? In particular, it would be helpful to distinguish generalization to different temporal resolutions from extrapolation to longer physical durations.
7 comments
Response to pqyY (2/3)
W2 . Relation to endpoint-conditioned generation
We agree that endpoint-conditioned generation exists. Sec. 2 cites it (Janner et al., 2022; Huang et al., 2023) as the conditioning mechanism we build on. The difference lies in where the endpoint comes from.
In prior work the endpoint is an input. In planning it is the goal, in motion or video in-betweening it is a key frame, in smoothing it is a later observation. The task is to interpolate between two known states. In simulation the endpoint is the unknown. Only s₀ is given, and every existing learned particle simulator we know of, autoregressive (GNS, GMN, SGNN, EGNO) or generative (LatentFlow, STFlow), generates from s₀ alone. Nevertheless, in contrast to previous fields, we investigate long-horizon and larger particle dynamics conditioned on initial particle states and physical attributes, with substantially longer and varying prediction horizons ranging from 70 to 452 steps, also larger particle numbers.
Our specific contributions are:
- The formulation. We recast forward simulation of dissipative systems as outcome prediction followed by boundary-value completion, p(s_T | s₀) · p(s_{1:T−1} | s₀, s_T). It rests on an observation that had not been used in learned simulators: the settled state is easier to predict in one shot than the transient.
- A stochastic steady-state predictor for particle systems. It maps (s₀, α) to a distribution over rest states in one step, with calibration reported in Table 7 and backbone independence in Table 9.
- Scalability. A trajectory denoiser for variable-size systems. The Double-UNet is convolutional along time and along the particle index, so one set of weights handles varying frame sequences and particles. Standard goal-conditioned diffusion models assume a fixed state dimension.
- Use as the dynamics module of a 3DGS pipeline (Sec. 4.3).
W2 / Q3. Generative baselines and standard deviations
We preserve LatentFlow’s generative formulation. The original LatentFlow operates on a regular image grid: conditioned on an RGB image, it generates 2D trajectories for image-grid points over frames. So our 3D particle trajectories cannot be passed directly into the original model. For our 3D particle data, we embed the trajectories particle-wise and add time embeddings. Masked averaging over particles and time yields a single latent. Conditioned on the initial state, the flow generates this latent, which is then decoded into a full trajectory. Thus, our adaptation retains the trajectory VAE → latent flow matching → trajectory decoding framework.
We preserve STFlow’s core methodological pipeline. Starting from the initial state , our implementation constructs random-walk trajectories. The vector field takes the current trajectory, the initial positions and physical attributes as input. It performs spatial message passing, followed by temporal convolutions. During training, conditional flow matching learns a flow from the random-walk trajectories to the ground-truth trajectories; at inference, ODE integration generates the predicted trajectory.
You can check the table in Appendix E, Table 15, for details:
| LatentFlow (Table 2, Physion) | STFlow (Fig. 3, FlexiFall) | |
|---|---|---|
| Model | trajectory VAE (per-particle encoder and decoder with learned time embedding; pooled 256-d latent) and flow matching in the latent space (vector field conditioned on the pooled initial state) | flow matching in trajectory space, starting from a random-walk prior anchored at s₀ (σ = 0.01); vector field of k-NN message-passing blocks (k = 8) and 4 dilated temporal convolution blocks, width 128 |
| Data | same as ReStep | same as ReStep |
| Training | AdamW, lr 10⁻⁴, weight decay 10⁻⁴, gradient clipping 1.0, 100 epochs, batch 2; loss = reconstruction + 10⁻⁴·KL + flow matching | AdamW, lr 10⁻⁴, weight decay 10⁻⁴, gradient clipping 1.0, 100 epochs |
| Test set | same as ReStep | same as ReStep |
How much of the improvement comes from our modeling approach? In our comparisons, the methods are evaluated on the same task-specific datasets under the same training setup, with the baselines and ReStep using models in the single-digit millions of parameters. Despite their different architectures, ReStep achieves the best results. The comparisons assess the effectiveness of the complete two-stage approach. In our ablation, we provided a more direct measure of the endpoint’s contribution within ReStep.
Clarify std. The reported standard deviations are computed across test scenes, not across repeated stochastic samples for the same scene or independent training runs. Additional runs with independent training seeds are underway; we will supplement the results once they finish.
Response to pqyY (3/3)
Q4. Definition of s_T and horizons
Definition of s_T. It differs between the two benchmarks.
- Physion: s_T is the rest state. Each trajectory is cut at the first frame after frame 50 at which the frame-to-frame displacement of the whole particle set falls below 10⁻⁴ (in norm).
- FlexiFall: s_T is the last frame of a fixed-length window. All scenes have settled by then.
Training horizons of the dynamics model. Both stages are trained on the full sequences defined above. On Physion a single model per scenario is trained on all sequence lengths at once.
Rendering experiments (Table 4). For each test scene, the Gaussian representation of both methods is fitted to the images of the first fixed number of frames only. "In-domain" are these frames; "out-of-domain" are the remaining frames of the same sequence, for which no image is ever seen. Temporal resolution versus physical duration. In the Dominoes temporal-resolution experiment, we insert one linearly interpolated frame between every pair of original frames. This changes sampled trajectory frames and the sampling interval, while leaving the physical duration unchanged. Like we said in the paper, we demonstrate the Anchored Diffusion process learns a robust, continuous mapping of physical evolution.
Response to pqyY (1/3)
The reviewer is right that Table 5 is incomplete. We have added the details below.
How the parameters enter. Flexifall (Fig. 3) : position, friction, restitution and mass of each object are the diferences between scenes. The input is "position + (friction, restitution, mass)" for the steady-state predictor. Physion (Table 2): position and initial motion, the official data contain no friction, restitution or mass. So the input is "position + initial velocity (Table 5)".
For Fig 3 and Table 3 experiments on Flexifall in which friction, restitution and mass vary per object. The network has the architecture of Table 5 with a parameter branch, MLP(3 → 128 → 128) on (friction, restitution, mass). Its output is concatenated with the position embedding before the three message-passing blocks, so the parameters influence every message and, through the pooled feature h₀, the prior and the decoder. We corrected the table 5 in the revised manuscript.
How the gradients are computed. The network is frozen. The only optimisation variables are the per-object parameters α ∈ ℝ^{M×3} (M objects). They are broadcast to the particles, the predictor is evaluated once, and the gradient of L_inv = ‖ŝ_T(α) − s_T‖² + 10⁻⁶‖α‖² is obtained by ordinary back-propagation through decoder, latent predictor, prior and encoder. No simulator roll-out is differentiated: one step of the optimiser costs one forward and one backward pass of a single network. We use Adam with learning rate 10⁻⁴ for 1000 steps.
The latent variable. Nothing is sampled during the optimisation. The latent is set to the prior mean, z = μ_p(h₀). Since h₀ depends on α, the gradient also flows through the prior. The objective is therefore a deterministic function of α.
Yes, the second UNet relies on a fixed particle order. A 1-D convolution along the particle index is not permutation-equivariant, and we do not claim that it is.
Which order, and how it is maintained. It is the canonical order of the particle representation, with no per-scene choice. Objects are concatenated in the simulator's object order. Within an object, the particles keep the order of its canonical point set, which is defined in the object's local frame and is therefore identical in every frame.
Experiment. We took the Table 2 checkpoints, without retraining, and applied one permutation consistently to the initial state, the predicted endpoint and the ground truth.
The accuracy of Stage 1 is unchanged. Stage 2 depends on the order, and the amount varies. Changing the order of the objects raises the error. So the model does use the order, and a different order at test time than in training degrades it, as it would for any convolutional model.
Our position. We view this as a practical architectural trade-off. Our denoiser operates on an object-wise serialization with persistent particle identities, allowing the same convolutional weights to process inputs with different particle counts. This supports variable-size systems under a consistent ordering convention, but does not imply permutation equivariance.
Using 1D convolutions along the particle dimension has a strong engineering rationale in physical-simulation diffusion models. The denoiser repeatedly processes trajectories of T timesteps over L denoising iterations. Fully connected GNN message passing among all P particles at every iteration incurs quadratic computational and memory costs when pairwise messages are explicitly constructed. Fixed-width, fixed-kernel 1D convolutions instead scale linearly with P, making repeated processing of large particle trajectories tractable. This is particularly relevant to engineering simulation and graphics pipelines, where persistent particle identities and fixed or alignable ordering are common.
Regarding object ordering, random test-time shuffling is an artificial stress test that disrupts the representation convention. Such a mismatch need not arise in a consistent simulation pipeline: the initial physical state supports deterministic spatial or kinematic sorting. Applying the same rule during training and inference, with deterministic tie-breaking as needed, maps different input orderings of the same scene to a common canonical object sequence. Under this protocol, arbitrary object indices do not cause ordering mismatches. Thus, sensitivity to deliberate shuffling does not undermine practical applicability when consistent serialization is maintained.