acceptodds
Under review as a conference paper at ICLR 2027

MeshPriorDiT: Hierarchical Modeling for Action-Conditioned Cloth Dynamics

Abstract

Action-conditioned cloth prediction must preserve material structure while limiting error accumulation during autoregressive rollouts. Mesh graph networks encode connectivity, but local propagation can miss distant interactions; global models capture such interactions without guaranteeing locally consistent motion. We introduce MeshPriorDiT, which treats a constrained mesh rollout as an explicit, correctable reference. A mesh GNN is autoregressively rolled out for five steps to form a motion prior. A deterministic spatiotemporal DiT regresses the corresponding displacement residuals in one forward pass per five-step window; a parameter-free topology-aware decoder coordinates neighboring corrections, and hard grasp projection enforces commanded motion. We construct action-conditioned cloth-dynamics datasets in SoftGym and Genesis and train and evaluate the model separately in each. In a nine-setting comparison on the full SoftGym TEST set under the same 15-step protocol, MeshPriorDiT outperforms all external baselines: its Global MSE is 60.56% lower than PGND-Point, the strongest external baseline, and 88.66% lower than our UniClothDiff-Dynamics reproduction. It is also 44.52% lower than the best Flow Matching-based internal alternative.

Primary area:
learning on time series and dynamical systems
Keywords:
Action-conditioned cloth dynamics, Deformable object dynamics, Mesh graph neural networks, Prior-residual modeling, Spatiotemporal Transformers

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_5X9sOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MeshPriorDiT, a method that uses a GNN to generate reference cloth dynamics, then trains a Transformer end-to-end to predict the motion residual based on these dynamics, which are integrated to produce complete, accurate cloth dynamics.

Critical strengths

The improvement in the cloth dynamics simulation is quite obvious in the proposed experimental scenarios.

Critical weaknesses
  1. The paper seems to lack generalization ability; the proposed scenarios are restricted to three types of motion, a specific cloth material, and a specific cloth shape. This limits the contribution to a narrow case compared with the baselines.
  2. The ablations comparing DiT-only, Prior-conditioned DiT, and Prior-residual DiT use flow matching, whereas the proposed method uses direct regression, a different training protocol. To isolate the contribution of the prior–residual formulation, the authors should conduct ablation studies in the direct-regression protocol that predict full displacements.
  3. The evaluation is limited to 15-step autoregressive rollouts. It remains unclear whether the reported accuracy gains persist over longer horizons and whether accumulated prediction errors remain controlled.
Questions for authors

I wonder whether it is possible to demonstrate generalization across different materials and shapes. This will improve the proposed method's effectiveness.

Reviewer_eJo8OpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper presents MeshPriorDiT, an action-conditioned cloth dynamics model that decomposes future motion into a structured mesh prior and a learned residual correction. A mesh GNN first autoregressively predicts a grasp-constrained five-step future motion over a fixed mesh topology. Using the predicted motion as a mesh prior, a deterministic DiT model regresses residual corrections conditioned on the motion history, planned actions, and mesh prior. The predicted residuals are further refined over neighboring mesh vertices, while grasped vertices are constrained to follow the commanded motion. MeshPriorDiT is evaluated on two cloth-manipulation datasets, SoftGym and Genesis. On SoftGym, the proposed method achieves lower prediction error than the evaluated baselines, and the authors provide comparisons with internal alternatives to validate the main component design choices.

Critical strengths
  1. The staged, divide-and-conquer formulation is well motivated. Rather than directly predicting future mesh motion with a single model, the method first estimates an initial motion trajectory as a mesh prior, then predicts residual corrections, and finally refines these corrections using the mesh topology. This decomposition provides a clear way to separate the roles of structured local motion modeling and global correction.
  2. The visualizations and component-wise analyses help clarify the proposed design. In particular, the mesh-motion visualization in Figure 4 effectively shows how the residual prediction refines the initial mesh prior toward the ground truth. The component-wise comparisons and ablations also help explain the role of the residual predictor and the motivation behind the main design choices.
Critical weaknesses

1. Limited differentiation from related residual dynamic methods

The proposed prior–residual formulation is not sufficiently discussed in relation to closely related residual dynamics approaches. In particular, GraphGarment and PGRD also combine a structured dynamics predictor with a learned residual correction model for deformable-object or garment dynamics. MeshPriorDiT differs in using a grasp-constrained mesh GNN rollout as an explicit multi-step motion prior and a spatiotemporal DiT for residual prediction, but these differences are not discussed in sufficient detail. A clearer comparison with these closely related methods would help clarify what is technically new beyond the general idea of correcting a structured dynamics prediction with a learned residual model.

2. Limited justification for the residual predictor architecture

The choice of a DiT architecture for residual prediction is not sufficiently justified. The proposed model does not use a diffusion training objective or iterative denoising, but instead performs deterministic residual regression with a fixed zero timestep. Although the paper compares direct regression with Flow Matching, this comparison does not isolate whether the DiT-specific architecture itself is important. It would be useful to report results after removing DiT-specific components, such as the timestep embedding and adaptive conditioning, while keeping the same spatiotemporal regression setup. A GNN-based residual predictor would also provide a useful comparison for determining whether global Transformer-based modeling is necessary for residual correction. These experiments would provide stronger evidence for the residual predictor model selection and the corresponding design choices.

3. Limited evaluation under diverse input conditions and unseen scenarios

The current evaluation is mainly conducted on fixed cloth meshes, a small set of folding and lifting actions, and constrained grasp settings. It is therefore unclear how the proposed prior–residual formulation behaves under more diverse input conditions. If feasible, evaluation on unseen mesh resolutions or topologies, unseen grasp locations or configurations, and moderately crumpled initial cloth states would help assess the robustness of both the mesh prior and residual correction. This would also be useful for understanding whether the advantage of deterministic regression over the generative alternatives remains consistent when the input conditions become more diverse.

Questions for authors
  • Can the authors report performance for longer autoregressive horizons beyond H15? Since one motivation of the proposed prior–residual formulation is to reduce error accumulation during rollout, it would be useful to see whether the performance advantage is maintained as the prediction horizon increases.
Reviewer_tj5ZOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MeshPriorDiT, a cloth dynamics model that combines GNN-based mesh motion priors with deterministic DiT residual prediction. The method uses a GNN to generate motion priors for the next five steps, then employs a DiT to predict residuals, followed by topology smoothing and grasp constraints to obtain the final predictions. Experiments on SoftGym and Genesis show that the proposed method outperforms the compared baselines in terms of 15-step prediction accuracy and inference efficiency.

Critical strengths

1.Valuable method design. The approach combines structured motion priors generated by a GNN with Transformer-based global residual prediction, integrating local mesh structure with global motion modeling.

2.Strong experimental results. On 3,106 SoftGym test trajectories, the method reduces Global MSE by 60.56% compared to the strongest external baseline, PGND-Point, and by 78.23% compared to GNN-Only.

3.Inference efficiency advantages. Deterministic residual regression avoids the multi-step sampling required by diffusion or Flow Matching methods, achieving lower prediction error and inference latency under the evaluated settings.

Critical weaknesses
  1. Lack of critical controlled ablation experiments. Both DiT-Only and Prior-conditioned DiT use Flow Matching, whereas MeshPriorDiT employs direct MSE regression. Without comparisons under the same regression objective and network configuration, it is difficult to determine whether the performance improvements stem from the Mesh Prior–Residual design or differences in training objectives.
  2. Inconsistent grasp constraints across external baselines. MeshPriorDiT applies hard grasp projection, while UniClothDiff-Dynamics only uses a grasp mask without enforcing the same output constraints. Since grasp constraints directly affect cloth motion prediction, this discrepancy weakens the validity of the main performance comparisons.
  3. Insufficient validation of the claimed physical consistency advantages. The proposed topology decoder essentially performs a single neighborhood smoothing operation, without explicitly enforcing edge-length constraints or preventing self-collisions. On Genesis, although Global MSE decreases by 82.13%, Edge-strain MSE improves by only 2.84%. These results are insufficient to demonstrate that the method substantially improves the physical structural consistency of cloth.
Questions for authors

See Critical Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.