MeshPriorDiT: Hierarchical Modeling for Action-Conditioned Cloth Dynamics

Action-conditioned cloth prediction must preserve material structure while limiting error accumulation during autoregressive rollouts. Mesh graph networks encode connectivity, but local propagation can miss distant interactions; global models capture such interactions without guaranteeing locally consistent motion. We introduce MeshPriorDiT, which treats a constrained mesh rollout as an explicit, correctable reference. A mesh GNN is autoregressively rolled out for five steps to form a motion prior. A deterministic spatiotemporal DiT regresses the corresponding displacement residuals in one forward pass per five-step window; a parameter-free topology-aware decoder coordinates neighboring corrections, and hard grasp projection enforces commanded motion. We construct action-conditioned cloth-dynamics datasets in SoftGym and Genesis and train and evaluate the model separately in each. In a nine-setting comparison on the full SoftGym TEST set under the same 15-step protocol, MeshPriorDiT outperforms all external baselines: its Global MSE is 60.56% lower than PGND-Point, the strongest external baseline, and 88.66% lower than our UniClothDiff-Dynamics reproduction. It is also 44.52% lower than the best Flow Matching-based internal alternative.