SimGarment: End-to-End Learning of Garment Sewing Patterns via Differentiable Simulation

We present SimGarment, the first method to integrate differentiable physics simulation directly into the training of a feed-forward garment reconstruction network. From a single-view image, SimGarment predicts 2D sewing panels and stitching using a Transformer model, assembles them into a garment mesh, and drapes it onto a parametric body via a differentiable simulator. This end-to-end differentiable pipeline allows gradients from 3D Chamfer and 2D silhouette losses to flow through the simulation stage, removing the requirement for ground-truth sewing panel annotations on all training data. This enables training on a mixture of synthetic data, real-world 3D scans, and in-the-wild RGB images, achieving strong generalization across garment categories unseen in synthetic training data. In zero-shot inference, outperforms all feed-forward baselines by a wide margin and matches the quality of optimization-based methods, while being nearly five orders of magnitude faster, running in under 0.1 seconds per image. Code will be released.