acceptodds
Under review as a conference paper at ICLR 2027

SimGarment: End-to-End Learning of Garment Sewing Patterns via Differentiable Simulation

Abstract

We present SimGarment, the first method to integrate differentiable physics simulation directly into the training of a feed-forward garment reconstruction network. From a single-view image, SimGarment predicts 2D sewing panels and stitching using a Transformer model, assembles them into a garment mesh, and drapes it onto a parametric body via a differentiable simulator. This end-to-end differentiable pipeline allows gradients from 3D Chamfer and 2D silhouette losses to flow through the simulation stage, removing the requirement for ground-truth sewing panel annotations on all training data. This enables training on a mixture of synthetic data, real-world 3D scans, and in-the-wild RGB images, achieving strong generalization across garment categories unseen in synthetic training data. In zero-shot inference, outperforms all feed-forward baselines by a wide margin and matches the quality of optimization-based methods, while being nearly five orders of magnitude faster, running in under 0.1 seconds per image. Code will be released.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Differentiable Simulation, Sewing Pattern Reconstruction

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_bru5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents SimGarment, a feed-forward framework for reconstructing garment sewing patterns from a single image. The method integrates differentiable cloth simulation into network training, enabling gradients from 3D Chamfer and 2D silhouette losses to supervise sewing pattern prediction. By combining synthetic data with annotated sewing patterns, real-world 3D scans, and in-the-wild RGB images, SimGarment reduces its reliance on explicit pattern annotations and achieves promising reconstruction results on the CloSe and 4DDress datasets.

Critical strengths

Integration of Differentiable Simulation. Unlike SewFormer, which relies on ground-truth sewing patterns, or Dress-1-to-3, which requires test-time optimization, SimGarment uses differentiable simulation to propagate geometric errors to the pattern prediction network. Its use of harmonic coordinates enables gradients to flow from simulated garments to panel contours, providing an effective technical solution. 

Flexible Training with Heterogeneous Supervision. By combining synthetic sewing pattern annotations, real-world 3D scans, and 2D silhouette supervision, SimGarment reduces reliance on synthetic training data and improves its applicability to real-world garments.

Critical weaknesses

Insufficient ablation of differentiable simulation. The paper's main contribution is integrating differentiable simulation into network training, but the ablations mainly focus on data sources and regularization terms. A comparison with a baseline trained on the same data without differentiable physics supervision is needed to isolate its contribution. Moreover, it remains unclear whether simulation gradients can optimize discrete stitching topology or only continuous panel geometry.

Lack of quantitative evaluation of sewing patterns. The evaluation primarily relies on the Chamfer Distance of reconstructed 3D garments, which does not necessarily reflect the correctness of the predicted 2D sewing patterns. Additional metrics for panel geometry accuracy, stitching correctness, and sewing validity would better support the paper's claims.

The reported inference speedup is unclear. The reported inference time of less than 0.1 seconds appears to cover only sewing pattern prediction, excluding the subsequent sewing and draping simulation. The authors should clarify the end-to-end runtime and ensure that the speed comparison with Dress-1-to-3 includes equivalent processing stages.

Missing Collision Evaluation. Although the method incorporates body-contact constraints through differentiable simulation, the paper lacks quantitative evaluations of garment-body penetration or collision. Such evaluations would provide stronger evidence for the physical plausibility of the reconstructed garments.

Questions for authors

Unclear training and evaluation data splits. Section 6.1 states that CloSe and 4DDress scans are used for training, while Section 6.4 evaluates on the same datasets. Could the authors clarify the train/test splits, particularly whether test garments, subjects, and adjacent scan frames are strictly disjoint from the training data?

Sewing pattern accuracy and validity. Could the authors provide quantitative evaluations of the predicted sewing patterns, including panel geometry accuracy, stitching correctness, and sewing validity, beyond the Chamfer Distance of reconstructed 3D garments?

End-to-end inference efficiency. Could the authors report the complete pipeline runtime and clarify whether the comparison with Dress-1-to-3 covers equivalent processing stages?

Physical plausibility of reconstructed garments. Some results in Figure 1 appear to exhibit garment-body interpenetration. Could the authors provide quantitative collision or penetration metrics to further validate the physical plausibility of the reconstructed garments?

5 comments
AuthorsOpenReview ↗

Response to Reviewer bru5 (1/2)

Q1. Training and evaluation data splits. For CloSe, we use 1,416 training scans and 27 evaluation scans. Subjects are shared between the splits, with training and evaluation scans captured in different poses. For 4D-DRESS, the 3D-supervision split uses 27 training subjects and five evaluation subjects with distinct subject IDs and outfits. Evaluation uses one fixed frame per subject. To additionally assess generalization to unseen subjects, we evaluate on 114 THuman2.0 scans without using THuman2.0 for training. SimGarment achieves a Chamfer distance of 1.660 cm, compared with 1.763 cm for released SewFormer. We will explicitly document the split units and counts in the revision.

Q2. Sewing pattern accuracy and validity. We directly evaluate predicted sewing patterns on 186 SewFactory designs excluded from fine-tuning, with two poses per design and identical original renders for all models:

\text{Model} & \text{Panel-count acc.}\uparrow & \text{Edge-count acc.}\uparrow & \text{Panel-shape L2 (cm)}\downarrow & \text{Stitch F1}\ \uparrow \\\\ \hline \text{Released SewFormer} & 91.67\\% & 97.94\\% & 3.733 & 96.12\\% \\\\ \text{Synthetic-only fine-tuned SewFormer} & 95.97\\% & 99.59\\% & 2.268 & 98.96\\% \\\\ \text{SimGarment} & 98.39\\% & 99.61\\% & 2.188 & 99.34\\% \end{array}$$ SimGarment improves panel geometry and stitching correctness over both baselines. For practical sewing feasibility, all three models produce sewn and draped outputs for all 32 reconstruction cases under the same evaluation protocol. SimGarment also achieves lower penetrating garment area: 0.099%, compared with 0.139% for released SewFormer and 0.449% for synthetic-only fine-tuning. We will include these results in the revision.
AuthorsOpenReview ↗

Response to Reviewer bru5 (2/2)

Q3. End-to-end inference efficiency. Our reported runtime of under 0.1 s refers to sewing-pattern prediction without per-image optimization. We additionally report the runtime through sewing and draping, with an SMPL body fit provided:

\text{Stage} & \text{Time (s)} \\\\ \hline \text{Sewing-pattern prediction} & 0.027 \\\\ \text{Pattern/stitch conversion} & 0.023 \\\\ \text{Meshing, body initialization, and solver setup} & 1.243 \\\\ \text{Sewing and draping} & 5.013 \\\\ \text{Output-coordinate transforms} & 0.003 \\\\ \hline \text{Total with body fit provided} & 6.339 \end{array}$$ All runtimes are tested on a single A100 GPU; body estimation is excluded. Even including sewing and draping, SimGarment takes approximately 6.34 s, compared with the approximately two hours of per-image garment optimization reported by Dress-1-to-3. We will separately report sewing-pattern prediction and garment-reconstruction runtimes, and explicitly state the hardware and included processing stages in the revision. **Q4. Physical plausibility of reconstructed garments.** We quantify garment–body interpenetration as the percentage of garment surface area located more than 1 mm inside the body, averaged over evaluation scans. Using identical body fits and evaluation protocols for all three models, we obtain: $$\begin{array}{lrr} \text{Model} & \text{CloSe + 4D-DRESS (32 scans)}\downarrow & \text{THuman2.0 (114 scans)}\downarrow \\\\ \hline \text{Released SewFormer} & 0.139\\% & 0.500\\% \\\\ \text{Synthetic-only fine-tuned SewFormer} & 0.449\\% & 0.469\\% \\\\ \text{SimGarment} & 0.099\\% & 0.410\\% \end{array}$$ SimGarment achieves the lowest penetration on both benchmarks. We acknowledge that small residual interpenetration remains, as in some Fig. 1 results. We will include these quantitative metrics in the revision.
Reviewer_bru5OpenReview ↗

Response to Authors

The new experiments show that synthetic-only fine-tuned SewFormer already achieves strong sewing-pattern accuracy, with relatively modest further improvements from SimGarment. Therefore, I remain concerned that the specific contribution of differentiable simulation supervision has not been sufficiently demonstrated.

Reviewer_wfVqOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes SimGarment, a feed-forward sewing-pattern predictor fine-tuned from SewFormer with a differentiable VBD cloth simulator in the training loop. Harmonic coordinates make panel meshing differentiable. Chamfer losses on real scans (CloSe, 4D-Dress) and silhouette losses on images (DeepFashion) used to supervise patterns without pattern labels. Synthetic pattern losses, a stretch regularizer and stitch distillation from a SewFormer teacher are also added. Validation uses Chamfer distance on CloSe and 4D-Dress against feed-forward and optimization baselines, plus regularizer and data-source ablations. The paper claims parity with Dress-1-to-3 at under 0.1 s per image.

Critical strengths
  1. The idea is sound and useful. Putting a differentiable drape inside the training loop of a feed-forward pattern predictor lets unannotated scans and images supervise sewing patterns, while inference stays a single forward pass. The implementation choices are well argued:

    • It differentiates directly through a fixed-iteration VBD solver instead of using an adjoint, which is correct because the forward solve is not run to convergence.
    • It uses harmonic coordinates to carry gradients through meshing.
  2. The data-source ablation (Supp. Table 1) supports the premise. Each real source helps: CD goes from 1.824 (synthetic only) to 1.589 with all sources.

Critical weaknesses
  1. Possible train/test overlap, and the "zero-shot" claim is not justified.

    • Overlap. The real 3D supervision comes from CloSe and 4D-Dress scans (Sec. 6.1), and Table 1 evaluates on CloSe and 4D-Dress. Neither the paper nor the supplement says how these datasets were split: by subject, by outfit, or by frame.
    • Near-duplicates. 4D-Dress consists of sequences, so different frames of the same outfit are near-duplicates.
    • Evidence of in-domain advantage. In Supp. Table 1, removing 4D-Dress or CloSe from training hurts the combined benchmark, which is what in-domain training would produce.
    • Unequal comparison. BCNet, ClothWild and SewFormer never saw these datasets, so calling SimGarment "zero-shot" and comparing on equal terms is not appropriate.
  2. Most of the improvement over SewFormer is not explained by physics supervision, and pattern quality itself is never measured.

    • Unexplained gap. SewFormer scores 2.233 in Table 1. The "Synthetic only" model, which has the same supervision, scores 1.824 in Supp. Table 1. That 0.41 gap is larger than the whole gain from adding all real data (0.235). Something else differs between the SewFormer row and the proposed pipeline, such as the draping, the ETCH or GVHMR body, or the alignment. SewFormer must be run through the exact same simulator and body pipeline.
    • Inconsistent subsets. From Table 1, every row's Overall value implies about 84% CloSe weighting, except GarmentRecovery's, which implies about 74%. So GarmentRecovery is evaluated on a different subset.
    • Different protocol for Dress-1-to-3. It is scored on author-provided meshes after an alignment step.
    • No pattern metric. Only the draped Chamfer distance is reported. Mask and Chamfer losses are ill-posed with respect to the pattern: different panels can drape to the same shape. Without panel or stitch accuracy on a held-out SewFactory split, nothing shows that the patterns remain valid sewing patterns.

    Other issues:

    • Eq. (1) typo. The inertial term is written as ‖x⁻ − y‖²_M, which does not depend on x; it should be ‖x − y‖²_M.
    • Runtime claim. "<0.1 s" excludes the 110-frame simulation and the SMPL estimation needed to produce the garment that is actually evaluated.
Questions for authors

Answers to the following question could change my assessment.

  1. What are the exact train/validation/test splits of CloSe and 4D-Dress (subjects, outfits, frames)? Do any Table 1 test items share a subject or outfit with training? How many test items are there per dataset, and for the Dress-1-to-3 row?
  2. Why does SewFormer score 2.233 while your synthetic-only model scores 1.824? Are the draping, body estimate and alignment identical between the two?
  3. Why does GarmentRecovery's Overall value imply a different CloSe/4D-Dress mix from every other row?
  4. Supp. 1.2 clips simulation gradients at 10⁻⁶. Is this per element? If so, aren't the gradients effectively sign-only?
  5. Fig. 5 shows a dress-vs-pants error being fixed. How are discrete decisions, such as panel existence or garment category, learned when the loss gradients only flow through continuous contours?
  6. Evaluate on a scan dataset that is not used in training, and report panel and stitch accuracy on the SewFactory test set before and after fine-tuning.
5 comments
AuthorsOpenReview ↗

Response to Reviewer wfVq (1/3)

Q1. We thank the reviewer for requesting clarification on the dataset splits and evaluation protocol. Following Dress-1-to-3, we evaluate on CloSe and 4D-DRESS using benchmark data provided by the authors. For CloSe, we use 1,416 training scans and 27 evaluation scans, with shared subjects captured in different poses across the two splits. For 4D-DRESS, we split Inner-outfit sequences by subject for 3D supervision, using 27 training subjects and five evaluation subjects with disjoint subject IDs and outfits. Evaluation uses one fixed frame per subject. Table 1 therefore contains 32 evaluation samples: 27 CloSe and five 4D-DRESS, including all 32 for Dress-1-to-3. Larger comparisons are prohibitive because optimization-based baselines require hours per sample and Dress-1-to-3 has no public implementation. We will explicitly document the splits and sample counts in the revision.

To further assess generalization to unseen subjects, we evaluate on 114 THuman2.0 scans without using THuman2.0 for training. SimGarment achieves a Chamfer distance of 1.660 cm, compared with 1.763 cm for released SewFormer, demonstrating improved reconstruction on this external dataset.

Q2. We thank the reviewer for highlighting this gap. Our synthetic-only baseline starts from released SewFormer and receives additional fine-tuning on SewFactory, using the original synthetic renders before SewFormer’s image-translation processing. Controlled comparisons with identical evaluation images, body estimates, draping, alignment, and scoring show that both additional fine-tuning and the training-image change contribute to the improvement. Cleaner garment cues and fewer image-translation artifacts are a plausible explanation for the latter benefit.

Crucially, SimGarment still outperforms synthetic-only fine-tuning. Both start from the same released checkpoint and use the same SewFactory synthetic data and original renders; SimGarment additionally learns from real data through differentiable physical simulation. Supplementary Table 1 reports a reduction in CD from 1.824 to 1.589. These results support the additional benefit of our physics-supervised training beyond synthetic fine-tuning alone. We will clarify these distinctions and evaluation settings in the revision.

AuthorsOpenReview ↗

Response to Reviewer wfVq (2/3)

Q3. We used the GarmentRecovery reconstructions supplied by the Dress-1-to-3 authors. Within our benchmark, these provided results cover 14 CloSe and five 4D-DRESS samples. All other baselines in Table 1—BCNet, ClothWild, SewFormer, and Dress-1-to-3—were evaluated on the full 32 samples (27 CloSe and five 4D-DRESS), as was SimGarment. We have now extended the GarmentRecovery evaluation to this same 32-sample benchmark. In this setting, GarmentRecovery achieves an overall CD ×100 of 2.246, compared with 1.589 for SimGarment. Thus our conclusion does not change. We will update the comparison and explicitly document the baseline provenance and per-dataset sample counts.

Q4. The threshold is applied to the L2 norm of each particle’s three-dimensional position or velocity gradient, rather than to individual coordinates: g~i=gimax⁡(1,∥gi∥2/10−6).\widetilde{g}_i = \frac{g_i}{\max\left(1, \lVert g_i \rVert_2 / 10^{-6}\right)}. Vectors below the threshold remain unchanged; larger vectors are uniformly rescaled. This preserves the direction and relative coordinate magnitudes within each vector, so the gradients are not sign-only. We will clarify this clipping operation in Supp. Sec. 1.2.

AuthorsOpenReview ↗

Response to Reviewer wfVq (3/3)

Q5. This is an interesting question. The model predicts panel shapes and stitching from shared image features. Simulation gradients update these features through the continuous panel shapes. Because the panel and stitching predictions depend on the same features, they can improve indirectly, even though discrete operations such as stitch pairing are not differentiated. Fig. 5 illustrates the resulting improvement in the trained model’s predictions. We will clarify this mechanism in the revision.

Q6. We evaluate on 114 THuman2.0 scans, a dataset not used for training or fine-tuning. All methods use identical input images and provided SMPL fits:

\text{Model} & \text{CD (cm)}\downarrow & \text{Penetrating area}\downarrow \\\\ \hline \text{Released SewFormer} & 1.763 & 0.500\\% \\\\ \text{Synthetic-only fine-tuned SewFormer} & 1.789 & 0.469\\% \\\\ \text{SimGarment} & 1.660 & 0.410\\% \end{array}$$ For sewing-pattern evaluation, we use 186 SewFactory designs excluded from our fine-tuning, with two poses per design and identical original renders for all models. Using SewFormer’s evaluation metrics, we obtain: $$\begin{array}{lrrrr} \text{Model} & \text{Panel-count acc.}\uparrow & \text{Edge-count acc.}\uparrow & \text{Panel-shape L2 (cm)}\downarrow & \text{Stitch F1}\ \uparrow \\\\ \hline \text{Released SewFormer} & 91.67\\% & 97.94\\% & 3.733 & 96.12\\% \\\\ \text{Synthetic-only fine-tuned SewFormer} & 95.97\\% & 99.59\\% & 2.268 & 98.96\\% \\\\ \text{SimGarment} & 98.39\\% & 99.61\\% & 2.188 & 99.34\\% \end{array}$$ SimGarment improves reconstruction on unseen real scans and panel/stitch prediction on the held-out SewFactory designs, outperforming both released SewFormer and synthetic-only fine-tuning in these comparisons. We will include these results in the revision. **Typo in Eq. (1).** We will correct the typo in Eq. (1): the inertial term should be $\frac{1}{2h^2}\lVert x-y\rVert_M^2$, where $x$ is the optimization variable.
Reviewer_un39OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SimGarment trains a feed-forward sewing-pattern predictor through differentiable sewing and draping. Synthetic patterns provide direct supervision, while real scans and images supervise the simulated garments through geometry and silhouette losses. Harmonic coordinates connect predicted panel boundaries to the simulation mesh, and stretch and teacher-derived stitching regularization stabilize training. The method aims to approach the reconstruction quality of instance-specific optimization while retaining fast pattern prediction.

Critical strengths

The strongest contribution is connecting physics-based supervision to a reusable pattern predictor. This could reduce dependence on expensive real pattern annotations and avoid fitting each new image through iterative optimization. Although the individual ingredients are established, integrating prediction, meshing, sewing, and draping into a training pipeline is technically meaningful. The contour-to-mesh mapping addresses a concrete obstacle to propagating simulation losses back to panel geometry.

The ablations provide some support for the implementation choices. Removing stitching regularization increases Chamfer distance from 1.589 to 2.035, while removing stretch regularization increases it to 2.076. These results suggest that preserving structural priors and controlling deformation are important for making the training procedure effective.

Critical weaknesses

My main concern is that the visual reconstructions remain too coarse to substantiate the practical impact claimed for the method. This goes beyond missing colors or textures. In supplementary case 0010, the input is a hoodie, but the reconstruction resembles a generic open-neck top, and the predicted panels contain no evident hood component. In case 0003, the input’s long, flared trousers are reconstructed as substantially narrower and shorter trousers. These examples suggest that recovering the correct broad category does not necessarily preserve the garment’s defining shape or construction.

Questions for authors

/

1 comment
AuthorsOpenReview ↗

Response to Reviewer un39

We thank the reviewer for highlighting these examples. Our main contribution is substantially faster sewing-pattern prediction without per-image optimization. Although some details remain imperfect, such as the trouser length and flare in case 0003, SimGarment achieves better overall reconstruction accuracy than existing feed-forward baselines and comparable accuracy to optimization-based methods, as demonstrated by our quantitative and qualitative comparisons. Hood recovery in case 0010 remains challenging because our training data lack hood examples, and the frontal input provides limited information about the hood’s full 3D structure.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.