acceptodds
Under review as a conference paper at ICLR 2027

SimWeaver: Zero-Shot RGB Sim-to-Real for Deformable Manipulation

Abstract

RGB sim-to-real for deformable manipulation has remained largely unsolved without real-world fine-tuning. We present SimWeaver, which trains zero-shot RGB VLA policies on 200 simulated demonstrations per task, reaching above 80% per-task and 91% average real-world success across 5 diverse deformable tasks including plastic-bag manipulation, without teleoperation or per-task calibration. SimWeaver combines a reliable measurement-backed simulator (SimWeaver-Sim) with an extensible asset framework supporting single-image generation(SimWeaver-Asset), a deterministic topology-aware trajectory synthesizer (SimWeaver-Syn), and a sim-to-real protocol with ISP-aware photometric augmentation (SimWeaver-Real). On silk grasping, the sim-trained policy reaches 100% under visual distribution shifts where real-data baselines drop to 9-70%, at two orders of magnitude lower per-trajectory cost. We will release SimWeaver and a representative asset subset.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Zero-Shot, Sim-to-Real, Deformable Manipulation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_di4dOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents SimWeaver, a complete sim-to-real pipeline that automatically generates manipulation demonstrations for deformable objects, renders RGB videos, and trains an end-to-end policy (pi0.5). SimWeaver consists of four core components: SimWeaver-Sim, SimWeaver-Syn, SimWeaver-Real, and SimWeaver-Asset. The resulting trained policy can be directly transferred to the real world. Both the methodology and experimental sections are fairly comprehensive.

Although the method proposed in SimWeaver-Sim is very promising, I still have concerns about the simulation baselines, data synthesis, and experimental evaluation. Taking these concerns into account, my initial recommendation is Weak Reject.

Critical strengths
  1. I consider SimWeaver-Sim to be the most important contribution. Achieving penetration-free cloth grasping is an important problem in robotics. Through the active-collision-region scheme, the authors achieve stable and replayable grasping simulation.

  2. SimWeaver-Asset is also well developed, particularly in providing a complete set of physical simulation parameters, which is largely absent in other work.

Critical weaknesses
  1. For SimWeaver-Sim, the authors should compare their approach with currently prevalent attachment-based grasping methods to demonstrate why friction-based grasping is better than attachment-based grasping. Attachment-based methods select cloth vertices based on their distance to the gripper and attach nearby vertices to it. If necessary, collisions between the gripper and the garment can be disabled to ensure simulation stability. This raises two further questions:

    a. For the tasks selected in this paper, attachment-based simulation is sufficient, and friction-based grasping simulation is not necessary. The authors could try tasks that can only be handled by friction-based grasping simulation, such as unfolding a sleeve that has already been folded, to better demonstrate the advantages of SimWeaver-Sim.

    b. The authors should discuss the relationship between simulated and real-world grasping success rates. If a grasp slips in simulation, could this be caused by limitations of the simulator itself, such as insufficient numerical solver accuracy or an inadequate friction model, even though the same grasp would succeed on the real robot? Conversely, if a grasp succeeds in simulation, does it necessarily succeed in the real world?

  2. I do not fully understand the SimWeaver-Syn component. The authors should explain it in more straightforward language and avoid presenting engineering implementations as more substantial methodological innovations than they are. Specifically, I have the following concerns:

    a. Why does TAS reduce Lift fail in the ablation study in Table 2? My understanding is that TAS selects better grasp pairs through the predefined graph G and uses F to select more feasible grasp points, for example, avoiding vertices on the underside or in folded regions. These seem to have little direct connection to Lift fail.

    b. Graph G is essentially a set of manually defined, task-specific and object-category-specific landmark pairs. Presenting G as "topological relations" seems to overstate what it represents.

  3. Regarding the sim-to-real experiments, I have the following questions:

    a. What exactly are the object randomization ranges for all tasks? A model trained on 200 demonstrations largely overfits to a particular class of manipulation trajectories, and I would like to know to what extent it can generalize to different object positions and initial states.

    b. Generalization to different positions can be achieved through large-scale synthesis of manipulation demonstrations in simulation. I do not understand why the authors use only 200 training demonstrations even in simulation.

    c. If possible, the authors could discuss how to train using a mixture of simulated and real-world data.

Questions for authors

Please refer to Critical Weaknesses. Please point out any inaccuracies in my comments. If the concerns above are clarified, I will raise my score.

5 comments

[Part 1/3: W1]

We thank the reviewer for recognizing SimWeaver-Sim and SimWeaver-Asset, and address the comments in three parts.

W1: Comparison with attachment-based grasping. We add the following discussion and experiments.

Limitations of attachment. Attachment decides whether the cloth is grasped by a distance rule rather than by contact, so it produces grasps that are not physically possible. As DexGarmentLab notes of GarmentLab's attach blocks, "even minimal contact … can establish attachment and lift the garment, which is clearly unreasonable." Prior work likewise adopts attachment or anchoring because a lasting physical grasp could not be simulated, and acknowledges that it cannot represent realistic finger–cloth contact (Matas et al., 2018; Nayak et al., 2026).

The root cause is that frictional grasping of thin cloth is hard to simulate: in Isaac Sim, a position-controlled gripper tends to penetrate PBD cloth or let it slip, and attachments to articulation links are not supported. Pipelines such as GarmentLab therefore bind the cloth vertices near the fingertip to a proxy body and filter gripper–cloth collision, so whether the cloth is grasped no longer depends on whether the fingers pinch it.

Experiment 1: grasp offset (W1(a)). To isolate the grasp model, we re-implement in our own simulator GarmentLab's attach block (a 3 cm cube) and the attachment block provided in DexGarmentLab's code (an 8 mm-radius sphere), with sizes and overlap offsets taken from their code. The cloth, physics parameters and trajectory are identical, and the real robot replays the same trajectory. Each setting is repeated 10 times.

Grasp offsetRealSimWeaver-SimAttach (GarmentLab)Attach (DexGarmentLab code)
No offset10/10✓ 10/10✓ 10/10✓ 10/10
10 mm above0/10✓ 0/10✗ 10/10✗ 10/10
30 mm outward0/10✓ 0/10✗ 10/10✗ 10/10

Entries: trials in which the cloth is lifted. ✓ / ✗: the simulated outcome matches / contradicts the real robot.

When the fingers close above the cloth or beside its corner, the real robot and SimWeaver-Sim miss, while both attachment schemes still lift the cloth even though the fingers never pinch it, which is the failure DexGarmentLab describes. Since only the grasp model differs, these false positives come from attachment itself, not from a particular simulator; shrinking the block only moves this boundary. Such imperfect grasps are the norm rather than contrived cases: on crumpled, self-occluded cloth, grasp points are never hit exactly, and policies evaluated in simulation grasp imprecisely. Attachment records them as successes, so demonstrations and evaluations include grasps that fail on the real robot. This also answers the first part of W1(a): Experiment 1 uses the silk from our silk-grasping task, so attachment is not sufficient for our tasks either.

Experiment 2: unfolding a folded sleeve (W1(a)). We also run the task the reviewer suggests: unfolding a sleeve folded onto the garment body (in simulation). All methods start from the same folded state and use the same grasp point and trajectory; each is repeated 10 times.

SimWeaver-SimAttach (GarmentLab)Attach (DexGarmentLab code)
Sleeve unfolded10/100/100/10

SimWeaver-Sim pinches only the sleeve and folds it back, leaving the body in place. Both attachment schemes also bind the body layer beneath the sleeve, because they select vertices by distance and cannot tell the layers apart; the whole garment is dragged along and the sleeve is not unfolded.

SimWeaver-Sim simulates real grasping through contact, which is exactly what attachment avoids. SimWeaver-Sim instead establishes reliable force interaction and coupling between the rigid gripper and the cloth: contact forces from the jaws enter the cloth solver correctly, and when the jaws compress the cloth, the active-collision-region scheme (Sec. 3.2) keeps contact and friction enabled while suspending the self-intersection repair that would otherwise conflict with them. The cloth is therefore held, or missed, according to how the jaws actually close on it. By coupling rigid–cloth contact reliably, SimWeaver-Sim solves at its root the problem that attachment works around.

We will add these discussions and experiments to Sec. 3 and the appendix.

[Part 2/3: W1(b), W2]

W1(b): Simulated vs. real grasp outcomes. We thank the reviewer and add the consistency between simulated and real grasps. Experiment 1 already includes the real robot; here we add the non-attachment simulators of Table 1, replaying the same bimanual garment grasp-and-lift open-loop on the real robot:

RealSimWeaver-SimIsaac Sim (PhysX)Newton VBD
Garment lifted10/10✓ 10/10✗ 0/10 (slips)✗ 0/10 (penetration / explosion)

In both comparisons, SimWeaver-Sim's grasp outcomes agree with the real robot, and policies trained only on its demonstrations reach 91.3% real-world success (Sec. 7.4).


W2: Clarity of SimWeaver-Syn. We thank the reviewer and clarify Syn below. Demonstration synthesis needs an invariant to decide where to grasp. Rigid objects provide one through object pose. Deformables have no stable pose under deformation, wrinkling and occlusion, so prior methods rely on teleoperation, learned sampling and post-hoc filtering. Our key idea is to use topology as the invariant for deformables: whether a bimanual grasp pair can complete a task is decided by how the two points are connected on the object, not by their current positions. In plain terms, Syn repeats three steps. It picks a grasp pair allowed by the topology graph GG, checks that the pair is visible and reachable in the current state (F\mathcal{F}), and executes and verifies the grasp, reselecting if it fails.


W2(a): Why TAS reduces Lift fail. We thank the reviewer for pointing this out; Table 2 mislabeled these failures. As the reviewer notes, F\mathcal{F} filters infeasible grasp points; it is retained in w/o TAS, which removes only GG. Grasping is therefore unaffected: the cloth is lifted, but the selected pair, e.g., a diagonal, cannot flatten it. We add a Config fail category (cloth lifted and moved as planned, but wrong final state), distinct from Lift fail. All w/o-TAS failures fall into it (Config fail 43.7%, Lift fail 0%), and we will revise Table 2, the main text and the appendix accordingly.


W2(b): Why GG is topological. GG encodes how landmarks are connected on the object, e.g., whether two corners are adjacent along the boundary. However the cloth is crumpled or folded, this connectivity does not change, which is what makes it a topological property. It also decides in principle whether a task can succeed. Grasping two boundary-adjacent corners flattens the cloth, while grasping a diagonal pair cannot. Geometry cannot replace this, since crumpling may bring diagonal points closer than adjacent ones. The ablation confirms it: removing GG drops the pass rate from 89.7% to 56.3%.

In practice, GG constrains boundary segments rather than fixed points: a landmark that is occluded or folded inward can be replaced by the nearest reachable point along the same boundary edge, which preserves the connectivity and hence the topological constraint; we will further clarify this in the appendix. Landmarks are labeled once on a category template and transferred automatically to every asset of that category. Edges follow a connectivity rule set by the task, e.g., boundary adjacency for flattening. Since topology is a property of the object category, defining it once per category is inherent to the problem, and the same definition holds for every instance under any deformation. We will add this definition to Sec. 4.

[Part 3/3: W3]

W3(a): Randomization ranges and generalization. The ranges are listed below (full list in Appendix A, Table 4). They are determined by the object size, the workspace of the two arms and IK solvability: the Piper arms have a limited workspace, while the garments and bags are large and already occupy most of it, so the pose offsets are kept within the region where both arms can reach the grasp points with feasible IK. Because the objects are large, these offsets still move the grasp points widely across the 200 demonstrations:

Silk graspingSilk unfoldingGarment foldingSnack packagingGarment unfolding
Pose range±10 cm, ±15°±5 cm, ±10° + fold±5 cm, ±15°±2 cm, ±5°– (hang-and-drop)
Grasp-point spread (cm)27×3542×2826×2410×1019×33

Coverage can be extended beyond this region in two simple ways: objects farther away can first be pulled back into it by a repositioning motion, or arms with a larger and more dexterous workspace can be used.

The policies also generalize beyond the training distribution. On silk grasping, trained with ±15° rotations, we test on the real robot under out-of-distribution 30° rotations, textures and lighting: the policy trained on 200 simulated demonstrations succeeds in all trials, while the policy trained on 100 real demonstrations reaches 8.7%, 13.0% and 69.6% (Sec. 7.5).


W3(b): Why 200 demonstrations. 200 demonstrations suffice because they are high-quality and sample-efficient. Under the same pose range (±5 cm, ±15°), SIM1's generated data reaches 0% real-world success on T-shirt folding at 200 samples and needs 10k samples, plus 200 teleoperated seed demonstrations, to reach 90% (their Fig. 8a); we reach 91.3% on garment folding with 200 synthesized demonstrations and no teleoperation. Our scaling curve also saturates at this budget: on silk grasping, real-world success rises from 60.9% at 50 to 87.0% at 100 and 100% at 200 demonstrations (Fig. 4a). Within our range, 200 demonstrations are thus already sufficient, and the range itself already covers the main operating region of the two arms. Objects placed farther away are hard for both arms to reach regardless of the amount of data, and can instead be pulled back into this region by a simple repositioning motion. We agree that more data can cover more positions and environment conditions; SimWeaver-Syn is built for this, generating 2,824 trajectories per day on a single 8-GPU server (Appendix I).


W3(c): Mixing simulated and real data. The two sources can be mixed directly: they share the same robot, cameras and action space, and the photometric augmentation is fitted to the real cameras (Sec. 5), so their distributions are consistent. Since simulated data provides broader coverage and stronger out-of-distribution generalization (Sec. 7.5), we suggest a sim-dominant mixture, with a small amount of real data to close the residual sim-to-real gap from effects not modeled in simulation, such as real lighting and scene details. We will add this discussion to the paper.

We will add all of the above to the revision. We hope these new experiments and clarifications address the reviewer's concerns, and we would appreciate it if the reviewer would reconsider the score in light of them. We are happy to answer any further questions.

Reviewer_Mzc7OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

SimWeaver combines deformable simulation, topology-aware demonstration generation, and camera-aware image augmentation to train RGB manipulation policies in simulation. It trains a separate policy for each of five tasks using 200 simulated demonstrations, without real-world policy fine-tuning. Across bag, garment, and silk manipulation, the paper reports 105 successes in 115 real-robot trials and includes ablations of contact handling, trajectory generation, and image augmentation.

Critical strengths
  1. The real-robot evaluation is the strongest part of the paper for me. Five tasks, 23 trials per task, and reported confidence intervals make a convincing case that the overall recipe transfers in the tested settings. This breadth of successful deployment is the main reason I lean toward acceptance.
  2. I appreciate the targeted ablations of contact handling and trajectory generation. They test concrete failure modes and help explain why the simulator produces usable demonstrations. That gives the simulation and data-generation contributions more support than the final task success rates alone would provide.
Critical weaknesses
  1. Two reporting issues make the experimental comparisons difficult to verify. I couldn't reconcile the 97.2% pass rate in Table 2 with 100 independently generated trajectories. Section 5 and Appendix A also disagree about which tasks use Sobol sampling. These may be straightforward to fix, but they affect the SIM1 comparison and the reproducibility of the training recipe.
  2. The evidence does not yet isolate the claimed advantage of the transfer design. Removing all augmentation shows that augmentation matters, but a generic augmentation control is missing. The silk sample-efficiency comparison also gives the simulated-data policy extra scene randomization. I therefore find the overall deployment result stronger than the claims about which ingredients explain it.

Missing references: Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World (IROS 2017), Sim-To-Real Transfer for Visual Reinforcement Learning of Deformable Object Manipulation for Robot-Assisted Surgery (IEEE RA-L 2023).

Questions for authors
  1. Could you give the success counts, denominators, and averaging procedure for Table 2, especially the 97.2% result? This would resolve my concern about the comparison with SIM1.
  2. Which tasks used Sobol sampling, and which used uniform sampling? Please clarify which description matches the reported experiments.
  3. Do you have a comparison with standard photometric augmentation under otherwise identical training? That result would help establish whether camera calibration adds value beyond augmentation itself.
  4. In the silk study, which scene-randomization factors differ between the simulated- and real-data policies? Is there a controlled result that separates their effect from the data source?
3 comments

We thank the reviewer for recognizing our real-robot deployment and our simulation and data-generation contributions, and for the careful reading that caught several inconsistencies, which we address below.

W1: Reporting issues in Table 2 and the sampling description.

Table 2 denominators (Q1). The 97.2% was computed over all 250 synthesis attempts on T-shirt folding (243 usable demonstrations), not over 100. For a matched comparison with SIM1's 100 attempts (24 usable, 24.0%), we will report the first 100 attempts in order in Table 2: 96/100 (96.0%), with the remaining 4 being Lift fails, and note the full 243/250. Each rate is computed once over all attempts, without averaging.


SIM1 comparison. In Table 2, SIM1's pass rate and replay were obtained with its official pipeline in its own simulator, and ours in our simulator. To separate the simulator from trajectory choice, we ran an additional cross-replay experiment: SIM1's released trajectory is replayed 100× in both simulators under the same final-state criterion.

TrajectoryOur simulatorSIM1 simulator
SIM1's released trajectory99/10024/100

With the trajectory and criterion fixed, the gap comes from the simulator. We also refined the final-state criterion, under which the SIM1 replay in Table 2 changes from 13/100 to 24/100; we will update Table 2 and Sec. 7.3 accordingly.


Sobol sampling (Q2). Appendix A is correct: only silk grasping uses Sobol sampling. Sobol covers the joint randomization space more evenly, which matters for silk grasping with its many randomization axes; for the other four tasks with only a few axes, uniform sampling already covers the space well. We will correct the statement in Sec. 5.


W2: Isolating the transfer design.

Calibrated vs. generic augmentation (Q3). To isolate the value of calibration beyond augmentation itself, we keep the training data, policy and all other settings identical on silk grasping and change only the photometric recipe, evaluating each under a camera white-balance shift comparable in magnitude to the measured cross-camera bias (10 trials each):

Photometric recipeSuccess
Ours (measurement-derived)10/10
Generic: wide, equally weighted ranges on all axes9/10
Ours without the calibration-derived hue term7/10
All ranges narrowed below the measured shift (±5% brightness/contrast, ±12.5% saturation/sharpness, ±0.0125 hue)7/10

With π0.5, a generic recipe with wide, equally weighted ranges already performs well (9/10), and we will add this to the discussion. Removing the hue term from our recipe drops success to 7/10, showing that hue is an important axis that must be covered. Narrowing all ranges below the measured shift also drops success to 7/10, showing that the coverage range matters as well.


Scene randomization in the silk study (Q4). Both policies share the same scene setup, the same position and rotation randomization (within ±15°) and the same photometric augmentation. The only difference is texture and lighting randomization, added to the sim data only, since texture and lighting are hard to vary in the real world and the real data were collected in a single scene. To separate the two factors, we trained the sim-data policy without texture and lighting randomization, i.e., in a single scene like the real data. It still reaches 100% in distribution, showing that the simulated data itself is effective, but degrades substantially under texture and lighting shifts. The robustness to texture and lighting therefore comes from the randomization, which is itself an advantage of simulation: texture and lighting variation is obtained at no cost, giving the policy stronger generalization to texture and lighting changes (100% under both shifts, Sec. 7.5). Notably, although both policies see the same ±15° rotations in training, the sim-data policy also handles 30° rotations (100% vs. 8.7%), suggesting that consistent simulated data better elicits the generalization of the pretrained VLA.


Missing references. We thank the reviewer and will cite and discuss domain randomization (Tobin et al., 2017) and sim-to-real transfer for deformable manipulation in robot-assisted surgery (Scheikl et al., 2023) in the related work.

We will add all of the above to the revision, and we are happy to discuss any further questions.

I appreciate the authors' thorough response, which has resolved my key concerns. I will keep my rating as Weak Accept.

Thank you for your response and for confirming that our rebuttal resolved your key concerns. We appreciate your careful review and will incorporate all of the corrections and new experiments into the revision.

Reviewer_zjvrOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a system SimWeaver for deformation object manipulation demonstration generation and visuomotor policy training. It combines RGBench-based simulation with tuned contact handling and topology-aware, closed-loop trajectory synthesis, along with measurement-backed material assignments and sensor-informed image augmentation. The experiment results show that the pipeline can enable sufficient policy fine-tuning using simulated trajectories, across multiple deformable categories.

Critical strengths
  • The sim-to-real experiment results are significant. Tab 3 reports results across all five tasks and the policies have high success rate using RGB observations as inputs.
  • The system addresses compressed mutilayer grasps, topology-dependent grasp selection, recovery and camera appearance differences. The huge engineering efforts make it robust and are not limited to tuning an existing solver and change contact handling.
  • The synthesis ablation is comprehensive, showing the effectiveness of topology-adjacency selection, contact-handling, randomization, etc
Critical weaknesses
  • The sim1 comparison does not isolate the argued causes of gap. In Sec 7.3 it describes identical or matched physics while Appendix H states sim2 replay uses its own simulation. It does not isolate simulation determinism as trajectory choice, contact margin and others also effect it
  • The sensor-specific augmentation is not separated from ordinary augmentation. Appendix B shows the results against removing image augmentation leading to significant difference. However it shows the importance of augmentation but does not necessarily show the additional value of fitting the ranges to measured ISP
Questions for authors

Please see above. Additionally,

  • what are the integer success counts and averaged scheme behind n=100 folding row? which comparisons use a shared simulator or method's native solver?
  • Can a small common-trajectory replay test separate simulator behavior from trajectory choices?
  • How does the measured-range photometric recipe compare with an otherwise identical generic augmentation recipe?
3 comments

We thank the reviewer for the positive assessment of our sim-to-real results and trajectory synthesis, and address the questions below.

W1: SIM1 comparison (Q1, Q2).

Counts and solvers (Q1). The 97.2% was computed over all 250 synthesis attempts on T-shirt folding (243 usable demonstrations). For a matched comparison with SIM1's 100 attempts (24 usable, 24.0%), we will report the first 100 attempts in order in Table 2: 96/100 (96.0%), with the remaining 4 being Lift fails, and note the full 243/250. Each rate is computed once over all attempts, without averaging. In Table 2, SIM1's pass rate and replay were obtained with its official pipeline in its own simulator, and ours in our simulator.


Common-trajectory replay (Q2). To separate the simulator from trajectory choice, we ran an additional cross-replay experiment: SIM1's released trajectory is replayed 100× in both simulators under the same final-state criterion.

TrajectoryOur simulatorSIM1 simulator
SIM1's released trajectory99/10024/100

With the trajectory and criterion fixed, the gap comes from the simulator. We also refined the final-state criterion, under which the SIM1 replay in Table 2 changes from 13/100 to 24/100; we will update Table 2 and Sec. 7.3 accordingly.


W2: Calibrated vs. generic augmentation (Q3). To isolate the value of calibration beyond augmentation itself, we keep the training data, policy and all other settings identical on silk grasping and change only the photometric recipe, evaluating each under a camera white-balance shift comparable in magnitude to the measured cross-camera bias (10 trials each):

Photometric recipeSuccess
Ours (measurement-derived)10/10
Generic: wide, equally weighted ranges on all axes9/10
Ours without the calibration-derived hue term7/10
All ranges narrowed below the measured shift (±5% brightness/contrast, ±12.5% saturation/sharpness, ±0.0125 hue)7/10

With π0.5, a generic recipe with wide, equally weighted ranges already performs well (9/10), and we will add this to the discussion. Removing the hue term from our recipe drops success to 7/10, showing that hue is an important axis that must be covered. Narrowing all ranges below the measured shift also drops success to 7/10, showing that the coverage range matters as well.

We will add all of the above to the revision, and we are happy to discuss any further questions.

Thank the author for the detailed rebuttal and feedback. It addressed my major concerns and I keep my recommendation as Weak acceptance

Thank you for your response and for confirming that our rebuttal addressed your major concerns. We appreciate your constructive feedback and will incorporate all of the clarifications and new experiments into the revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.