acceptodds
Withdrawn from ICLR 2027

DexSeed: Scalable Physics-Grounded Demonstration Generation for Sim-to-Real Dexterous Manipulation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Learning visuomotor policies for contact-rich dexterous manipulation requires demonstrations that are diverse and physically valid, yet teleoperation is costly and embodiment-specific, while human videos are not directly executable by robot hands. Existing data-generation methods either replay kinematically transformed demonstrations, which break under physics variation, or track reference trajectories in simulation, which inherit the source timing and contact schedule and limit diversity. We introduce DexSeed, a physics-grounded pipeline that converts a handful of human RGB-D videos into large-scale dexterous manipulation datasets under joint randomization of layout, geometry, visual, and physics. Rather than treating a human demonstration as a trajectory to copy, DexSeed uses it as an object-centric, phase-aligned prior over task-relevant contact structure, combining temporal style augmentation, task-conditioned objective, and sampling-based optimization through full physical rollouts. Across four contact-rich tasks, DexSeed achieves the highest data-generation success from both teleoperated demonstrations and human videos, improves simulated downstream policy success by 33.2% on average, and achieves a 45% zero-shot real-world success rate using only 3 human videos per task. Videos are available at dexseed.github.io.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robotic data generation, Sim-to-real transfer, Dexterous manipulation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_EyteOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents a scalable physics-grounded demonstration generation method for sim-to-real dexterous manipulation. To address the reliance of existing physics-based methods on per-frame tracking, it treats human demonstrations as phase-aligned references and augments data in a normalized phase space. Temporal style augmentation varies execution timing across subtasks, while contact-mask weighting relaxes strict tracking constraints. The method achieves higher generation success and broader trajectory coverage, with strong performance in cross-embodiment data generation and zero-shot sim-to-real policy transfer.

Critical strengths
  1. The proposed temporal style augmentation introduces variation in overall duration and execution speed across task phases, enriching the generated demonstrations.
  2. The phase-aligned reference with contact-mask weighting relaxes the restrictions of fixed per-frame tracking, allowing greater flexibility in timing and intermediate motion.
  3. The method supports cross-embodiment data generation and real-world deployment without additional policy fine-tuning.
Critical weaknesses
  1. The presentation requires further refinement. In lines 226–234, the contact mask c(σ)c(σ) is introduced, but its relationship to α(σ)α(σ) is only defined in Appendix B.5. Moving this definition into the main text would make the objective easier to understand. The frequent use of compound modifiers also makes several passages unnecessarily difficult to follow. These expressions should be simplified where possible, and hyphenation should be consistent, such as the "Contact-mask–weighted reference prior" in line 226.
  2. All experiments use three source demonstrations per task. The scaling study varies the number of generated demonstrations but does not establish how generation success and downstream policy performance depend on the number or selection of source demonstrations.
  3. The "w/o task and contact objective" ablation replaces both the terminal task objective and the contact-mask-weighted prior with a tracking-style objective (Appendix F.1). This joint replacement does not isolate the contribution of either component, leaving the specific benefit of contact weighting unclear.
Questions for authors
  1. Why are three source demonstrations used per task? Could the authors justify this choice or evaluate how source count and selection affect generation success and downstream policy performance?
  2. Given the substantial differences in hand configurations, could the authors expand the experimental section with qualitative examples and discuss performance differences and failure modes across the four hands?
  3. Since LtaskL_{task} and LrefL_{ref} are distinct terms in JJ, could the authors independently ablate the terminal task objective and contact-mask weighting to clarify their individual contributions?
Reviewer_MymHOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

DexSeed proposes a pipeline for generating dexterous manipulation datasets from a small number of human RGB-D videos. Rather than tracking a source trajectory frame-by-frame through physics, the method treats each human demonstration as an object-centric, phase-aligned prior over contact structure. A contact mask derived from fingertip-to-object distance gates how strongly the reference is tracked, temporal style augmentation varies rollout horizon and phase timing, and a terminal task cost provides the primary success signal. Optimization is performed via MPPI-style sampling over full physical rollouts. Evaluated on four tasks, DexSeed reports improved data-generation success and up to 45% zero-shot real-world success from 3 human videos per task.

Critical strengths

a. The paper evaluates data-generation success, downstream policy performance, cross-embodiment transfer, sim-to-sim transfer, zero-shot sim-to-real deployment, and data scaling, all from just 3 source videos. The experimental design across randomization regimes (D0/D1/D2) is well-structured and the gains are consistent.

b. Fig. 5 shows that temporal style augmentation and the task/contact objective matter primarily for sim-to-sim transfer rather than in-distribution performance. This is a useful finding that suggests these components primarily improve robustness to distribution shift.

Critical weaknesses

a. The method is an incremental engineering combination, not an algorithmic contribution. The three modifications on top of SPIDER are each straightforward: a distance-based contact mask, standard time-warping, and a terminal cost. The paper dresses these up with elaborate notation and a "contact prior" narrative, but the underlying technical novelty is insufficient. The object-centric transform is from MimicGen, the MPPI optimizer and annealing schedule are from SPIDER, and the phase-aligned representation is standard in trajectory optimization.

b. The "scalability" claim is contradicted by heavy per-task engineering. The pipeline requires: (a) manual subtask segmentation with annotated boundaries, (b) task-specific anchor frame selection (voltmeter/basket, box/lid, nut/bolt, hammer/holder), (c) hand-designed terminal success predicates for each task, (d) task-specific asset sets and feasible workspace definitions. The paper acknowledges this as a limitation in one sentence but provides no analysis of how much expert effort is needed per task or how the pipeline would handle a genuinely novel task. This is not scalable it is a carefully engineered pipeline for four specific tabletop tasks.

c. Real-world results are modest and the sim-to-real gap is large. 45% average zero-shot success from 3 human videos sounds reasonable until you examine the per-task breakdown: 60% (Pick Volt: a simple pick-and-place), 45% (Open Lid), 40% (Set Nut), 35% (Place Hammer). For the two most interesting tasks (Set Nut and Place Hammer), success is below 40%. The paper does not compare these numbers against simply collecting more teleoperated data or training with RL, making it impossible to judge whether DexSeed is the right approach.

d. Ablations are limited and incomplete. All ablations are on a single task (Place Hammer). There is no per-task breakdown to understand whether the same components matter equally across tasks with different contact structures. The ablation definitions are also coarse: "w/o task and contact objective" collapses two distinct contributions (terminal task loss AND contact-mask weighting) into a single removal, making it impossible to disentangle their effects.

Questions for authors
  1. What is the total wall-clock time and GPU-hours to generate 3000 demonstrations for one task under D2, including rejected attempts? This is essential for assessing practical utility.

  2. Can you provide a quantitative diversity metric (e.g., coverage of contact modes, entropy of trajectory distributions, or number of distinct grasp strategies discovered) rather than relying solely on PCA visualizations?

  3. How does DexSeed compare against training a policy end-to-end with RL (e.g., PPO/GRPO with domain randomization) for the same total compute budget? This is the most important missing baseline.

  4. How much human expert time is required to set up the pipeline for a new task (subtask annotations, anchor selection, success predicate design, asset preparation)?

Reviewer_FaftOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents DexSeed, a pipeline that turns a few human RGB-D demonstrations into dexterous robot demonstrations under varied layouts, object properties, and dynamics. It uses the source motion to construct an object-centric, phase-aligned contact prior, then optimizes robot controls through physical rollouts with a task-level terminal objective. Successful rollouts are rendered and used to train visuomotor policies.

I view this as a meaningful contribution to demonstration generation. The paper identifies a real limitation of frame-aligned retargeting: faithfully reproducing a source motion is not necessarily the best way to complete the task in a changed environment. The reported gains in generation success and downstream policy performance, together with real-robot execution, make a credible case for the proposed system. My reservations concern how to interpret the claimed departure from source tracking and which information in the human demonstrations is indispensable.

Critical strengths
  1. The paper makes a useful conceptual shift from treating a human demonstration as an exact motion target to treating it as guidance for task-relevant interaction. The terminal task objective gives the physical optimizer room to find successful controls in changed environments.

  2. The method combines object-centric scene adaptation, temporal variation, and physical rollout optimization in a coherent pipeline. Its outputs are evaluated by actual task completion, rather than reference-tracking error alone.

  3. The evaluation follows the generated data through to policy learning and real-robot execution. This helps establish that the additional demonstrations have practical value beyond appearing diverse or physically plausible in simulation.

Critical weaknesses
  1. Limited novelty compared to SPIDER. DexSeed shares substantial similarities with SPIDER. Its main contributions appear to be contact-dependent reference loss weighting and temporal trajectory augmentation. These modifications are useful but seem more like extensions and combinations of existing techniques than fundamentally new methods.

  2. Task-specific objective limits generality. The task loss LtaskL_{\text{task}} and success criteria require task-specific design. Moreover, not all manipulation tasks have explicitly defined completion conditions, which limits the generality of the framework.

  3. Limited evidence of contact-strategy diversity. The paper uses PCA to demonstrate trajectory diversity. However, a broader motion distribution does not necessarily imply different contact strategies. The generated trajectories may still follow the same contact pattern as the source demonstration.

Questions for authors
  1. Baseline selection in Figure 3. Why does Figure 3 use methods such as ManipTrans, which primarily focus on motion retargeting, as baselines for demonstration augmentation? Would a comparison with SPIDER be more appropriate?

  2. Generalization across object meshes. SPIDER has demonstrated adaptation from a single demonstration to objects with different meshes. Can DexSeed conduct a similar experiment to better evaluate its advantages over existing methods?

  3. Quantifying contact-strategy diversity. Can the authors quantify distinct contact patterns generated from the same demonstration, such as differences in active fingers, contact locations, or contact sequences, rather than relying only on PCA-based motion diversity?

  4. Cost of extending to new task types. DexSeed is evaluated on four manipulation tasks. What time, human effort, and task-specific engineering would be required to extend the pipeline to a new type of task? In particular, which parts of subtask segmentation, anchor selection, terminal-loss design, and success checking would need to be redesigned?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.