acceptodds
Under review as a conference paper at ICLR 2027

OneDemo: Unified Policy Learning and Data Generation from a Single Demonstration

Abstract

Learning robot policies from demonstrations is often limited by the cost of collecting diverse, high-quality expert data. While an appealing alternative is to expand a single successful demonstration into a diverse set of successful trajectories, existing augmentation methods typically rely on manually designed transformation rules or complex trajectory planning, requiring substantial task-specific engineering and human effort. We introduce OneDemo, a reinforcement learning framework that learns to edit a single successful demonstration for new task configurations. We formulate demonstration editing as a structured reinforcement learning problem, in which an editor learns, from sparse task-success rewards, both when to transition between object-centric trajectory segments and how to geometrically adapt each segment. The learned editor serves both as a task policy and as a reusable demonstration generator. Across 15 Meta-World tasks, OneDemo reaches a 92.7% success rate within 53.7k environment interactions on average. Across 9 DexMimicGen tasks, it generates successful demonstrations with a 98.4% average success rate, and BC-RNN policies trained on the generated data achieve 99.6% average success at matched dataset scale. These results show that learning to edit a single successful demonstration provides a structured and effective basis for both policy learning and scalable demonstration generation.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Data Augmentation, Reinforcement Learning, Dexterous Manipulation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_7VQmOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

OneDemo uses sparse-reward reinforcement learning to adapt a single demonstration to new object poses. The editor learns segment durations and object-relative geometric corrections within a fixed sequence of semantic stages. It serves as both a task policy and a demonstration generator for downstream imitation learning, evaluated in simulation.

Critical strengths
  • Restricting exploration to timing and geometric edits provides a clear, practical way to reuse demonstrated motion for policy learning and data generation.
  • Generated data substantially improves downstream policies across nine DexMimicGen tasks at matched dataset size, with multiple training seeds. Ablations support the roles of timing and geometric adaptation, and RoboTwin adds complementary results.
Critical weaknesses
  • Applicability is narrowly constrained by the source strategy. Fixed stage order, arm assignments, and hand commands prevent discovery of a different strategy when a new configuration makes the original one infeasible. Two RoboTwin tasks require separate demonstrations and editors for different modes. This is a major limitation for generating data across materially different initial states, despite strong results on compatible pose variations.
  • The main policy-learning and sample-efficiency evidence comes from relatively simple Meta-World tasks, limiting its relevance to complex manipulation. Harder benchmarks demonstrate data quality but lack complete editor-training and generation costs; the Threading ablation alone uses 20 million interactions. Comparison controls are also insufficient: Figure 3 uses Meta-World v3 for OneDemo and v2 for MoDem, while DexMimicGen policy baselines are taken from published results. Matched comparisons on harder tasks, including other learned editors, are needed.
Questions for authors

N/A

3 comments
AuthorsOpenReview ↗

Response to Reviewer 7VQm

W1: Applicability is constrained by the source strategy and cannot generalize to different behavior modes.

RW1: We agree with the reviewer that OneDemo preserves the high-level strategy of the source demonstration, including stage ordering, arm assignments, and hand commands.

This is an intentional design choice that separates spatial generalization from behavior-mode learning: by treating each source demonstration as specifying one feasible behavior mode, OneDemo focuses RL exploration on learning its temporal and geometric adaptation across task configurations, rather than discovering new manipulation strategies. This structured formulation makes sparse-reward learning more tractable in the single-demonstration setting for complex, long-horizon manipulation tasks.

When additional behavior modes are necessary, OneDemo can accommodate them through additional source demonstrations and independently trained editors, as demonstrated on RoboTwin in Appendix A.8. We acknowledge that automatically discovering and selecting new behavior modes remains outside the scope of the current work.

We have clarified this distinction in Section 1 (Introduction) and Section 5 (Limitations and Future Work) of the revised manuscript.

W2: Need stronger matched comparisons on complex tasks and training costs.

RW2: We address the reviewer's concerns regarding matched comparisons, training costs, and benchmark differences as follows.

1. Matched comparisons on challenging tasks. Please refer to MR3 in our Meta Review. The additional residual RL comparison demonstrates that OneDemo's structured trajectory adaptation substantially improves learning efficiency over low-level residual learning on challenging bimanual tasks, highlighting the benefit of restricting exploration to temporally and geometrically meaningful edits.

2. PPO training and data-generation costs. OneDemo incurs an upfront PPO training cost once per learned editor. After training, the editor achieves near-perfect trajectory-generation success rates, enabling successful demonstrations with very few wasted rollouts. Thus, the upfront training cost is amortized as more demonstrations are generated. We clarify this distinction in Section 4.4 and Appendix A.11.

3. Meta-World version difference. Regarding the Meta-World version difference, MoDem's released implementation uses Meta-World v2, reflecting the benchmark version available at the time of its publication. We evaluated OneDemo on the updated Meta-World v3 to reflect the current benchmark implementation, rather than restricting our experiments to an older version solely for strict numerical comparison.

4. DexMimicGen downstream policy baselines. We follow the standard practice of comparing against published benchmark results. These comparisons are not presented as independently reproduced evaluations. To supplement them, we conducted additional controlled evaluations on challenging bimanual tasks, including the DexMimicGen trajectory-generation reproduction summarized in MR1 of our Meta Review and reported in Section 4.4 and Appendix A.11 (Table 8 and Figures 7–8) of the revised manuscript.

Thank the authors for the response. Most of my concerns are addressed. I have updated my rating.

We sincerely thank the reviewer for taking the time to reconsider our submission and for their constructive feedback. We appreciate your thoughtful engagement with our rebuttal and updated experimental results.

Reviewer_Sp75OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes OneDemo, a reinforcement learning framework for adapting a single successful demonstration to different spatial task configurations. The method first uses an LLM to initialize an ordered sequence of semantic options and their associated anchor objects. At each option boundary, an RL policy predicts the duration of the corresponding source-trajectory segment and a residual SE(3) transformation. The source trajectory is then adapted through anchor-based translation alignment and the learned geometric residuals. The policy is trained using PPO with sparse task-success rewards. The resulting trajectory editor can be directly executed as a task policy or used to generate successful demonstrations for downstream imitation learning.

Critical strengths
  1. The paper is well motivated and clearly presented. The proposed option-based trajectory editing framework provides a structured and intuitive way to reduce the exploration space of reinforcement learning from a single demonstration.

  2. The experiments cover diverse manipulation tasks and demonstrate strong performance in both policy learning and demonstration generation. The ablation studies further support the effectiveness of learning temporal segmentation and geometric residual transformations.

Critical weaknesses
  1. The method relies on accurate anchor poses, which may limit its real-world applicability. Such information is not always readily available in real-world settings, and the paper does not evaluate how sensitive the method is to anchor pose estimation errors.

  2. Some key aspects of the method are insufficiently specified. Especially, it is unclear whether the executed trajectory must match the source demonstration in length and how the learned option durations are constrained by the remaining source trajectory. These details are important for understanding the flexibility and limitations of the proposed trajectory-editing mechanism.

  3. The experiments do not fully establish the advantages of learned trajectory editing over simpler alternatives. The Meta-World ablations lack a strong fixed-segmentation baseline, such as human-specified/LLM-predicted boundaries or observation-independent learned durations. It remains unclear whether the learned boundaries meaningfully adapt to execution-time observations or simply reflect a largely fixed segmentation of the source demonstration. Moreover, the comparison with DexMimicGen does not account for the substantial additional PPO training cost.

Questions for authors
  1. How sensitive is OneDemo to inaccurate anchor pose estimates? Could the authors evaluate its robustness under different levels of position and rotation noise?

  2. Does the executed trajectory have to have the same number of steps as the source demonstration? How is the categorical duration range determined, and how are duration choices constrained by the remaining source trajectory, especially for the final option?

  3. How are the Gaussian residual actions mapped to valid SE(3) transformations? Are the translation and rotation components normalized or bounded during training?

  4. Could the authors compare against stronger fixed-segmentation baselines, such as human-specified or LLM-predicted boundaries, using (1) anchor-based translation alone and (2) anchor-based translation plus learned geometric residuals? Also, how much do the learned boundaries actually vary across different task configurations?

  5. Could the authors report DexMimicGen's trajectory generation success rate (the fraction of attempted rollouts that succeed) on the same tasks, for direct comparison with OneDemo? Additionally, how do their total data-generation costs compare, including OneDemo's additional PPO training interactions?

1 comment
AuthorsOpenReview ↗

Response to Reviewer Sp75

Critical Weaknesses

W1: Reliance on accurate anchor poses may limit real-world applicability.

RW1: See RQ1.

W2: Insufficient details on executed trajectory length and option-duration constraints.

RW2: See RQ2.

W3: Insufficient evidence for learned adaptation over simpler baselines and concerns about PPO training cost.

RW3: See RQ4 and RQ5.

Questions for Authors

Q1: How sensitive is OneDemo to anchor-pose estimation errors?

RQ1: We conducted robustness evaluations under position and rotation noise, detailed in MR4 of our Meta Review.

Despite Threading's approximately 3 mm clearance, OneDemo maintains 90% success under 2 mm / 1° noise, versus 100% under clean observations. Transport maintains 97%, versus 99.7% clean. These results exceed the reported image-based BC baselines (69.3% on Threading and 83.3% on Transport), which do not require explicit anchor-pose estimation. Although not strictly matched, these comparisons suggest that moderate pose errors do not eliminate the benefits of structured adaptation.

We acknowledge the reliance on explicit object poses and identify richer perceptual inputs as future work in Section 5.

Q2: Must the executed trajectory match the source length? How are durations constrained?

RQ2: No, the executed trajectory does not need to have the same number of steps as the source demonstration. The duration policy partitions the source trajectory into segments, and additional interpolation steps are inserted between transformed segments during execution.

For source length TT and commitment interval cc, the categorical duration policy uses ⌈T/c⌉\lceil T/c \rceil bins. Invalid choices are masked to reserve at least one duration bin for each subsequent non-final option and one source step for the final option. The final option consumes all remaining source steps.

Details are added to Appendix A.5 (Option-PPO Architecture).

Q3: How are Gaussian residual actions mapped to valid SE(3) transformations? Are they bounded during training?

RQ3: The residual policy outputs a six-dimensional Gaussian action uu, containing three translation and three rotation-vector components. Each is clipped to [−1,1][-1,1] and scaled by its corresponding bound:

r=s⊙clip⁡(u,−1,1)r=s\odot\operatorname{clip}(u,-1,1)

where ss contains the per-dimension bounds.

The rotation vector is converted to a rotation matrix via the SO(3) exponential map, defining a valid SE(3) transformation with the translation vector. Clipping and scaling apply during both training and evaluation.

Details are added to Appendix A.5 (Option-PPO Architecture).

Q4: How do stronger fixed-segmentation baselines perform, and do learned boundaries vary across configurations?

RQ4: Please refer to MR2 and Section 4.4 and Appendix A.10 (Table 7, Figure 6).

Manual segmentation with anchor-based translation achieves 10% and 17% success on Threading and Transport. Even with learned phase-wise residuals, fixed segmentation achieves only 52.1% on Transport, versus 99.7% for full OneDemo. Thus, geometric corrections alone cannot compensate for fixed temporal boundaries.

On Transport, learned option durations correlate strongly with payload orientation flips (Spearman |ρ| = 0.85–1.00, all familywise-adjusted p < 0.001). The correlations across options suggest that the policy redistributes execution time between phases in response to changes in payload orientation, providing evidence that learned segmentation captures configuration-dependent structure in the manipulation task.

Q5: How do DexMimicGen's trajectory-generation success rates and total costs compare?

RQ5: Please refer to MR1 and Section 4.4 and Appendix A.11 (Table 8, Figures 7–8).

Since DexMimicGen's original generation implementation and success rates were not released, we reproduced its pipeline using the same single source demonstration. Our reproduction achieves 21% and 34% success on Threading and Transport, versus 100% and 99.7% for OneDemo. Paired evaluations further demonstrate OneDemo's broader spatial coverage.

OneDemo incurs an upfront PPO training cost once per learned editor. After training, near-perfect generation success rates minimize wasted rollouts, whereas reproduced DexMimicGen requires more attempts per successful trajectory. The training cost is amortized as more demonstrations are generated.

Total cost depends on the number of demonstrations, PPO interactions, rollout lengths, and computational overhead. We clarify this distinction in Section 4.4 and Appendix A.11.

Reviewer_S4NvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes OneDemo, a reinforcement learning framework that adapts a single successful demonstration to different task configurations. The method uses an LLM to initialize semantic stages and learns temporal segmentation and geometric residuals through PPO. The learned editor can serve as both a task policy and a demonstration generator. Experiments on Meta-World and DexMimicGen show strong performance in both policy learning and downstream imitation learning.

Critical strengths

1.The method is relatively simple. It reuses an existing successful trajectory and only learns temporal segmentation and geometric residuals, rather than exploring directly in the low-level action space.

2.The experimental results are strong. OneDemo achieves a 92.7% average success rate on 15 Meta-World tasks, while BC-RNN trained on its generated data achieves 99.6% average success across 9 DexMimicGen tasks.

3.The experiments cover different manipulation settings, including single-arm, bimanual, and dexterous tasks. The ablation studies also help show the contributions of temporal segmentation and geometric residuals.

Critical weaknesses

1.The paper mentions developing a "more structured demonstration editing method," but existing methods such as MimicGen already use trajectory segmentation and geometric transformations. In what sense is the proposed method more structured? What specific limitations of existing methods does it address?

2.The Naive baseline uses a randomly initialized duration policy. Would manually setting reasonable segment boundaries also lead to significant improvements? In addition, the paper does not compare against editing the entire trajectory with a single global residual, without segmentation. Therefore, it is unclear whether the proposed segmentation is actually necessary.

3.On Meta-World, the Naive baseline already achieves a 67.3% success rate, and learning temporal segmentation alone increases it to 89.7%. Adding geometric residuals only provides an additional 3 percentage points. Does this suggest that simple adjustments to the original trajectory are already sufficient for most test configurations? Can OneDemo still perform well when new configurations require substantial changes to the original motion?

4.The paper mainly compares against MoDem and DexMimicGen, but lacks comparisons with other relevant demonstration editing and residual RL methods, such as DemoGrasp and DexFlyWheel. This makes it difficult to assess how much OneDemo actually improves over existing methods.

Questions for authors

See weakness.

1 comment
AuthorsOpenReview ↗

Response to Reviewer S4Nv

W1: What makes OneDemo more structured than existing methods, and what limitations does it address?

RW1: We thank the reviewer for the comment. The phrase "more structured demonstration editing method" in the original context referred specifically to the preceding learning-based approaches (DemoGrasp and DexFlyWheel), rather than MimicGen. We have revised the sentence in Section 1 (Introduction) to:

Building on these recent works, we investigate jointly learning temporal segmentation and phase-specific geometric transformations for demonstration adaptation to long-horizon, bimanual, and dexterous manipulation.

OneDemo addresses two limitations of existing approaches:

  1. Limited spatial coverage of heuristic adaptation methods such as DexMimicGen, as demonstrated in MR1 of our Meta Review.
  2. Large exploration spaces of low-level RL-based adaptation methods such as DexFlyWheel, further investigated through our residual RL comparison in MR3.

OneDemo's fundamental distinction is its intentional separation of spatial generalization from behavior-mode learning. Each source demonstration specifies one feasible behavior mode, allowing RL to focus on temporal and geometric adaptation rather than discovering new strategies. This structured formulation makes sparse-reward learning more tractable for complex, long-horizon manipulation.

We clarified this design principle in Section 1 and the project's scope in Section 5. Additional behavior modes can be accommodated through separate source demonstrations and independently trained editors, as illustrated in Appendix A.8.

W2: Are learned segmentation and phase-specific residuals necessary compared with manual segmentation or a single global residual?

RW2: We thank the reviewer for the suggestion. Please refer to MR2 of our Meta Review for detailed ablations, also added to Section 4.4 and Appendix A.10 (Table 7).

Full OneDemo clearly outperforms manual-segmentation and single-global-residual variants on both tasks. On the long-horizon Transport task, jointly learning temporal segmentation and phase-specific geometric corrections improves success by over 50 percentage points relative to the single-global-residual variant.

W3: Are simple trajectory adjustments sufficient, and can OneDemo handle substantial motion changes?

RW3: The simpler Meta-World tasks often require only modest geometric corrections, which explains why temporal adaptation accounts for most of the original ablation gains. This does not hold for challenging bimanual tasks.

As detailed in MR1 and MR2 and Section 4.4 and Appendices A.10–A.11, simpler trajectory-editing variants substantially underperform full OneDemo. Furthermore, MR1 demonstrates substantially broader spatial coverage than the reproduced DexMimicGen baseline, including configurations that differ significantly from the source demonstration.

W4: How does OneDemo compare with other demonstration editing and residual RL methods?

RW4: DemoGrasp focuses on generalizing grasping strategies across object geometries, rather than multi-stage manipulation.

DexFlyWheel is more closely related but requires multiple rounds of imitation learning, residual RL, and data augmentation. OneDemo instead learns a trajectory editor directly from a single demonstration using sparse success rewards. DexFlyWheel reports 89.8% average generation success across four simple tasks (Grasp, Pour, Lift, and Handover), versus 98.4% for OneDemo across nine DexMimicGen tasks, including long-horizon, bimanual, and dexterous manipulation.

To further address the reviewer's concern, we implemented a low-level residual RL baseline adapted from ResFiT, detailed in MR3 and Appendix A.9 (Figure 5).

The baseline achieves near-perfect success on Box Cleanup within approximately 500k interactions but remains below 20% success on Threading and Transport throughout 22M interactions. In contrast, OneDemo reaches 99.6% and 72.8%, respectively, within the same budget.

These results demonstrate that structured trajectory editing substantially improves learning efficiency over low-level residual RL on challenging bimanual tasks.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Mata Review

We thank all reviewers for their constructive feedback. We conducted additional experiments on challenging bimanual tasks, incorporated into Section 4.4 and Appendices A.9–A.11 of the revised manuscript. We also clarified OneDemo's generalization scope in Sections 1 and 5 and expanded implementation details in Appendix A.5.

Given the limited rebuttal period, we selected two representative DexMimicGen tasks addressing complementary challenges:

  • Threading: Precision-sensitive bimanual needle insertion with approximately 3 mm clearance, testing geometric adaptation.
  • Transport: Long-horizon bimanual manipulation involving asymmetric object handover, testing adaptive segmentation and geometric corrections.

We will extend the comparisons to all tasks in the final manuscript.

MR1: DexMimicGen Reproduction and Spatial Coverage

To evaluate OneDemo's spatial generalization against heuristic demonstration generation, we compare success rates and spatial coverage with DexMimicGen using the same single source demonstration. We reproduced DexMimicGen from its paper description, as its original generation implementation and success rates were not released.

TaskDexMimicGen (Reproduced)OneDemo
Threading21%100%
Transport34%99.7%

Paired evaluations on 100 identical configurations demonstrate substantially broader spatial coverage. DexMimicGen succeeds primarily at needle orientations similar to the source on Threading, and only for the same payload orientation flip on Transport. OneDemo achieves near-perfect success even on substantially different configurations.

Results are added to Section 4.4 and Appendix A.11 (Table 8, Figures 7–8).

MR2: Additional Ablations on Segmentation and Residual Learning

To isolate the contributions of learned segmentation and phase-specific geometric corrections, we extend our Meta-World ablations to both DexMimicGen tasks. Manual phase durations were recovered from DexMimicGen metadata.

MethodThreadingTransport
Manual segmentation + translation-only alignment10.0%17.0%
Learned segmentation + single global geometric residual89.4%46.2%
Manual segmentation + learned phase-wise residuals—52.1%
Full OneDemo100%99.7%

These results show that both components substantially improve demonstration adaptation, particularly on long-horizon Transport.

Results are added to Section 4.4 and Appendix A.10 (Table 7, Figure 6).

MR3: Comparison with Residual Reinforcement Learning

To evaluate the learning-efficiency benefit of structured trajectory editing, we compare against residual RL adapted from ResFiT. To match the single-demonstration setting, we replace its pretrained BC policy with manual segmentation and translation-only alignment, learning residual corrections over low-level actions.

The baseline achieves near-perfect success on Box Cleanup within approximately 500k interactions, validating its implementation. However, on Threading and Transport, residual RL remains below 20% success throughout 22M interactions, while OneDemo achieves 99.6% and 72.8%, respectively, within the same budget.

These results demonstrate the learning-efficiency advantage of structured trajectory editing over low-level residual learning on challenging bimanual tasks.

Results are added to Section 4.4 and Appendix A.9 (Figure 5).

MR4: Robustness to Anchor Pose Estimation Errors

To assess robustness to pose estimation errors in OneDemo's state-based Option-PPO, we inject zero-mean Gaussian noise into object-pose observations at option boundaries without retraining.

Anchor Pose NoiseThreadingTransport
Clean (no noise)100%99.7%
2 mm / 1°90%97%
5 mm / 2°47%61%

The results show that OneDemo tolerates moderate anchor-pose errors, although larger perturbations degrade performance more substantially, especially on precision-sensitive tasks.

Hi everyone, please acknowledge the authors’ rebuttal with a brief reply, even if your assessment is unchanged (if you already did or are about to do so, please ignore this message).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.