acceptodds
Under review as a conference paper at ICLR 2027

DiscoDemo: Discovering Efficient and Diverse Robot Demonstrations for Imitation Learning

Abstract

Behavior cloning for robotic manipulation is highly sensitive to the coverage of its training data, yet collecting diverse successful demonstrations remains costly. We study reinforcement learning (RL) not as a deployable controller, but as a data generator for downstream visuomotor imitation learning. Our key premise is that multiple successful solution modes cover a broader range of successful trajectories than repeatedly collecting a single narrow solution. We introduce DiscoDemo, which trains a privileged latent-conditioned RL generator with a parallel reverse curriculum and a success-gated diversity objective to discover diverse solutions while preserving task success. The trained generator produces multiple successful closed-loop behaviors, which are safety-filtered and exported as a static dataset for downstream imitation learning. Across four manipulation tasks, DiscoDemo produces broader successful-state coverage and improves downstream visuomotor policies in both simulation and the real world. Averaged across tasks, downstream success increases to 62% from 40% for the strongest baseline in simulation, and to 78% from 39% on the real robot, respectively. These results suggest that success- ful solution diversity improves imitation learning for a pretrained flow-matching VLA and remains complementary to corrective data collection, highlighting RL as an effective mechanism for constructing broader imitation-learning datasets.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robotics, Manipulation, Imitation Learning, RL, Data Generation for Robot Learning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_7AyFOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces DiscoDemo, which trains an RL policy in simulation, and then uses it as a data generator to generate thousands of demonstrations for downstream imitation learning. The key idea that differentiates this from prior work is that this RL policy is trained to not only succeed at the task, but is also explicitly rewarded for the diversity of its trajectories, as long as those trajectories are successful (they do not reward diverse trajectories that are unsuccessful to avoid reward hacking). They argue that this diversity is important for creating an effective imitation learning dataset. Concretely, their method builds on prior work that trains a privileged RL policy in simulation with reverse curriculum learning using 20 human demonstrations. The policy is conditioned in a latent variable z that is sampled at the start of the episode and is held fixed throughout each episode. Once the RL policy is trained, it is used to generate thousands of demonstrations, which are then filtered for success and safety. This is then used to fine-tune the pretrained pi0.5 policy. DiscoDemo is evaluated on 4 tasks in both simulation and in the real world: pick and place banana into bowl, stacking cubes, and two FMB insertion tasks. Compared to a wide range of baselines, it achieves higher performance in simulation and substantially higher performance in the real-world. They also perform additional experiments to study diversity, coverage, generation efficiency, safety filtering, and how it works with corrective data collection.

Critical strengths

Strength 1: Well-motivated method and thorough experimental results The method is easy to understand, intuitive, and well-motivated. The paper includes extensive comparisons to baselines and interesting analyses of diversity, coverage, generation efficiency, and safety. The appendix is detailed and insightful.

Strength 2: Reasonably strong empirical results and analysis DiscoDemo demonstrates consistent improvement over baselines in simulation and in the real world. They also study how varying the diversity parameter alpha affects diversity and success rate, which is helpful.

Critical weaknesses

Weakness 1: Limited novelty in the method The method combines existing techniques, including reverse curriculum learning and latent-conditioned diversity optimization, which makes the work feel incremental. While the improvements are interesting, the additional benefit of diversity after corrective data collection is mixed. For example, DAgger nearly closes the performance gap on PnP-Banana and reduces the gap on FMB-SqCircle.

Weakness 2: Lacking more insightful experiments that would validate this specific method While the paper has extensive experimental results, it does not answer important questions about diversity and scale. In particular, all of their methods use 20 human demonstrations and 3000 generated demonstrations. It is unclear how much diversity there is in the human demonstration, how this method and others would perform with 1 or 100 human demonstrations, and how this method and others perform with 300 or 30000 generated demonstrations. This would provide more insight into how diversity interacts with scale. Additionally, it is unclear if this specific diversity approach is needed. For example, it would be helpful to compare with training multiple separately trained RL teachers pooling trajectories from all of them as another form of diversity.

Questions for authors

Question 1: How sensitive is the method to the number and diversity of human demonstrations? Does it still discover diverse solutions with 1 human demo? Does it still discover distinct solutions if all of the human demos approach the problem the same way?

Question 2: How does downstream performance scale with the number of generated demonstrations, beyond the fixed 3000 used in the paper?

Question 3: What are the dominant downstream failure modes? Does DiscoDemo demonstrate improved recovery behavior?

Question 4: Could the authors extend the DAgger/DART experiments in Figure 8 to all four tasks and include uncorrected baselines? The performance gap after correction becomes quite small on PnP-Banana, so understanding when solution diversity provides additional benefits would be valuable.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 7AyF [1/3]

Thank you for the thoughtful feedback. We extended corrective collection to all four tasks, varied the number of generated and seed demonstrations, compared pooled RL generators, and analyzed downstream failure modes and recovery.

Novelty and contribution (W1)

DiscoDemo builds on established curriculum RL and skill-discovery methods, but uses them for a different purpose: discovering multiple successful solutions to construct more effective datasets for downstream imitation learning.

Rather than optimizing task success alone, we explicitly encourage diversity among successful RL trajectories and export them as a reusable static dataset. Our controlled P-RFCL comparison isolates this contribution by holding the RL backbone, curriculum, safety criteria, dataset size, and downstream training fixed while disabling the diversity objective (Sections 3 and 4.1). We also introduce a massively parallel reverse curriculum, achieving approximately 17× faster reverse-phase training than the original RFCL in our five-demonstration PnP-Banana comparison (Appendix B.5).

Complementarity with corrective data collection (W1, Q4)

We thank the reviewer for this suggestion. We extended DAgger and DART-style correction to all four tasks and included the uncorrected baselines in Figure 11 (Section 4.6). For each method, we use its own frozen RL generator as the teacher for corrective collection (Appendix C).

The table below summarizes downstream success (%) from Figure 11; each cell lists P-RFCL / DiscoDemo.

TaskNo correction+DAgger+DART-style
PnP-Banana73 / 8597 / 9989 / 94
StackCube29 / 5748 / 8955 / 65
FMB-Round39 / 6157 / 7051 / 65
FMB-SqCircle21 / 4564 / 7437 / 78

DiscoDemo maintains higher downstream success with either correction method on all four tasks, although the improvement varies by task. We agree that DAgger nearly closes the gap on PnP-Banana, where both methods approach the performance ceiling, and reduces it on FMB-SqCircle. However, substantial gaps remain elsewhere, particularly on StackCube with DAgger (48% vs. 89%) and FMB-SqCircle with DART-style correction (37% vs. 78%).

Importantly, both correction methods also improve DiscoDemo on every task, showing that diverse successful demonstrations do not eliminate the need for corrective supervision. These results demonstrate that solution diversity and corrective collection are complementary at the pipeline level, rather than competing approaches.

AuthorsOpenReview ↗

Response to Reviewer 7AyF [2/3]

Diversity and scale (W2, Q2)

Scaling with generated demonstrations

Thank you for this suggestion; the new scaling study directly examines how diversity interacts with scale. We evaluated downstream performance using 100–30,000 demonstrations under the same 20k-step π0.5\pi_{0.5} fine-tuning budget, with three training seeds and 50 evaluation episodes per seed (Section 4.4).

The table below summarizes mean downstream success (%) from Figure 8 and Table 10; each cell lists P-RFCL / DiscoDemo.

Task300 demonstrations3,000 demonstrations30,000 demonstrations
PnP-Banana55 / 7873 / 8568 / 85
StackCube17 / 4329 / 5729 / 52
FMB-Round45 / 5339 / 6144 / 70
FMB-SqCircle14 / 4321 / 4521 / 41

DiscoDemo outperforms P-RFCL at every tested dataset size, with even 300 DiscoDemo demonstrations exceeding 30,000 P-RFCL demonstrations on all four tasks. However, DiscoDemo itself shows diminishing returns beyond 3,000 demonstrations on most tasks, with FMB-Round showing the clearest additional improvement. Thus, simply collecting more demonstrations from P-RFCL does not close the performance gap under the fixed training budget.

Diversity of human demonstrations

To examine whether the generated datasets simply reproduce the diversity present in the human demonstrations, we compared the original 20 human seeds with equally sized subsets of generated demonstrations (Figure 15; Appendix G.1).

DiscoDemo achieves higher coverage than its human seeds on all four tasks, whereas P-RFCL falls below the seeds on three. On PnP-Banana, both RL generators discover grasp configurations absent from the human demonstrations, but DiscoDemo produces the greatest variation in grasp positions and closing angles within a fixed scene (Figure 17; Appendix G.2). These results suggest that RL can discover behaviors beyond the seeds, while the diversity objective further expands the range of successful solutions.

We also tested generators trained from fewer human demonstrations and deliberately narrow motion-planning demonstrations. The detailed results are presented in our response to Q1 below (Appendix H.7).

Pooling independently trained RL generators

To test whether diversity from independent RL training runs could replace explicit diversity optimization, we trained five P-RFCL generators with different random seeds per task and pooled 600 demonstrations from each, keeping the final dataset at 3,000 demonstrations. We applied the same filtering, rendering, and downstream fine-tuning protocol (Appendix H.3).

The table below summarizes downstream success (%) from Figure 23 and task-averaged generator training costs from Appendix I.1; the pooled cost includes all five training runs but excludes rollout and rendering.

Data sourcePnP-BananaStackCubeFMB-RoundFMB-SqCircleAverageTraining GPU-h
P-RFCL73293921406.1
P-RFCL, pooled734047314830
DiscoDemo85576145628.2

Pooling improves on a single P-RFCL generator on three tasks but remains below DiscoDemo on every task, despite approximately 3.7 times the generator training cost. In a fixed scene, the pooled trajectories exhibit some variation across seeds but remain concentrated along a few narrow paths, roughly one per generator (Figure 22). Thus, independent RL training randomness provides some diversity, but does not replace explicit diversity optimization in this comparison.

AuthorsOpenReview ↗

Response to Reviewer 7AyF [3/3]

Number and diversity of seed demonstrations (Q1)

We varied the number and source of seed demonstrations on PnP-Banana, using the same 50M-step RL training budget and α=0.3\alpha=0.3 as in the main comparison (Appendix H.7). We tested 1, 5, 10, and 20 human demonstrations, as well as 20 and 100 scripted motion-planning (MP) demonstrations.

Fewer human demonstrations.

Our default curriculum retains RFCL's reverse curriculum but omits its prioritized level replay (PLR)-based forward curriculum. With only one human demonstration, the default generator fails to learn the task from the full initial-state distribution. We therefore additionally tested restoring RFCL's PLR-based forward curriculum for the 1- and 5-demonstration settings.

The table below summarizes generator success (%) from Figure 27, evaluated from the original task distribution ρ0\rho_0.

Seed demonstrationsDefault curriculumWith PLR
1 human055
5 human2968
10 human75–
20 human94–
20 scripted MP78–
100 scripted MP86–

With the default curriculum, generator success improves as more human demonstrations are provided. However, restoring PLR enables learning from even a single human demonstration, reaching 55% generator success. Despite this lower success rate, the single-demonstration generator discovers distinct successful trajectories and grasp configurations beyond its seed example (Figures 29 and 30). This suggests that a single demonstration can support solution discovery with the appropriate curriculum, although reliable task completion still benefits from more demonstrations.

Discovery from deliberately narrow seeds.

To examine whether DiscoDemo can discover solutions absent from its seeds, we replaced the human demonstrations with 20 scripted MP demonstrations that use concentrated grasp strategies. Despite these narrow seeds, DiscoDemo reaches 78% generator success, produces varied trajectories from the same initial scene, and discovers grasps outside the dominant seed grasp configuration (Figures 27, 29, and 30).

The table below summarizes the PnP-Banana diversity and coverage results from Figure 28. Fixed-scene diversity is undefined for the human seeds because they start from different scenes.

Generator or seed sourceFixed-scene diversityCoverage
DiscoDemo, 20 human seeds7.614.1
DiscoDemo, 10 human seeds6.015.1
DiscoDemo, 20 MP seeds5.015.0
P-RFCL, 20 human seeds1.78.2
20 human seeds–9.4
MP1.44.3

DiscoDemo trained from narrow MP seeds achieves substantially higher diversity than MP or P-RFCL, while maintaining coverage comparable to DiscoDemo trained from human seeds. Increasing the MP seed count to 100 also yields 86% generator success (Figure 27). These results suggest that scripted demonstrations could reduce the need for human teleoperation, although we have only evaluated this possibility on PnP-Banana. The 100-demonstration condition uses scripted MP seeds rather than human demonstrations.

Together, these experiments show that DiscoDemo can discover diverse successful solutions beyond its seed demonstrations, rather than simply reproducing their original behavioral variations.

Failure modes and recovery (Q3)

To investigate downstream failures and recovery behavior, we evaluated one π0.5\pi_{0.5} policy per dataset on the same 10 initial scenes with 10 policy-noise seeds per scene, yielding 100 rollouts per task and dataset (Appendix H.6). We identify errors directly from simulator states, including missed grasps, slips, and drops for pick-and-place tasks, and missed holes and insertion jams for insertion tasks. An episode is counted as recovered if it succeeds after its first detected error, and recovery rate is computed over episodes containing at least one detected error.

The table below summarizes recovery rates (%) from Figure 25; Figure 26 provides qualitative examples.

TaskP-RFCL recoveryDiscoDemo recovery
PnP-Banana2524
StackCube04
FMB-Round2867
FMB-SqCircle722

Missed grasps and missed holes are the most common first detected errors, while drops are also frequent for P-RFCL on StackCube. The clearest recovery improvements occur on the insertion tasks, where DiscoDemo increases recovery from 28% to 67% on FMB-Round and from 7% to 22% on FMB-SqCircle. On the pick-and-place tasks, recovery is similar or rare for both methods, and DiscoDemo's main advantage is a greater number of successful episodes without detected errors.

These results distinguish two benefits of DiscoDemo demonstrations: reducing the frequency of downstream errors and, particularly for insertion tasks, improving recovery after errors occur.

AuthorsOpenReview ↗

Follow-up Response to Reviewer 7AyF

Real-robot results over three fine-tuning seeds (W1)

We evaluated all methods on the real robot with three fine-tuning seeds (20 trials per task and seed) and now report mean ±\pm s.d. (Figure 3 and Table 8).

TaskStrongest baselineDiscoDemo
PnP-Banana36.7 ± 7.6 (P-RFCL)86.7 ± 5.8
StackCube55.0 ± 13.2 (MG)76.7 ± 2.9
FMB-Round48.3 ± 10.4 (P-RFCL)81.7 ± 7.6
FMB-SqCircle23.3 ± 7.6 (P-RFCL)61.7 ± 5.8
Average36.3 ± 2.2 (P-RFCL)76.7 ± 1.4

On all four tasks, the ±1\pm 1 s.d. range of DiscoDemo does not overlap with that of any baseline. We have updated Section 4.1, Appendix A, and the Reproducibility Statement accordingly, and removed the single real-robot seed from the limitations in Section 5.

We hope these results address your concern. If there are any remaining questions or suggestions, we would be glad to address them during the discussion period.

Reviewer_jBnROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DiscoDemo studies RL as a demonstration generator for downstream imitation learning. It combines a parallel demonstration-based reverse curriculum with latent-conditioned, success-gated diversity rewards to generate varied successful behaviors. Successful trajectories are safety-filtered and used to fine-tune a pretrained π0.5 policy. Across four manipulation tasks, the resulting policies outperform those trained on several alternative data sources in simulation and on a real robot. Additional experiments examine trajectory diversity, the diversity–performance trade-off, generation cost, and compatibility with corrective data collection.

Critical strengths
  1. The paper provides useful evidence that teacher success alone is insufficient to characterize demonstration quality. P-RFCL and DiscoDemo both achieve approximately 95% generation success before filtering, yet their datasets produce substantially different downstream performance. The comparison against P-RFCL—the same generator setup with diversity disabled—is particularly informative because it isolates the value of the added diversity objective more directly than comparisons against unrelated pipelines.

  2. The downstream improvements are substantial and consistent across the evaluated tasks. Average success increases from approximately 40% to 62% in simulation and from 39% to 78% on the real robot, using the same student architecture and equal numbers of successful demonstrations. The diversity-weight sweep further shows that more diversity is not always better, while the corrective-collection experiments suggest that successful-behavior diversity complements recovery supervision. Together, these results make the paper informative beyond a single aggregate performance comparison.

Critical weaknesses
  1. The methodological contribution is primarily an integration of existing ideas: demonstration-based reverse curricula, METRA-style skill diversity, and success-aware diversity incentives. The paper's significance therefore rests heavily on the empirical insight and its generality. Evaluation is limited to four relatively simple tabletop tasks and one downstream policy family, with only one fine-tuning seed deployed for 20 real-world trials per task. This supports effectiveness in the reported setting but leaves the breadth of the claimed benefit uncertain.

  2. The explanation in terms of broader useful coverage is plausible but not fully established. “Coverage” is measured through diversity in a selected trajectory representation, rather than coverage of states encountered by the imitation learner when it deviates. The diversity objective can also change trajectory duration, visited configurations, and which successful behaviors survive filtering. The matched P-RFCL comparison supports the usefulness of the objective as a whole, but does not establish which resulting dataset properties drive the gains. The finding that excessive diversity hurts makes this distinction particularly relevant.

Questions for authors
  1. How diverse are the 20 seed demonstrations in terms of solution strategies? Does DiscoDemo discover successful behaviors absent from those demonstrations, or primarily expand variations already present? A comparison of seed and generated behaviors—or an experiment with deliberately narrow seeds—would help establish this dependency.

  2. Would pooling demonstrations from independently trained P-RFCL generators provide similar benefits? A comparison at a fixed dataset size, with generation compute reported, would help establish the value of latent-conditioned diversity beyond variation obtainable through ordinary RL training randomness. This is a suggested additional experiment.

  3. After success and safety filtering, do the P-RFCL and DiscoDemo datasets have comparable initial-scene distributions and total numbers of action transitions? Equal episode counts may conceal differences in task-instance coverage or supervision volume. Please clarify these properties so the downstream gains can be interpreted more precisely.

4 comments
AuthorsOpenReview ↗

Response to Reviewer jBnR [1/2]

Thank you for the thoughtful feedback. We added a second learner, varied dataset size, controlled supervision volume, and examined learner deviations, seed dependence, pooled generators, and retained initial scenes.

Novelty and contribution (W1)

DiscoDemo builds on established curriculum RL and skill-discovery methods, but uses them for a different purpose: discovering multiple successful solutions to construct more effective datasets for downstream imitation learning.

Rather than optimizing task success alone, we explicitly encourage diversity among successful RL trajectories and export them as a reusable static dataset. Our controlled P-RFCL comparison isolates this contribution by holding the RL backbone, curriculum, safety criteria, dataset size, and downstream training fixed while disabling the diversity objective (Sections 3 and 4.1). We also introduce a massively parallel reverse curriculum, achieving approximately 17× faster reverse-phase training than the original RFCL in our five-demonstration PnP-Banana comparison (Appendix B.5).

Generalization to another downstream learner (W1)

We additionally evaluated Diffusion Policy trained from scratch on the same generated datasets, without regenerating demonstrations (Appendix F.1).

The table below summarizes downstream success (%) from Figure 13, using one training seed and 50 evaluation episodes per task.

TaskP-RFCL dataDiscoDemo data
PnP-Banana7070
StackCube2450
FMB-Round04
FMB-SqCircle28

DiscoDemo matches or exceeds P-RFCL on all four tasks, with the clearest improvement on StackCube. However, both datasets yield low success on insertion tasks, so broader generalization across policy architectures remains to be established.

We acknowledge the limited scope of our four tabletop tasks and have explicitly discussed this limitation in Section 5. We also plan to evaluate additional real-world fine-tuning seeds before the rebuttal period ends.

Which dataset properties account for the gains? (W2)

Supervision volume and dataset scaling

DiscoDemo trajectories are generally longer than P-RFCL trajectories, resulting in more action transitions despite having the same number of demonstrations. To control for this difference, we matched transition counts in both directions: reducing the number of DiscoDemo demonstrations to match P-RFCL's transitions and increasing P-RFCL demonstrations to match DiscoDemo's. We retain complete episodes, matching transition counts to within one episode's length (Appendix H.4; Table 6).

The table below summarizes downstream success (%) from Figure 24; each cell lists P-RFCL / DiscoDemo.

TaskAt P-RFCL's transition countAt DiscoDemo's transition count
PnP-Banana73 / 8172 / 85
StackCube29 / 5227 / 57
FMB-Round39 / 6045 / 61
FMB-SqCircle21 / 3819 / 45

DiscoDemo consistently outperforms P-RFCL at both transition budgets, indicating that its advantage cannot be explained simply by having more action transitions. DiscoDemo outperforms P-RFCL at every dataset size, with 100 DiscoDemo demonstrations matching or exceeding 30,000 P-RFCL demonstrations on most tasks (Figure 8).

Coverage of learner deviations and recovery

To examine whether generated demonstrations cover states encountered by downstream learners, we evaluated the π0.5\pi_{0.5} policies from the main experiments (three fine-tuning seeds, 50 episodes each). We represent each state using all object positions, robot joint positions, and gripper finger position, standardized with statistics pooled across both datasets. For each learner state, we measure its distance to the nearest state on its generator's trajectory from the same initial scene.

A deviation is defined as a state whose distance exceeds the 90th percentile of distances observed in successful learner episodes. We then measure the fraction of these deviation states covered by the training dataset at varying distance thresholds (Section 4.4; Appendix H.5). DiscoDemo provides equal or better coverage on three of four tasks, with P-RFCL performing better at intermediate thresholds on FMB-SqCircle (Figure 7). Because deviation thresholds are learner-specific, this analysis is diagnostic rather than causal evidence.

We further evaluated recovery using one policy per dataset on 10 shared scenes with 10 policy-noise seeds each (100 rollouts per task). We detected errors such as missed grasps, slips, drops, and insertion jams from simulator states, and counted recovery when an episode succeeded after its first detected error. Recovery improves from 28% to 67% on FMB-Round and from 7% to 22% on FMB-SqCircle with DiscoDemo data (Figure 25; Appendix H.6). On the pick-and-place tasks, the main benefit is more successful episodes without detected errors rather than improved recovery.

AuthorsOpenReview ↗

Response to Reviewer jBnR [2/2]

Diversity of the seed demonstrations and narrow seeds (Q1)

We appreciate this question, which led us to test deliberately narrow seeds directly. DiscoDemo expands the behavioral variation present in its seeds. We compare the 20 human seeds with subsets of 20 generated demonstrations, keeping set size equal across data sources (Appendix E.1). DiscoDemo has higher coverage than its seeds on every task, whereas P-RFCL falls below them on three of the four (Figure 15; Appendix G.1).

On PnP-Banana, both RL generators produce grasp labels absent from the human seeds, but DiscoDemo exhibits the greatest variation in grasp position and closing angle within a fixed scene (Figure 17; Appendix G.2). Thus, RL itself can discover behaviors beyond the seeds, and the diversity objective broadens this variation.

We also trained DiscoDemo from 20 scripted MP demonstrations with concentrated grasp strategies on PnP-Banana. The generator reaches 78% success, and most of its grasps fall outside the dominant seed label (Figure 27 and Figure 29). Its fixed-scene diversity is 5.0, compared with 1.7 for P-RFCL trained from human seeds, while its coverage is comparable to DiscoDemo trained from human seeds (Figure 28). These results show that narrow seed strategies can still support varied successful solutions (Appendix H.7).

Pooling independently trained P-RFCL generators (Q2)

We trained five P-RFCL generators per task and pooled 600 demonstrations from each, keeping the final dataset at 3,000 and applying the main filtering, rendering, and fine-tuning protocol (Appendix H.3).

The table below summarizes downstream success (%) from Figure 23, together with task-averaged generator training cost. Costs are estimated as described in Appendix I.1 and include all five runs for the pool; rollout and rendering are excluded.

Data sourcePnP-BananaStackCubeFMB-RoundFMB-SqCircleAverageTraining GPU-h
P-RFCL73293921406.1
P-RFCL, pooled734047314830
DiscoDemo85576145628.2

Pooling improves on a single P-RFCL generator on three tasks but remains below DiscoDemo on every task, despite approximately 3.7 times the generator training cost. In a fixed scene, the pooled rollouts vary across seeds but remain concentrated along a few narrow paths, roughly one per generator (Figure 22). This directly tests variation from independent RL runs at a fixed final dataset size.

Initial scenes and number of transitions (Q3)

The exported target-placement distributions broadly cover the sampled range in most settings, consistent with the generators' generally high success rates. Figure 18 compares the bowl, blue-cube, or hole positions in the exported datasets with samples from ρ0\rho_0 (Appendix G.3). The main exception is P-RFCL on FMB-Round: its generator does not fully learn the task within the training budget and fails to reliably solve board placements close to the robot. These placements are consequently underrepresented in its dataset, while DiscoDemo handles and retains them.

For supervision volume, Table 6 reports the episode and transition counts, and the bidirectional matching experiment in our first response shows that DiscoDemo's advantage persists at either transition budget (Figure 24; Appendix H.4). We therefore control supervision volume while documenting the remaining difference in retained scene coverage.

Thank you for the detailed response and the additional experiments. They addressed several of my concerns, particularly around supervision volume and pooled generators. A few points still keep me at my current score:

On FMB-Round, which shows the largest simulated gain, the P-RFCL generator did not fully learn the task and its dataset is missing near-robot board placements, so this particular comparison reflects a difference in task competence as well as diversity. I also noticed that the response mentions plans to run additional real-robot seeds, while the revised paper already reports three; it would help to clarify which is the case, since the headline numbers changed as well. Finally, the Diffusion Policy results are encouraging on StackCube, but with a single seed and near-zero success on the insertion tasks, I don’t think they yet establish that the benefit generalizes beyond π0.5.

Overall, I still view this as a solid contribution and am maintaining my score of 3.

Thank you for acknowledging our additional experiments and for your constructive follow-up feedback.

Regarding the real-robot evaluation, we would like to clarify that we expanded the evaluation to three fine-tuning seeds following reviewer 7AyF's suggestion, and the revised manuscript already reports these updated results (Figure 3 and Table 8). We apologize for the confusion caused by the outdated statement in our earlier response about planned additional seeds.

We also agree that the remaining concerns warrant further investigation. In particular, we will extend the Diffusion Policy evaluation to three seeds and conduct an additional FMB-Round comparison with a more extensively trained P-RFCL generator, aiming to achieve higher task success and better isolate the effect of solution diversity from task competence. Although the limited discussion period prevents us from completing these experiments here, we commit to incorporating both evaluations into the camera-ready version.

We sincerely appreciate your careful assessment and constructive suggestions, which have helped us strengthen the paper.

Reviewer_RTkvOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The authors propose DiscoDemo, a novel framework for generating highly diverse simulation trajectories from a small set of human teleop trajectories. They consider the setting of robot manipulation where there are a small number (~20) of real-world human teleop and an accurate simulation of the task. The method trains a privileged RL teacher policy conditioned on a latent representation _z_ which is learned to control behaviors. The expert policy then generates thousands of successful demonstrations which is used to fine-tune pi0.5.

The method is tested on a set of tasks on a real Franka FR3 with a reconstructed Isaac Sim scene. Experiments demonstrate DiscoDemo does better than prior methods like MimicGen, and also show that the diversity reward is useful. Interestingly, task success rate increases but peaks before falling as diversity increases, demonstrating that excess diversity may lead to reduced task completion. Furthermore, the method works in tandem with corrective data.

Critical strengths
  1. There's significant interest and understanding on different methods of using simulation to achieve better results for training robot foundation models like pi0.5. This work gives new insight into the set of techniques for eliciting better performance within this highly importnat field.
  2. The results are quite impressive, especially compared to strong baselines like MimicGen. In particular, the experiments support the core thesis that (some) diversity is important.
Critical weaknesses
  1. Better stating and ideally comparing against DLR directly, to most clearly highlight the novelty on top of the prior state of the art.
Questions for authors
  1. I would strongly suggest comparisons with DLR on the same task; it would be very insightful to compare as the methods are close both in motivation and mechanism.
  2. A baseline of multiple P-RFCL policies with different seeds would also be an interesting comparison to highlight the necessity of the diversity reward.
  3. If there's further analysis of the effect of diversity strength -- for example, what kind of trajectories lead to good policy behavior -- that would be highly insightful. Is zz interpretable? Can we predict α\alpha which maximizes task reward? Any insight would be appreciated.
2 comments
AuthorsOpenReview ↗

Response to Reviewer RTkv

Thank you for the thoughtful feedback. We clarified the relation to DLR, added a comparison with a DLR-style adaptation and pooled P-RFCL generators, and analyzed the skill latent zz and the choice of α\alpha.

Relation to DLR (W1, Q1)

Both methods build diverse datasets with RL, but DLR infers discrete behavioral patterns from the demonstrations before task-reward RL, whereas DiscoDemo optimizes a success-gated, latent-conditioned diversity objective during RL (Section 2 and Section 3.2). To compare them on the same tasks, we evaluated a DLR-style adaptation that conditions P-RFCL on the discrete patterns inferred by DLR's Discover stage. P-RFCL's demonstration replay and reverse curriculum replace DLR's behavior-cloning stage, and its RL updates replace PPO. This is an adaptation for a shared-backbone comparison, rather than a reproduction of the original DLR pipeline (Appendix F.2). We fine-tuned π0.5\pi_{0.5} on 3,000 demonstrations from each method using the same recipe.

The table below summarizes mean downstream success (%) from Figure 14 and Table 9.

Data sourcePnP-BananaStackCubeFMB-RoundFMB-SqCircleAverage
P-RFCL7329392140
DLR-style (P-RFCL backbone)7729432343
DiscoDemo8557614562

The DLR-style variant matches or improves on P-RFCL, but DiscoDemo remains stronger on every task. Under this shared backbone, optimizing diversity during RL provides a larger downstream benefit than the implemented seed-pattern conditioning approach.

Pooling P-RFCL generators with different seeds (Q2)

Thank you for suggesting this baseline; it directly tests whether the diversity reward is necessary. We trained five P-RFCL generators per task and pooled 600 demonstrations from each into a 3,000-demonstration dataset, using the main filtering, rendering, and fine-tuning protocol (Appendix H.3).

The table below summarizes downstream success (%) from Figure 23, together with task-averaged generator training cost. Costs are estimated as described in Appendix I.1 and include all five runs for the pool; rollout and rendering are excluded.

Data sourcePnP-BananaStackCubeFMB-RoundFMB-SqCircleAverageTraining GPU-h
P-RFCL73293921406.1
P-RFCL, pooled734047314830
DiscoDemo85576145628.2

Pooling improves on a single P-RFCL generator on three tasks but remains below DiscoDemo on every task, despite approximately 3.7 times the generator training cost. In a fixed scene, the pooled rollouts vary across seeds but remain concentrated along a few narrow paths, roughly one per generator (Figure 22). Thus, variation from independent RL runs helps, but does not replace explicit diversity optimization in this comparison.

Diversity strength, interpretability of zz, and choosing α\alpha (Q3)

Different latent skills produce distinct, structured behaviors. Even from the same initial scene, changing zz leads to different trajectories, including variations in approach paths and grasp positions (Figure 21). Moreover, trajectories become more different as the distance between their latent skills increases, consistently across all four tasks (Mantel permutation test, p≤0.003p \le 0.003; Figure 20). Individual latent dimensions also correlate with specific behaviors, such as grasp position on StackCube (Spearman ρ=0.92\rho=0.92), although their interpretation is task-dependent (Appendix H.2).

Generator success provides a practical guide for choosing α\alpha. Increasing α\alpha produces more diverse trajectories, but when the generator's success rate falls below approximately 90%, downstream performance also begins to decline or stops improving on three tasks (Figure 19). FMB-Round is the exception: generator success remains high, and downstream performance continues to improve across the tested range. This suggests that the most useful diversity is achieved by increasing α\alpha while maintaining reliable task success, rather than maximizing diversity alone (Appendix H.1).

Thank you for the additional experiments. My questions are well-addressed by the rebuttal. I will keep my score of clear accept.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We thank the reviewers for their constructive feedback and for recognizing the strengths of our work across its motivation, method, experiments, and analysis. The reviewers acknowledged the importance of using simulation to train robot foundation models such as π0.5\pi_{0.5} (RTkv), and found our approach well motivated, intuitive, and easy to understand (7AyF). They highlighted substantial and consistent downstream improvements in simulation and on the real robot (RTkv, jBnR, 7AyF), including comparisons against strong baselines such as MimicGen (RTkv), and appreciated the controlled P-RFCL comparison that isolates the contribution of the diversity objective (jBnR).

Beyond performance gains, the reviewers recognized the new insight offered by the work (RTkv): teacher success alone does not fully characterize demonstration quality (jBnR), greater diversity is not always better (RTkv, jBnR), and successful-behavior diversity complements recovery supervision (jBnR). The detailed analyses of diversity, coverage, efficiency, and safety, together with the insightful appendix, were also noted as strengths (7AyF).

The revision expands the experimental evidence in four directions:

  • Controlled comparisons: We added a DLR-style comparison (Appendix F.2) and examined whether pooling independently trained P-RFCL generators could reproduce the benefits of our diversity objective (Appendix H.3). We also compared methods with matched action-transition counts to control for differences in supervision volume (Appendix H.4).
  • Scalability and robustness: We evaluated how downstream performance changes as the dataset size increases from 100 to 30,000 demonstrations (Figure 8) and tested whether the improvements generalize to a second downstream learner (Appendix F.1). We further examined generators trained from fewer or deliberately narrow seed demonstrations (Appendix H.7) and introduced a retrospective heuristic for selecting α\alpha based on generator success (Appendix H.1).
  • Dataset characterization: We compared the coverage of generated demonstrations against the original seed demonstrations at equal set sizes (Appendix G.1), quantified grasp diversity (Appendix G.2), and examined which initial scenes were retained after safety filtering (Appendix G.3). We also analyzed the behaviors associated with different latent skills (Appendix H.2).
  • Downstream behavior and correction: We investigated how well generated demonstrations cover states encountered when the downstream learner deviates from expert behavior (Section 4.4; Appendix H.5) and further analyzed recovery behaviors (Appendix H.6). We also extended DAgger- and DART-style correction experiments to all four tasks to examine how solution diversity interacts with on-policy correction (Section 4.6).

In the following comments, we present the main results directly below, with references to the revised manuscript.

We have uploaded the revised manuscript. Changes are marked in teal, with the relevant reviewer IDs in the margins to help locate the corresponding revisions.

Dear AC and Reviewers,

Thank you for overseeing our submission and for the constructive feedback. We appreciate that the reviewers recognized the importance and clear motivation of our approach (RTkv, 7AyF), the substantial and consistent downstream improvements in simulation and on real robots (RTkv, jBnR, 7AyF), and the controlled comparison isolating the contribution of solution diversity (jBnR). They also highlighted the insights that teacher success alone does not determine demonstration quality and that greater diversity is not always better (RTkv, jBnR), together with our extensive analyses of diversity, coverage, efficiency, and safety (7AyF).

Building on these strengths, we have substantially strengthened the paper through new controlled comparisons, scaling studies, and analyses addressing the reviewers' major concerns. All changes are marked in teal, with the relevant reviewer IDs in the margins. We summarize the key results below.

  1. Novelty and necessity of the diversity objective (RTkv, jBnR, 7AyF). DiscoDemo repurposes curriculum RL and skill discovery to construct imitation datasets from multiple successful solutions. Our massively parallel reverse curriculum also achieves approximately 17× faster reverse-phase training than the original RFCL in a five-demonstration comparison (Appendix B.5). Beyond the controlled P-RFCL comparison, which disables only the diversity objective, we added two baselines: a DLR-style variant achieves 43% average downstream success, and pooling five independently trained P-RFCL generators achieves 48% at 3.7× the training cost, compared with 62% for DiscoDemo, which leads on every task (Appendices F.2 and H.3).

  2. What drives the gains (jBnR, 7AyF). DiscoDemo maintains its advantage on all four tasks when action-transition counts are matched in either direction, showing that the gains are not simply due to greater supervision volume (Appendix H.4). Its demonstrations also provide equal or better coverage of states encountered when the downstream π0.5 learner deviates on three of four tasks, which we present as diagnostic rather than causal evidence (Section 4.4). Recovery rates after errors further improve from 28% to 67% on FMB-Round and 7% to 22% on FMB-SqCircle (Appendix H.6).

  3. Scale, seed dependence, and useful diversity (RTkv, jBnR, 7AyF). Just 300 DiscoDemo demonstrations outperform 30,000 P-RFCL demonstrations on all four tasks under the same downstream training budget (Figure 8). DiscoDemo also discovers varied successful behaviors from deliberately narrow scripted seeds and even a single human demonstration when the appropriate curriculum is restored (Appendix H.7). Further analyses show that learned latent skills correspond to distinct trajectory behaviors (Mantel test, p≤0.003p \leq 0.003), while the diversity-weight sweep highlights the importance of preserving generator success rather than maximizing diversity alone (Appendices H.1–H.2).

  4. Complementarity with corrective collection (jBnR, 7AyF). We extended DAgger and DART-style correction to all four tasks, including uncorrected baselines. DiscoDemo maintains higher success in all eight corrected settings. Although the gap narrows on PnP-Banana as both methods approach the performance ceiling, substantial improvements remain elsewhere, including 89% vs. 48% on StackCube with DAgger and 78% vs. 37% on FMB-SqCircle with DART-style correction (Section 4.6).

  5. Real-robot robustness (jBnR, 7AyF). We expanded real-robot evaluations to three fine-tuning seeds, with 20 trials per task and seed. DiscoDemo achieves 76.7 ± 1.4% average success versus 36.3 ± 2.2% for the strongest baseline, with non-overlapping ±1 s.d. ranges against all baselines on every task (Figure 3; Table 8). These results replace the original single-seed evaluation and strengthen the evidence for consistent real-world improvements.

Our core finding remains consistent: DiscoDemo improves average downstream success from 40% to 62% in simulation and from 36% to 77% on the real robot. We have also clarified the remaining limitations. On FMB-Round, differences in generator competence and retained-scene coverage contribute to the comparison; nevertheless, excluding this task, average simulation success still improves from 41% to 62%. A preliminary evaluation with Diffusion Policy also shows improvement on StackCube (24% to 50%), although its single-seed results and low insertion-task success do not establish broad generalization beyond π0.5 (Section 5; Appendix F.1).

We sincerely appreciate the reviewers' constructive feedback, which has substantially strengthened our empirical evidence and clarified the scope of our contribution. We kindly ask that these additional results and revisions be considered during the final discussion.

Sincerely,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.