- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces DiscoDemo, which trains an RL policy in simulation, and then uses it as a data generator to generate thousands of demonstrations for downstream imitation learning. The key idea that differentiates this from prior work is that this RL policy is trained to not only succeed at the task, but is also explicitly rewarded for the diversity of its trajectories, as long as those trajectories are successful (they do not reward diverse trajectories that are unsuccessful to avoid reward hacking). They argue that this diversity is important for creating an effective imitation learning dataset. Concretely, their method builds on prior work that trains a privileged RL policy in simulation with reverse curriculum learning using 20 human demonstrations. The policy is conditioned in a latent variable z that is sampled at the start of the episode and is held fixed throughout each episode. Once the RL policy is trained, it is used to generate thousands of demonstrations, which are then filtered for success and safety. This is then used to fine-tune the pretrained pi0.5 policy. DiscoDemo is evaluated on 4 tasks in both simulation and in the real world: pick and place banana into bowl, stacking cubes, and two FMB insertion tasks. Compared to a wide range of baselines, it achieves higher performance in simulation and substantially higher performance in the real-world. They also perform additional experiments to study diversity, coverage, generation efficiency, safety filtering, and how it works with corrective data collection.
Critical strengths
Strength 1: Well-motivated method and thorough experimental results The method is easy to understand, intuitive, and well-motivated. The paper includes extensive comparisons to baselines and interesting analyses of diversity, coverage, generation efficiency, and safety. The appendix is detailed and insightful.
Strength 2: Reasonably strong empirical results and analysis DiscoDemo demonstrates consistent improvement over baselines in simulation and in the real world. They also study how varying the diversity parameter alpha affects diversity and success rate, which is helpful.
Critical weaknesses
Weakness 1: Limited novelty in the method The method combines existing techniques, including reverse curriculum learning and latent-conditioned diversity optimization, which makes the work feel incremental. While the improvements are interesting, the additional benefit of diversity after corrective data collection is mixed. For example, DAgger nearly closes the performance gap on PnP-Banana and reduces the gap on FMB-SqCircle.
Weakness 2: Lacking more insightful experiments that would validate this specific method While the paper has extensive experimental results, it does not answer important questions about diversity and scale. In particular, all of their methods use 20 human demonstrations and 3000 generated demonstrations. It is unclear how much diversity there is in the human demonstration, how this method and others would perform with 1 or 100 human demonstrations, and how this method and others perform with 300 or 30000 generated demonstrations. This would provide more insight into how diversity interacts with scale. Additionally, it is unclear if this specific diversity approach is needed. For example, it would be helpful to compare with training multiple separately trained RL teachers pooling trajectories from all of them as another form of diversity.
Questions for authors
Question 1: How sensitive is the method to the number and diversity of human demonstrations? Does it still discover diverse solutions with 1 human demo? Does it still discover distinct solutions if all of the human demos approach the problem the same way?
Question 2: How does downstream performance scale with the number of generated demonstrations, beyond the fixed 3000 used in the paper?
Question 3: What are the dominant downstream failure modes? Does DiscoDemo demonstrate improved recovery behavior?
Question 4: Could the authors extend the DAgger/DART experiments in Figure 8 to all four tasks and include uncorrected baselines? The performance gap after correction becomes quite small on PnP-Banana, so understanding when solution diversity provides additional benefits would be valuable.
4 comments
Response to Reviewer 7AyF [2/3]
Diversity and scale (W2, Q2)
Scaling with generated demonstrations
Thank you for this suggestion; the new scaling study directly examines how diversity interacts with scale. We evaluated downstream performance using 100–30,000 demonstrations under the same 20k-step fine-tuning budget, with three training seeds and 50 evaluation episodes per seed (Section 4.4).
The table below summarizes mean downstream success (%) from Figure 8 and Table 10; each cell lists P-RFCL / DiscoDemo.
| Task | 300 demonstrations | 3,000 demonstrations | 30,000 demonstrations |
|---|---|---|---|
| PnP-Banana | 55 / 78 | 73 / 85 | 68 / 85 |
| StackCube | 17 / 43 | 29 / 57 | 29 / 52 |
| FMB-Round | 45 / 53 | 39 / 61 | 44 / 70 |
| FMB-SqCircle | 14 / 43 | 21 / 45 | 21 / 41 |
DiscoDemo outperforms P-RFCL at every tested dataset size, with even 300 DiscoDemo demonstrations exceeding 30,000 P-RFCL demonstrations on all four tasks. However, DiscoDemo itself shows diminishing returns beyond 3,000 demonstrations on most tasks, with FMB-Round showing the clearest additional improvement. Thus, simply collecting more demonstrations from P-RFCL does not close the performance gap under the fixed training budget.
Diversity of human demonstrations
To examine whether the generated datasets simply reproduce the diversity present in the human demonstrations, we compared the original 20 human seeds with equally sized subsets of generated demonstrations (Figure 15; Appendix G.1).
DiscoDemo achieves higher coverage than its human seeds on all four tasks, whereas P-RFCL falls below the seeds on three. On PnP-Banana, both RL generators discover grasp configurations absent from the human demonstrations, but DiscoDemo produces the greatest variation in grasp positions and closing angles within a fixed scene (Figure 17; Appendix G.2). These results suggest that RL can discover behaviors beyond the seeds, while the diversity objective further expands the range of successful solutions.
We also tested generators trained from fewer human demonstrations and deliberately narrow motion-planning demonstrations. The detailed results are presented in our response to Q1 below (Appendix H.7).
Pooling independently trained RL generators
To test whether diversity from independent RL training runs could replace explicit diversity optimization, we trained five P-RFCL generators with different random seeds per task and pooled 600 demonstrations from each, keeping the final dataset at 3,000 demonstrations. We applied the same filtering, rendering, and downstream fine-tuning protocol (Appendix H.3).
The table below summarizes downstream success (%) from Figure 23 and task-averaged generator training costs from Appendix I.1; the pooled cost includes all five training runs but excludes rollout and rendering.
| Data source | PnP-Banana | StackCube | FMB-Round | FMB-SqCircle | Average | Training GPU-h |
|---|---|---|---|---|---|---|
| P-RFCL | 73 | 29 | 39 | 21 | 40 | 6.1 |
| P-RFCL, pooled | 73 | 40 | 47 | 31 | 48 | 30 |
| DiscoDemo | 85 | 57 | 61 | 45 | 62 | 8.2 |
Pooling improves on a single P-RFCL generator on three tasks but remains below DiscoDemo on every task, despite approximately 3.7 times the generator training cost. In a fixed scene, the pooled trajectories exhibit some variation across seeds but remain concentrated along a few narrow paths, roughly one per generator (Figure 22). Thus, independent RL training randomness provides some diversity, but does not replace explicit diversity optimization in this comparison.
Response to Reviewer 7AyF [3/3]
Number and diversity of seed demonstrations (Q1)
We varied the number and source of seed demonstrations on PnP-Banana, using the same 50M-step RL training budget and as in the main comparison (Appendix H.7). We tested 1, 5, 10, and 20 human demonstrations, as well as 20 and 100 scripted motion-planning (MP) demonstrations.
Fewer human demonstrations.
Our default curriculum retains RFCL's reverse curriculum but omits its prioritized level replay (PLR)-based forward curriculum. With only one human demonstration, the default generator fails to learn the task from the full initial-state distribution. We therefore additionally tested restoring RFCL's PLR-based forward curriculum for the 1- and 5-demonstration settings.
The table below summarizes generator success (%) from Figure 27, evaluated from the original task distribution .
| Seed demonstrations | Default curriculum | With PLR |
|---|---|---|
| 1 human | 0 | 55 |
| 5 human | 29 | 68 |
| 10 human | 75 | – |
| 20 human | 94 | – |
| 20 scripted MP | 78 | – |
| 100 scripted MP | 86 | – |
With the default curriculum, generator success improves as more human demonstrations are provided. However, restoring PLR enables learning from even a single human demonstration, reaching 55% generator success. Despite this lower success rate, the single-demonstration generator discovers distinct successful trajectories and grasp configurations beyond its seed example (Figures 29 and 30). This suggests that a single demonstration can support solution discovery with the appropriate curriculum, although reliable task completion still benefits from more demonstrations.
Discovery from deliberately narrow seeds.
To examine whether DiscoDemo can discover solutions absent from its seeds, we replaced the human demonstrations with 20 scripted MP demonstrations that use concentrated grasp strategies. Despite these narrow seeds, DiscoDemo reaches 78% generator success, produces varied trajectories from the same initial scene, and discovers grasps outside the dominant seed grasp configuration (Figures 27, 29, and 30).
The table below summarizes the PnP-Banana diversity and coverage results from Figure 28. Fixed-scene diversity is undefined for the human seeds because they start from different scenes.
| Generator or seed source | Fixed-scene diversity | Coverage |
|---|---|---|
| DiscoDemo, 20 human seeds | 7.6 | 14.1 |
| DiscoDemo, 10 human seeds | 6.0 | 15.1 |
| DiscoDemo, 20 MP seeds | 5.0 | 15.0 |
| P-RFCL, 20 human seeds | 1.7 | 8.2 |
| 20 human seeds | – | 9.4 |
| MP | 1.4 | 4.3 |
DiscoDemo trained from narrow MP seeds achieves substantially higher diversity than MP or P-RFCL, while maintaining coverage comparable to DiscoDemo trained from human seeds. Increasing the MP seed count to 100 also yields 86% generator success (Figure 27). These results suggest that scripted demonstrations could reduce the need for human teleoperation, although we have only evaluated this possibility on PnP-Banana. The 100-demonstration condition uses scripted MP seeds rather than human demonstrations.
Together, these experiments show that DiscoDemo can discover diverse successful solutions beyond its seed demonstrations, rather than simply reproducing their original behavioral variations.
Failure modes and recovery (Q3)
To investigate downstream failures and recovery behavior, we evaluated one policy per dataset on the same 10 initial scenes with 10 policy-noise seeds per scene, yielding 100 rollouts per task and dataset (Appendix H.6). We identify errors directly from simulator states, including missed grasps, slips, and drops for pick-and-place tasks, and missed holes and insertion jams for insertion tasks. An episode is counted as recovered if it succeeds after its first detected error, and recovery rate is computed over episodes containing at least one detected error.
The table below summarizes recovery rates (%) from Figure 25; Figure 26 provides qualitative examples.
| Task | P-RFCL recovery | DiscoDemo recovery |
|---|---|---|
| PnP-Banana | 25 | 24 |
| StackCube | 0 | 4 |
| FMB-Round | 28 | 67 |
| FMB-SqCircle | 7 | 22 |
Missed grasps and missed holes are the most common first detected errors, while drops are also frequent for P-RFCL on StackCube. The clearest recovery improvements occur on the insertion tasks, where DiscoDemo increases recovery from 28% to 67% on FMB-Round and from 7% to 22% on FMB-SqCircle. On the pick-and-place tasks, recovery is similar or rare for both methods, and DiscoDemo's main advantage is a greater number of successful episodes without detected errors.
These results distinguish two benefits of DiscoDemo demonstrations: reducing the frequency of downstream errors and, particularly for insertion tasks, improving recovery after errors occur.
Follow-up Response to Reviewer 7AyF
Real-robot results over three fine-tuning seeds (W1)
We evaluated all methods on the real robot with three fine-tuning seeds (20 trials per task and seed) and now report mean s.d. (Figure 3 and Table 8).
| Task | Strongest baseline | DiscoDemo |
|---|---|---|
| PnP-Banana | 36.7 ± 7.6 (P-RFCL) | 86.7 ± 5.8 |
| StackCube | 55.0 ± 13.2 (MG) | 76.7 ± 2.9 |
| FMB-Round | 48.3 ± 10.4 (P-RFCL) | 81.7 ± 7.6 |
| FMB-SqCircle | 23.3 ± 7.6 (P-RFCL) | 61.7 ± 5.8 |
| Average | 36.3 ± 2.2 (P-RFCL) | 76.7 ± 1.4 |
On all four tasks, the s.d. range of DiscoDemo does not overlap with that of any baseline. We have updated Section 4.1, Appendix A, and the Reproducibility Statement accordingly, and removed the single real-robot seed from the limitations in Section 5.
We hope these results address your concern. If there are any remaining questions or suggestions, we would be glad to address them during the discussion period.
Response to Reviewer 7AyF [1/3]
Thank you for the thoughtful feedback. We extended corrective collection to all four tasks, varied the number of generated and seed demonstrations, compared pooled RL generators, and analyzed downstream failure modes and recovery.
Novelty and contribution (W1)
DiscoDemo builds on established curriculum RL and skill-discovery methods, but uses them for a different purpose: discovering multiple successful solutions to construct more effective datasets for downstream imitation learning.
Rather than optimizing task success alone, we explicitly encourage diversity among successful RL trajectories and export them as a reusable static dataset. Our controlled P-RFCL comparison isolates this contribution by holding the RL backbone, curriculum, safety criteria, dataset size, and downstream training fixed while disabling the diversity objective (Sections 3 and 4.1). We also introduce a massively parallel reverse curriculum, achieving approximately 17× faster reverse-phase training than the original RFCL in our five-demonstration PnP-Banana comparison (Appendix B.5).
Complementarity with corrective data collection (W1, Q4)
We thank the reviewer for this suggestion. We extended DAgger and DART-style correction to all four tasks and included the uncorrected baselines in Figure 11 (Section 4.6). For each method, we use its own frozen RL generator as the teacher for corrective collection (Appendix C).
The table below summarizes downstream success (%) from Figure 11; each cell lists P-RFCL / DiscoDemo.
DiscoDemo maintains higher downstream success with either correction method on all four tasks, although the improvement varies by task. We agree that DAgger nearly closes the gap on PnP-Banana, where both methods approach the performance ceiling, and reduces it on FMB-SqCircle. However, substantial gaps remain elsewhere, particularly on StackCube with DAgger (48% vs. 89%) and FMB-SqCircle with DART-style correction (37% vs. 78%).
Importantly, both correction methods also improve DiscoDemo on every task, showing that diverse successful demonstrations do not eliminate the need for corrective supervision. These results demonstrate that solution diversity and corrective collection are complementary at the pipeline level, rather than competing approaches.