- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents DEC, a framework for sample-efficient real-world reinforcement learning with limited human intervention. DEC uses a dynamics-aware discriminator to decide when human takeover is needed, a robust value-learning scheme for long-horizon sparse-reward tasks, and positive/negative policy branches that utilize both successful and suboptimal experience. The method is evaluated on ROBOSUITE tasks and four real-world bimanual manipulation tasks. The results show substantial policy improvement with relatively limited interaction and human supervision, including challenging insertion and long-horizon assembly tasks.
Critical strengths
-
The paper addresses an important practical problem in real-world robot learning. Reducing both robot interaction and human supervision is highly relevant for real-world RL, and the DEC formulation provides a coherent way to make better use of the collected experience.
-
The real-world validation is strong. The evaluation includes challenging bimanual tasks such as pen assembly, rather than simple short-horizon simulation tasks. The improvements over the base policy and competing methods are substantial, which makes the practical contribution convincing.
Critical weaknesses
A critical weakness I observe is that discriminator and value models are trained from task-specific experience, and their ability to transfer across tasks or environments is not verified. As a core component, this may limit how easily DEC can be reused when adapting a pretrained policy to many different tasks. Retrain a specific discriminator or value model for each new task may bring higher cost to reducing human intervene time.
Questions for authors
- Were the real-world training experiments repeated with multiple independent runs? If not, how sensitive did you observe DEC to different rollout data or human intervention trajectories?
- Is there a ablation study that directly compares removing each major component from the full DEC system to see the contribution of each component to the final performance?
- The core problem the paper tries to solve is reduce the time of human intervene in RL training. Based on the results show in Fig.7, the time of human intervene reduce about 20%-50% on real world task. However, I assume the fact that during real-world RL training, human need to stay by the side all the time to monitor, reset, and intervene sometimes. In this case, does the reducation of intervene time real matter as it only takes a smaller part of the whole training time. Answer this question will greatly help me understand how this method really contributes to the efficiency of real world RL.
Please take a moment to respond to the rebuttal. Thank you!