- Rating
- 3
- Confidence
- 1
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This work proposes COACH, a method for interactive imitation learning, in which the method selects its own training curriculum and a supervising expert. Thus, the method tries to optimize in each training step finding the best training scenario, and for the scenario the best supervisor at lowest supervision cost. These two selections are treated as bandit problems.
Critical strengths
Experimental procedure seems solid, with a wide selection of baselines and supervision experts, both RL policies and human participants. Experimental ablations seem to confirm, that the methods benefit come from the joint selection of training samples (curriculum) and supervision expert. Additional experiments also confirm that the method is not simply selecting the lowest cost expert.
I believe the insight, that the method seems to favor low-cost "junior" experts at later stages in training, could be interesting for future research. Intuitively, I would have guessed, that the model learns difficult tasks late, thus requiring expert supervision mostly in late training stages.
Critical weaknesses
None, but as detailed below, I am not really familiar with this topic.
Questions for authors
I find the finding very interesting, that a mature learner prefers Junior experts. I agree with the appendix, that this likely means a junior expert mainly provides low-cost monitoring. I would be interested in seeing COACH behavior with a fourth "zero" expert option. Thus, the model could indicate which scenarios it is confident in solving independently, or even signal when the training process is finished. I would welcome the authors thoughts on this idea.
2 comments
The new experiments and clarification are appreciated. I think they point towards interesting avenues for future research and further strengthen the contribution.
I maintain my original rating and support acceptance.
Q1 Zero Expert Extension
Thank you for this insightful suggestion. We have extended COACH with a fourth option, Zero Expert, which allows the learner to execute without expert monitoring or intervention at zero supervision cost. We conducted additional experiments on MetaDrive over three random seeds and introduced three metrics: Zero Selection, the percentage of episodes assigned to Zero Expert; Late-Stage Zero, the corresponding percentage during the final 20% of training; and Success@Zero, the success rate of episodes assigned to Zero Expert.
The results show that introducing Zero Expert reduces supervision cost from 342 to 265.84 ± 7.50, while maintaining a 92.0% success rate and an episodic return of 360.30 ± 6.20. The Zero Selection and Late-Stage Zero ratios reach 25.79% ± 1.09% and 56.67% ± 5.77%, respectively, suggesting that COACH increasingly favors autonomous execution as the learner improves. Success@Zero reaches 76.50% ± 7.06%, indicating that most episodes assigned to Zero Expert are completed successfully, although autonomous execution is not always reliable.
We also agree that Zero Expert selection could provide useful signals, but it does not necessarily indicate permanent scenario mastery or training completion. In online learning, previously acquired skills may deteriorate due to forgetting, requiring renewed supervision. Training completion should instead be assessed through learning convergence and task performance. Moreover, under HG-DAgger, Zero Expert provides no new corrective demonstrations, even though learning may continue from previously collected data. Nevertheless, this extension offers a promising way to further reduce supervision costs. We have added the results to the revised Section 5.3 and Appendix C.3.