acceptodds
Under review as a conference paper at ICLR 2027

What to Learn and Who to Ask: Adaptive Curriculum and Expert Allocation for Interactive Imitation

Abstract

Interactive Imitation Learning (IIL) enables agents to master complex behaviors through real-time human interventions. However, its scalability is hindered by high supervision costs. Prior work has studied adaptive curricula and teacher selection, but not their coupled non-stationarity in intervention-based online IIL, where scenario value and supervision needs evolve with the learner. We propose COACH (Cost-Optimized Adaptive Curriculum & Human-supervision), which formulates this setting as a coupled non-stationary decision problem. COACH tracks scenario-specific learning dynamics to determine what to train next, while routing episode-level supervision according to the selected scenario, learner proficiency, and expert cost. COACH adopts a factorized but context-coupled architecture and uses Lazy Re-evaluation to selectively refresh non-stationary curriculum estimates. Extensive experiments on MetaDrive and MetaWorld demonstrate that COACH outperforms state-of-the-art baselines in task performance and cost reduction, results further confirmed through validation with human participants. Finally, we show COACH generalizes to skill-to-task matching, autonomously identifying human domain specialists without cost incentives.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Interactive Imitation Learning, Human-in-the-loop, Curriculum Learning, Bandit

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_hzPZOpenReview ↗
Rating
3
Confidence
1
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This work proposes COACH, a method for interactive imitation learning, in which the method selects its own training curriculum and a supervising expert. Thus, the method tries to optimize in each training step finding the best training scenario, and for the scenario the best supervisor at lowest supervision cost. These two selections are treated as bandit problems.

Critical strengths

Experimental procedure seems solid, with a wide selection of baselines and supervision experts, both RL policies and human participants. Experimental ablations seem to confirm, that the methods benefit come from the joint selection of training samples (curriculum) and supervision expert. Additional experiments also confirm that the method is not simply selecting the lowest cost expert.

I believe the insight, that the method seems to favor low-cost "junior" experts at later stages in training, could be interesting for future research. Intuitively, I would have guessed, that the model learns difficult tasks late, thus requiring expert supervision mostly in late training stages.

Critical weaknesses

None, but as detailed below, I am not really familiar with this topic.

Questions for authors

I find the finding very interesting, that a mature learner prefers Junior experts. I agree with the appendix, that this likely means a junior expert mainly provides low-cost monitoring. I would be interested in seeing COACH behavior with a fourth "zero" expert option. Thus, the model could indicate which scenarios it is confident in solving independently, or even signal when the training process is finished. I would welcome the authors thoughts on this idea.

2 comments

Q1 Zero Expert Extension

Thank you for this insightful suggestion. We have extended COACH with a fourth option, Zero Expert, which allows the learner to execute without expert monitoring or intervention at zero supervision cost. We conducted additional experiments on MetaDrive over three random seeds and introduced three metrics: Zero Selection, the percentage of episodes assigned to Zero Expert; Late-Stage Zero, the corresponding percentage during the final 20% of training; and Success@Zero, the success rate of episodes assigned to Zero Expert.

MethodExpert Cost ↓Return ↑Success ↑Zero Selection (%)Late-Stage Zero (%)Success@Zero (%)
COACH342360.50 ± 2.592.0% ± 1.0%---
COACH + Zero265.84 ± 7.5360.3 ± 6.292.0% ± 1.0%25.79 ± 1.0956.67 ± 5.7776.50 ± 7.06

The results show that introducing Zero Expert reduces supervision cost from 342 to 265.84 ± 7.50, while maintaining a 92.0% success rate and an episodic return of 360.30 ± 6.20. The Zero Selection and Late-Stage Zero ratios reach 25.79% ± 1.09% and 56.67% ± 5.77%, respectively, suggesting that COACH increasingly favors autonomous execution as the learner improves. Success@Zero reaches 76.50% ± 7.06%, indicating that most episodes assigned to Zero Expert are completed successfully, although autonomous execution is not always reliable.

We also agree that Zero Expert selection could provide useful signals, but it does not necessarily indicate permanent scenario mastery or training completion. In online learning, previously acquired skills may deteriorate due to forgetting, requiring renewed supervision. Training completion should instead be assessed through learning convergence and task performance. Moreover, under HG-DAgger, Zero Expert provides no new corrective demonstrations, even though learning may continue from previously collected data. Nevertheless, this extension offers a promising way to further reduce supervision costs. We have added the results to the revised Section 5.3 and Appendix C.3.

The new experiments and clarification are appreciated. I think they point towards interesting avenues for future research and further strengthen the contribution.

I maintain my original rating and support acceptance.

Reviewer_zzYvOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes COACH, a framework for jointly adapting curriculum selection and expert allocation in interactive imitation learning. It uses an Exp3-based curriculum bandit to select training scenarios based on learning velocity, a NeuralUCB bandit to allocate experts based on task context and learner proficiency, and a lazy re-evaluation mechanism to reduce computational overhead. Experiments on MetaDrive and MetaWorld demonstrate improved task performance and reduced supervision costs, with additional validation involving human participants.

Critical strengths

S1. Comprehensive experimental evaluation.

The paper provides extensive experiments across two different environments, including comparisons with multiple baselines, ablation studies, scalability analysis, and human-expert validation. The results demonstrate a favorable trade-off between task performance and supervision cost.

S2. Clear presentation and practical relevance.

The paper addresses an important problem in interactive imitation learning. The motivation, framework, and technical details are clearly presented, making the proposed method easy to understand.

Critical weaknesses

W1. Limited methodological novelty and a gap between the optimization objectives.

The proposed framework primarily combines existing techniques, including Exp3, NeuralUCB, and learning velocity. More importantly, while Eq. (1) formulates a joint utility maximization problem, the Curriculum Bandit optimizes absolute loss velocity (Eq. (4)) without explicitly considering supervision costs, whereas the Supervision Bandit optimizes cost-sensitive episode utility (Eq. (6)). It is unclear whether these separate objectives effectively optimize the stated joint objective. Furthermore, the regret analysis in Appendix A only covers surrogate bandit objectives rather than the joint utility.

W2. Insufficient comparative evaluation with human experts.

Section 5.4 reports COACH's performance with nine human participants but does not provide corresponding baseline results. However, Appendix B (HIL Experiment Details) states that participants evaluated nine algorithms. Since the main comparisons in Table 1 rely on synthetic experts, the absence of human-expert baseline results makes it difficult to determine whether COACH's performance and cost advantages consistently hold under real human supervision.

Questions for authors

Q1. Could the authors further clarify how optimizing the two surrogate objectives relates to maximizing the overall joint utility?

Q2. Could the authors provide the complete baseline comparison results for the human-expert experiments described in Appendix B?

5 comments

[W1] Methodological novelty

We thank the reviewer for raising this important concern. Our methodological contribution lies in addressing a distinct challenge in intervention-based online imitation learning: the coupled non-stationarity of curriculum scheduling and expert allocation. In this setting, expert interventions continuously update the learner, causing both the training value of each scenario and its supervision requirements to evolve. Existing curriculum methods primarily focus on which scenarios to prioritize, while expert-selection methods focus on allocating supervision for a given task. Treating these decisions independently may overlook their evolving dependencies, whereas directly exploring all scenario–expert pairs incurs substantial costs under limited human-interaction budgets. The central challenge is therefore to jointly adapt curriculum scheduling and expert allocation to the evolving learner while avoiding the prohibitive exploration cost of the full scenario–expert space.

The novelty of COACH lies in its factorized but context-coupled decision framework designed to address this challenge. Rather than treating scenario–expert pairs as independent arms, COACH factorizes the decision space while conditioning expert allocation on the selected scenario and learner proficiency. More importantly, expert interventions continuously update the learner, which in turn changes future curriculum priorities and supervision requirements. This design reduces the exploration burden of joint decision-making while retaining the dynamic dependencies between the two decisions. Learning velocity, Exp3, and NeuralUCB are established techniques used to instantiate this framework, rather than the source of its methodological novelty. We will clarify this framework-level contribution in the revised manuscript.

[W1-Q1] Objective Alignment

We thank the reviewer for this important question. Eq. (1) defines our overall joint objective, with its cost-sensitive utility instantiated by Eq. (6). Rather than directly exploring all scenario–expert pairs, COACH adopts a factorized approach in which the two stages contribute to the overall objective through policy improvement and cost-effective supervision.

The key connection is HG-DAgger, which links curriculum selection to future task utility through policy learning. Specifically, the Curriculum Bandit prioritizes scenarios exhibiting significant learning dynamics (Eq. (4)), aiming to collect informative expert interventions that HG-DAgger uses to update the learner's policy and minimize imitation loss. More effective policy learning is expected to increase task success probability and, consequently, the expected utility in Eq. (6). As the learner becomes more proficient, lower-cost experts may successfully supervise scenarios that previously required expensive experts. The Supervision Bandit captures this evolving performance–cost trade-off through Eq. (6), further improving supervision efficiency. In summary, curriculum selection promotes policy improvement, while supervision allocation exploits the learner's evolving capabilities to reduce supervision costs, together contributing to the joint objective in Eq. (1).

We acknowledge that this factorization does not guarantee global optimality for Eq. (1), and our theoretical analysis characterizes the surrogate decisions rather than joint optimality. Nevertheless, experimental results demonstrate its effectiveness: COACH achieves the highest success rate and lowest expert cost among the compared methods in both environments (Table 1). Furthermore, Table 3 shows that COACH outperforms Joint MAB as the decision space grows (84% vs. 56% success at 50×10). We have clarified these relationships in the revised Section 4.3.

[W2-Q2] Human Expert Comparison

We thank the reviewer for pointing out the incomplete reporting of our human-expert evaluation. We have extended the experiments to include all the methods under the same human-supervision protocol, reporting mean ± std across three human-expert groups in the revised Table 4. To mitigate potential learning and fatigue effects, we counterbalanced the algorithm evaluation order across the three groups.

The additional results confirm COACH's effectiveness with real human supervision. In MetaDrive, COACH achieves the highest success rate (95% ± 2%) and episodic return (358.03 ± 4.07), along with the lowest safety cost (0.48 ± 0.03). Its expert cost (341 ± 18) is also lower than those of the joint-selection baselines SGIM, CEAL, and Joint MAB. In MetaWorld, COACH achieves the highest success rate (92% ± 8%) and episodic return (2602.68 ± 99.2), while incurring the lowest expert cost (166 ± 3) among all compared methods.

These results are consistent with our synthetic-expert evaluations and further support the effectiveness of COACH under real human supervision. We have added the complete comparison results to Section 5.4 (Table 4) and the corresponding experimental protocol to Appendix B.

MethodMetaDrive Expert DataExpert Cost ↓Episodic Return ↑Safety Cost ↓Success Rate ↑MetaWorld Expert DataExpert Cost ↓Episodic Return ↑Success Rate ↑
Random4033± 196454 ± 40293.66 ± 11.74.34 ± 0.30.58 ± 0.044261± 210223 ± 181638.23 ± 74.10.63 ± 0.05
Heuristic4300± 203411 ± 34331.12 ± 18.64.54 ± 0.40.61 ± 0.075019± 213240 ± 171821.12 ± 76.60.60 ± 0.02
Geometric3660± 84456 ± 18319.37 ± 25.23.68 ± 0.10.75± 0.144170± 45232 ± 42153.51 ± 87.30.70 ± 0.13
DQM3490± 140393 ± 28303.77 ± 12.65.08 ± 0.40.74 ± 0.083469± 32198 ± 32045.54 ± 57.70.76 ± 0.04
RgC3435± 35396 ± 7331.20 ± 25.04.28 ± 0.20.85 ± 0.043141± 101201 ± 102191.88 ± 78.90.84 ± 0.01
SMT3663± 78309 ± 11347.77 ± 3.01.59 ± 0.10.89 ± 0.072805± 144194 ± 161974.32 ± 110.20.80 ± 0.06
SGIM3190± 54407 ± 12330.73 ± 10.53.05 ± 0.10.81 ± 0.033081± 55224 ± 62152.88 ± 249.20.87 ± 0.05
CEAL3044± 102469 ± 28347.30 ± 12.26.13 ± 0.50.84 ± 0.042773± 99238 ± 142393.45 ± 294.00.86 ± 0.08
Joint MAB3036± 39426 ± 6334.13 ± 16.83.53 ± 0.020.84 ± 0.032956± 66217 ± 82277.65 ± 70.50.86 ± 0.09
SMT+COACH2518± 31344 ± 49354.13 ± 13.30.63 ± 0.010.92 ± 0.042845± 69186 ± 72426.47 ± 97.80.87 ± 0.03
COACH (Ours)2533± 79341 ± 18358.03 ± 4.070.48 ± 0.030.95 ± 0.022601± 35166 ± 32602.68 ± 99.20.92 ± 0.08
Reviewer_TWimOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies interactive imitation learning, where an expert watches the learner and takes over when it is about to fail. It assumes several experts with different skill levels and costs. In each round, COACH decides which scenario to train on and which expert should supervise. Scenarios are chosen by a bandit (Exp3) that prefers scenarios where the imitation loss is changing fast, and only a few scenario losses are recomputed each round to save cost (Lazy Re-evaluation). The expert is chosen by a contextual bandit (NeuralUCB) that trades off expected success against cost, based on the scenario and the learner's recent success rate. On MetaDrive and MetaWorld, with three RL policies acting as experts, COACH nearly matches always using the best expert at about half the expert cost. There is also a small study with nine student participants.

Critical strengths
  1. The problem is practical and the idea is natural. Moving to a cheaper expert once the learner is good enough makes sense, and in MetaDrive COACH nearly matches Senior-Only success at about half the expert cost. The SMT vs SMT+COACH comparison isolates the effect of expert allocation.

  2. The paper is clear about its limits. It reports that the flat Joint MAB wins in the small 3x2 setting (Table 3), and Appendix A.1 states that the bound is on a surrogate regret, not on final policy quality.

Critical weaknesses
  1. It is unclear how much the learned components add over simple alternatives. There is no simple rule baseline for expert choice, such as switching from Senior to Junior once training success passes a threshold. For Lazy Re-evaluation, refreshing a random subset of scenarios is not tested, and full re-evaluation is as good or slightly better.
  2. The method is mostly a combination of known parts, and the coupling it stresses is only loosely modeled. The curriculum signal follows the learning-progress idea of [1] and is used with Exp3, and expert choice is NeuralUCB [2] with a replay that down-weights old data. Section 6 also calls both bandits replaceable backends. The paper frames the two decisions as one coupled problem, but the curriculum bandit ignores expert cost, and the only link is that the chosen scenario is passed to the expert bandit as context.

Reference:

  • [1] Matiisen et al., Teacher-student curriculum learning. TNNLS 2019.
  • [2] Zhou et al., Neural contextual bandits with UCB-based exploration. ICML 2020.
Questions for authors
  1. Could you add a simple rule for expert choice, such as switching from Senior to Junior once recent training success passes a threshold, at matched cost? If such a rule came close to COACH, my view of the expert-allocation part would change.
  2. Each human group ran nine algorithms (Appendix B), but Section 5.4 reports only COACH. Could you report all nine in the same way as Table 1, so that COACH can be compared with the baselines when the experts are people?
  3. Section 5.1 defines Expert Cost as the assigned expert's per-unit-time cost summed over all training steps, and training uses a fixed step budget. Under this definition, the single-expert costs should be about 1:5:10. This holds in MetaWorld (38/188/375) but not in MetaDrive (94/355/466). How is the reported Expert Cost obtained in MetaDrive?

Minor:

  1. This is very minor. Usually some of the illustrations or descriptions will be provided in the main text so that readers have a quick senses of what are the problems that this paper want to resolve.
7 comments

[W1-Q1] Threshold-Based Expert Allocation

We thank the reviewer for this valuable suggestion. To examine whether simple proficiency-based switching can replace adaptive expert allocation, we introduce a threshold-based baseline that switches from Senior to Junior supervision once the learner's average success rate over the most recent 20 training episodes exceeds 0.8. This threshold is chosen to achieve supervision costs comparable to COACH for a fair comparison.

EnvironmentMethodExpert Cost ↓Episodic Return ↑Success Rate ↑
MetaDriveThreshold (0.8)336341.25 ± 6.283.0% ± 2%
COACH (Ours)342360.50 ± 2.592.0% ± 1.0%
MetaWorldThreshold (0.8)1662443.33 ± 45.482.0% ± 1%
COACH (Ours)1732592.98 ± 183.392.0% ± 4.0%

As shown above, COACH achieves higher success rates than threshold switching on both MetaDrive (92% vs. 83%) and MetaWorld (92% vs. 82%), at comparable expert costs. It also achieves higher episodic returns in both environments.

While threshold switching adapts supervision to overall learner proficiency, it applies a uniform rule across scenarios. In contrast, COACH conditions expert allocation on both scenario characteristics and learner proficiency, allowing simpler scenarios to transition earlier to Junior supervision while retaining stronger experts for challenging scenarios (Fig. 2). These results suggest that context-dependent expert allocation offers benefits beyond simple proficiency-based switching. We have incorporated these results into Table 2 and revised Section 5.3 accordingly.

[W1] Lazy Re-evaluation

We thank the reviewer for this suggestion. We add a Random Re-evaluation baseline that uniformly refreshes five scenarios per update, matching COACH's evaluation budget while keeping all other components unchanged.

EnvironmentMethodRe-evaluated ScenariosEpisodic Return ↑Success Rate ↑
MetaDriveRandom Re-evaluation5334.80 ± 25.10.82 ± 0.04
Lazy Re-evaluation (Ours)5360.50 ± 2.50.92 ± 0.01
Full Re-evaluation10362.46 ± 7.70.93 ± 0.01
MetaWorldRandom Re-evaluation52546.69 ± 83.80.88 ± 0.03
Lazy Re-evaluation (Ours)52592.98 ± 183.30.92 ± 0.04
Full Re-evaluation102595.97 ± 211.60.93 ± 0.03

At the same evaluation budget, Lazy Re-evaluation outperforms random refreshing on MetaDrive, improving success from 82% to 92% and episodic return from 334.80 to 360.50. On MetaWorld, it also achieves higher mean success (92% vs. 88%) and episodic return (2592.98 vs. 2546.69).

We agree that full re-evaluation achieves slightly better performance. However, Lazy Re-evaluation achieves nearly identical results while evaluating only half as many scenarios per update. These findings suggest that targeted re-evaluation offers a favorable trade-off between task performance and evaluation overhead. We have added the new baseline to Table 2 and revised Section 5.3 accordingly.

[W2] Methodological Contribution and Coupling

We agree that learning velocity, Exp3, and NeuralUCB are established techniques. Our contribution lies in addressing a distinct challenge in online IIL: as the learner improves, both the value of training on each scenario and the supervision required for that scenario continuously change. COACH brings these two evolving decisions into a unified adaptive framework. We use Exp3 and NeuralUCB to handle their different feedback structures, rather than claiming novelty in the individual bandit algorithms.

Importantly, the coupling goes beyond passing the selected scenario as context. The Curriculum Bandit prioritizes scenarios with significant learning dynamics, aiming to improve the learner through informative expert interventions. These interventions update the learner's policy, which in turn changes its proficiency influencing future decisions of both bandits. As the learner becomes more proficient, task success may improve and lower-cost experts may become sufficient, allowing the Supervision Bandit to adapt the performance–cost trade-off accordingly. While expert cost is not directly included in the curriculum reward, it is explicitly considered during expert allocation. Thus, the two decisions are connected through the evolving learner, rather than through a shared reward function.

This design also avoids independently exploring every scenario–expert pair, which can become costly under limited interaction budgets. Our comparison with Joint MAB (Table 3) illustrates this trade-off: while Joint MAB performs better in the small 3×2 setting, COACH achieves higher success as the decision space grows (84% vs. 56% at 50×10). We clarified this motivation and the feedback mechanism in the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.