acceptodds
Under review as a conference paper at ICLR 2027

Faster Policy Learning from Multiple Experts: Stitch, Imitate, Optimize

Abstract

How can we train a new policy from a set of expert policies, when none of the experts fully solve the task and none are available at deployment? We propose algorithms to learn to stitch the best experts across states, and to further improve upon the stitched experts. We jointly train two policies: a *student*, which will ultimately be deployed, and a *selector*, which at each state selects an expert or the student to act in the environment. The selector is updated with standard reinforcement learning (RL) to maximize returns. The student is updated with standard RL on its own actions, and with behavior cloning (BC) on actions taken by selected experts. Because the selector tends to select experts only if they outperform the student, the BC can automatically stop as the student outperforms the experts. Overall, this simple mechanism allows the student to combine the strengths of multiple experts, learn quickly from them in the early phase, and continue improving beyond them in the later phase, with the selector automatically guiding the phase transition. Empirically, we measure returns on all combinations of eight environments (four classic control environments, four higher-dimensional MuJoCo environments) with up to twelve expert sets. On all expert sets, averaging over the environments, our most robust algorithm's return CIs overlap or exceed both max-aggregation's (Cheng et al., 2020) and max-aggregation's (Liu et al., 2024), exceeding them by up to .

Primary area:
reinforcement learning
Keywords:
reinforcement learning, imitation learning, policy learning from multiple experts, policy improvement, policy optimization, behavior cloning, oracles, teachers, distillation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Yj1pOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Stitch-Imitate-Optimize (SIO), a framework for learning a deployable policy from multiple imperfect experts. SIO jointly trains a selector and a student: the selector learns to choose an expert or the student at each state, while the student uses behavior cloning on selected expert actions and reinforcement learning on its own actions. The paper also introduces two variants, AMSIO and ISSIO, to improve robustness. Theoretical analysis and experiments across classic control and MuJoCo environments demonstrate the benefits of combining expert stitching with policy optimization.

Critical strengths
  1. Intuitive and practical framework. Learning a state-dependent selector to combine imperfect experts is an appealing idea. The combination of selective behavior cloning and student-side RL also provides a natural way to benefit from expert actions without permanently restricting the student to them.

  2. Thorough empirical and theoretical analysis. The experiments cover a wide range of expert qualities, complementary expert constructions, and both discrete and continuous control tasks. I also appreciate the selector ablations and the theoretical treatment of expert stitching and student improvement.

Critical weaknesses
  1. Limited direct evidence of effective expert stitching. The complementary experts in Comp1–Comp4 are constructed from a shared strong policy, with random actions outside their designated regions. While the return improvements are convincing, the selection-frequency plots do not clearly establish whether the selector consistently identifies the appropriate expert at each state.

  2. The imitation-to-RL transition is sensitive to the student-selection floor. Table 4 shows that ξ=0.5 improves performance with random experts but reduces returns on several good and complementary expert sets. This makes the claimed automatic balance between imitation and RL somewhat dependent on a manually chosen parameter.

  3. Gap between theoretical and practical optimization. Theorem 2 assumes a state-dependent BC coefficient tied to the student-selection probability, whereas Algorithm 1 uses a fixed coefficient and an additional selection floor. Although the authors acknowledge the idealized setting, the theoretical guarantees do not directly characterize the implemented algorithm.

Questions for authors
  1. Could the authors clarify how accurately the selector identifies the appropriate expert within the designated regions of Comp1–Comp4? Comparing the selector's stitched behavior with the final student policy would also help establish how effectively the student inherits this behavior.

  2. Table 4 suggests that the preferred student-selection floor depends strongly on expert quality. How should ξ be chosen when the relative quality of the experts is unknown?

  3. Theorem 2 relies on a state-dependent BC coefficient, while the practical implementation fixes β=1. Which aspects of the theoretical argument are expected to carry over to the implemented SIO objective?

  4. There has been a significant amount of similar work recently (such as 2602.17038); could the authors provide a comparison with these existing studies?

3 comments
AuthorsOpenReview ↗

Part 1 of 2

 

Thank you for your review.

 

Strengths: This reviewer notes our intuitive and practical framework, and thorough empirical and theoretical analysis.

 

Weaknesses:

Limited direct evidence of effective expert stitching. The complementary experts in Comp1–Comp4 are constructed from a shared strong policy, with random actions outside their designated regions. While the return improvements are convincing, the selection-frequency plots do not clearly establish whether the selector consistently identifies the appropriate expert at each state.

A fine-grained empirical analysis would be an interesting direction for future work.

On the theoretical side, Appendix B.2, "Optimal Stitching Counterexample", provides evidence of effective expert stitching (through SIO's selector, under some assumptions). The counterexample shows that, under some assumptions, MaxFollowing and MaxAgg-NoSI correspond to policies with linear regret compared to optimal stitching, but Theorem 1 proves our theoretical SIO's selector is no-regret against optimal stitching of the experts.

We also agree that the return improvements are indeed quite convincing. To emphasize that: among the combinations of four Comp expert sets and discrete-action vs. continuous-action tasks, Tables 1 and 2 show AMSIO gets higher returns than PPO on all but one of the eight combinations. Moreover, in half of those settings, AMSIO gets over double the returns of PPO. Additional evidence includes the combination of: much lower AMSIO returns on the "random" expert sets, the constructions of the Comp expert sets, and the fact that the AMSIO returns on the Comp expert sets often match the returns on the "great" expert set.

 

The imitation-to-RL transition is sensitive to the student-selection floor. Table 4 shows that ξ=0.5\xi=0.5 improves performance with random experts but reduces returns on several good and complementary expert sets. This makes the claimed automatic balance between imitation and RL somewhat dependent on a manually chosen parameter.

We agree this is a weakness to an extent. However, note that:

  • With a static ξ=0.5\xi=0.5, averaging over classic control environments and separately over higher-dimensional environments, each considering twelve expert sets, AMSIO's return CIs overlap or exceed both baseline CIs in all 24 settings — exceeding them by up to 5$\times$.
  • Those baselines, MaxAgg-SI and MaxAgg-NoSI, also have an imitation-vs-RL hyperparameter, λMA\lambda_{\text{MA}}.

 

Gap between theoretical and practical optimization. Theorem 2 assumes a state-dependent BC coefficient tied to the student-selection probability, whereas Algorithm 1 uses a fixed coefficient and an additional selection floor. Although the authors acknowledge the idealized setting, the theoretical guarantees do not directly characterize the implemented algorithm.

We agree that our theory does not exactly match our empirical algorithm, though we want to emphasize that Theorem 1 does not require the state-dependent BC coefficient, that some amount of theoretical-empirical gap is not uncommon, and that we think our empirical results are already strong by themselves. As we say in our paper, we view the analysis as a verification of our basic SIO idea, rather than a practical performance guarantee.

AuthorsOpenReview ↗

Part 2 of 2

 

Questions:

Could the authors clarify how accurately the selector identifies the appropriate expert within the designated regions of Comp1–Comp4?

(See our discussion on this in Part 1 of this reply.)

 

Table 4 suggests that the preferred student-selection floor depends strongly on expert quality. How should ξ\xi be chosen when the relative quality of the experts is unknown?

If you do not know how good the experts are, we suggest using ξ=0.5\xi=0.5, the value shown in our main experiment tables. Our main experiment tables and Table 4 show this gives generally high returns.

 

Theorem 2 relies on a state-dependent BC coefficient, while the practical implementation fixes β=1\beta=1. Which aspects of the theoretical argument are expected to carry over to the implemented SIO objective?

Just to emphasize, Theorem 1 does not require a state-dependent β\beta. And, again, we view the theory as a verification of our basic SIO idea, rather than a practical performance guarantee. Our empirical results indeed suggest that β=1\beta=1 works well in practice.

 

There has been a significant amount of similar work recently (such as 2602.17038); could the authors provide a comparison with these existing studies?

"Experts" in Mixture-of-Experts methods (including that method) are parts of a single neural network, trained along with their analog of our selector, and used at deployment time. In contrast: our "experts" are more abstract (similar to MaxAgg, our experts could be e.g. Python programs); used as static inputs to our algorithm; and unused at deployment time. Investigating the effects of that method's additional temporal consistency might be an interesting direction for future work.

I feel that the authors' response has not addressed most of my concerns. Many of these issues could have been resolved through further experimental validation, but since I do not see the results I was looking for, I will maintain my current score.

Reviewer_8BjxOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors propose a new mechanism to train a student agent using multiple suboptimal demonstrators. They introduce a selector, an entity who is tasked to select a demonstrator to provide the imitation behavior to the student. The selector is trained using RL with its action set equal to the set of demonstrators. In one of the algorithm variants, its action set is then also appended with student policy so that it naturally allows student to interact with the environment and learn directly, when it is better than other demonstrators. Paper compares different algorithmic variants on classic control domains and shows that their framework enables faster convergence against baselines. Theoretical analysis talks about the regret of the student establishing O(T)O(\sqrt{T}) upper bound.

Critical strengths
  • The paper highlights an important issue in the previous multi-demonstrator value/action supervision methods. They can lead to constant switching between the demonstrators when executing a behavior. (Probably, the stitching is necessary for interpretability: constant switching between the demonstrator behaviors can seem haphazard, relying on one demonstrator's behavior for moderate number of time-steps in its region of strength can bring sense of safety in the human users.)

  • The paper's writing is pretty strong. It flows well and explains complicated ideas in a common thread. I had a great time reading this work.

Critical weaknesses
  • No clear purpose behind theory: I am not sure how the current regret analysis helps the overall paper because it is done with certain assumptions which might not hold in reality. Not just that, in general it leaves out more important questions: (1) why "stitching" benefits compared to using value supervision that can switch between demonstrators? (2) how does the regret of SIO variants compare against those of the previously established approaches? (3) how does error in selector harm the student's learning?

  • It is not clear that the selector will not fall in the trap of switching between the demonstrators constantly.

Questions for authors
  1. Can you give some empirical analysis on how much "stitching" of consecutive demonstrator actions improves when we use SIO variants as against other baselines?
  2. What are some important applications for this sort of multidemonstrators-selector-learner setup? Can you show some empirical analyses for such simulated but more realistic setups?
  3. Are you counting the selector's interactions in the environment when you represent the x-axis in Figure 2? If not, I think those should also be accounted for. Otherwise, it is like providing very strong learning signal to student with very few interactions of its own.
  4. Please remove the claim of "faster policy learning" in the title unless your theory shows guaranteed faster convergence rate.

Overall, I feel the work is promising, it requires a few more empirical and theoretical analyses to have a stronger, coherent pitch. I am inclined to increase the score if authors can satisfactorily provide these analyses.

3 comments
AuthorsOpenReview ↗

Part 1 of 2

 

Thank you for your review.

 

Strengths: This reviewer notes the strength of our writing, as well as the importance of the theoretical stitching issue we solve (under some assumptions).

 

Weaknesses:

No clear purpose behind theory: I am not sure how the current regret analysis helps the overall paper because it is done with certain assumptions which might not hold in reality.

We think theory can help support a paper even when it does not match every detail of an empirical algorithm. As we say at the beginning of our theory section, we view our theory as additional evidence that the basic ideas are worthwhile.

 

Not just that, in general it leaves out more important questions: (1) why "stitching" benefits compared to using value supervision that can switch between demonstrators?

As our introduction says:

Both approaches admit guarantees of learning a policy at least as good as the best single expert. However, this does not ensure optimal stitching. Specifically, the state value function used in these methods measures the return from continuing with one expert, rather than with switching experts. It can therefore underestimate the usefulness of actions whose benefit relies on subsequent switches to other experts, causing the methods to miss useful combinations of experts (Section B.1).

The most relevant parts of B.1 are the first two, "Approach/Surpass Optimal Stitching?" and "Action/Value Supervision?". The first points to a counterexample where, under some assumptions, MaxAgg-NoSI and MaxFollowing provably fail to learn the optimal policy, whereas SIO's selector provably succeeds. We also provide further, informal arguments there: we argue that MaxAgg-SI still suffers from this suboptimality in its imitation learning component (whereas SIO does not). The second part of B.1 argues that action supervision may be faster early in training (by directly cloning expert actions), while value supervision allows surpassing all the experts, and SIO may benefit from both.

 

(2) how does the regret of SIO variants compare against those of the previously established approaches?

The counterexample shows that, under some assumptions, MaxFollowing and MaxAgg-NoSI correspond to policies with linear regret compared to optimal stitching, but Theorem 1 proves our theoretical SIO's selector is no-regret against optimal stitching of the experts.

The MaxAgg prior works, Cheng et al. (2020) and Liu et al. (2024), give return guarantees based on the maximum expert value function at each state. Liu et al. (2024) also include the student's value function. In contrast, our Theorem 1 regret comparator (for the behavior policy from the selector) is the optimal expert at each state, for maximizing returns, allowing the use of different experts at future states. Our Theorem 2 regret comparator (for the student) is the optimal policy in the original MDP.

 

(3) how does error in selector harm the student's learning?

As we state in the paper, for example, "The student’s RL update controls the cumulative advantage term, with BC contributing at most λT\lambda T to the bound." That is, we bound the harm that the selector can cause to the student by giving bad actions for behavior cloning (BC).

 

It is not clear that the selector will not fall in the trap of switching between the demonstrators constantly.

We train our selector to maximize returns, and in our setting there is no switching cost directly from the environment, so frequent switching is not necessarily an issue. Also note that expert sets similar to the "checkerboard" patterns of Comp3 and Comp4 may require frequent switching for the optimal selector. Investigating temporal consistency approaches similar to the method Reviewer Yj1p cited might be an interesting direction for future work.

Further, we think our experiments provide convincing empirical evidence that our algorithm likely works significantly better than our baselines in practice. And, again, we think our theory provides some additional justification of our basic ideas as well, even though it does not match every detail of the empirical setting.

 

(cont. below)

AuthorsOpenReview ↗

Part 2 of 2

 

Questions:

Can you give some empirical analysis on how much "stitching" of consecutive demonstrator actions improves when we use SIO variants as against other baselines?

We do not measure this directly, but our higher returns give some evidence. See our reply to Reviewer Yj1p for more details. We think further analysis here is an interesting direction for future work.

 

What are some important applications for this sort of multidemonstrators-selector-learner setup? Can you show some empirical analyses for such simulated but more realistic setups?

One potential application is robotics. You might already have specialized controllers for individual tasks, like the grasp vs. place example in our introduction, and want to use those to more quickly learn an improved, general robotics policy. Indeed, large language models (LLMs) are often trained similarly, which is another important potential application. Many standard LLMs, e.g. from DeepSeek, Z.ai (GLM), and Xiaomi, use some form of "multi-expert distillation." Section B.3 elaborates. Our algorithmic contributions might help there, but we leave such extensions for future work. Like the prior works of the main baselines we consider (MaxAgg-NoSI and MaxAgg-SI), we focus on simpler, somewhat synthetic setups, so that we can more deeply understand how our algorithms and our baselines perform with fewer confounding variables.

 

Are you counting the selector's interactions in the environment when you represent the x-axis in Figure 2?

Yes, in all our experiments, we count our selector's interactions against our method.

 

Please remove the claim of "faster policy learning" in the title unless your theory shows guaranteed faster convergence rate.

"Faster Policy Learning" in the title means "faster than baselines". This is an empirical claim based on e.g. AMSIO's return CIs (averaged over training steps) overlapping or exceeding both MaxAgg-NoSI's and MaxAgg-SI's on all expert sets — exceeding them by up to 5×5\times.

 

Thanks for your response. The core of my concerns remains unanswered. I will have to unfortunately stick to my earlier rating. I did have fun reading the paper though and I believe that with some adjustments to the theory and a slightly stronger empirical analysis this paper will add valuable insights for all of us in the community.

Reviewer_ZSRDOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper is on imitation learning from multiple experts, with the aim to "stitch" together the relevant expertise of the different experts on different segments of the task while continuing to perform standard reinforcement learning. The paper proposes an approach, some theoretical results, and experimental results on four classic control tasks (CartPole, Acrobat, MountainCar, and Pendulum) and some MuJoCo environments.

Critical strengths

The paper builds on a series of papers on imitation learning from multiple experts in the reinforcement learning setting. It incorporates key elements from this earlier work and innovates by introducing an additional layer of control, a meta-agent that selects among the various experts.

Critical weaknesses

The empirical evaluation is quite narrow. The experimental evaluation on the four classic control tasks considers only two levels of expertise: none (random behaviour) and high (near optimal policy). These two levels of expertise are combined to create various different "experts", for example, one that behaves near optimally in some regions and randomly in others. This experimental scheme introduces different "experts" with different regions of great performance. A wider investigation would be needed to test the approach more throughly, one that includes expertise at varying levels of performance. Finally, the "experts" are created such that, at every state, there is a single great expert and one or more other experts that behave randomly. Again, this is a rather restricted setting. It would be informative to include experts with varying different levels of expertise in various different (and overlapping) regions of the state space. (I see that Appendix J.2.1 refers to a "mediocre" discrete-action expert but I see no reference to this in the main paper, including in the results presented; with Table 8 in the Appendix including limited results with two "mediocre" experts, and no learning curves). The results on higher-dimensional, continuous-action environments are explained and discussed very briefly in the main text, with results presented in the Appendix, pages 19-22, in tables. It would be useful to see learning curves in these domains. Overall, I do not see a robust evaluation of the proposed approach.

The writing is often poor and careless. For instance, it is not specified anywhere in the paper which domains were used to obtain the results in Tables 2 and 3. Pseudocode for the proposed/analysed algorithms have been included; however, the paper also needs a clear exposition and discussion of the main ingredients of the approach in the main text. As one example, in section 3, on line 206, the paper mentions a "meta-MDP", with no further explanation of definition. Some claims and explanations in the paper are unwarranted. For instance, on line 228, the authors state, "SIO can automatically use BC only when it is increasing returns". This is too strong a claim and is not supported by evidence.

Questions for authors

I will not pose any questions for the authors, as I believe the paper needs substantial additional experimentation and improved exposition.

1 comment

 

Thank you for your review.

 

Strengths: This reviewer emphasizes the novelty of our selector compared to our baselines.

 

Weaknesses:

The experimental evaluation on the four classic control tasks considers only two levels of expertise: none (random behaviour) and high (near optimal policy).

On both the four classic control tasks and the four higher-dimensional environments, we include random experts, near-optimal experts, two different "mediocre" experts, and as you later mention, many experts that each individually mix random and near-optimal expertise in various ways.

The number of expert policies for every environment (across all expert sets) is 25: 8 random (different seeds), 1 great, 2 mediocre, 2 for each of Comp1-Comp3, and 8 for Comp4.

 

The "experts" are created such that, at every state, there is a single great expert and one or more other experts that behave randomly. Again, this is a rather restricted setting. It would be informative to include experts with varying levels of expertise in various different (and overlapping) regions of the state space. (I see that Appendix J.2.1 refers to a "mediocre" discrete-action expert but I see no reference to this in the main paper, including in the results presented; with Table 8 in the Appendix including limited results with two "mediocre" experts, and no learning curves).

As you point out, our "mediocre" expert sets M and 2M indeed test outside of that setting. The Table 2 caption in the main paper pointed to the appendix for M and 2M, but we've now made this clearer. Our R, 2R, and 8R expert sets also test outside of that setting, in that there is no great expert anywhere. Our default learning rate experiments (e.g. Table 2, Table 8) are on all of those expert sets, for both our discrete-action environments and our higher-dimensional, continuous-action environments. Our learning rate sweep experiments (Table 1) omit the "mediocre" experts.

 

It is not specified anywhere in the paper which domains were used to obtain the results in Tables 2 and 3.

Thanks, we've made these clearer. (For Table 2, it was specified in the paper, but only in the appendix. For Table 3, "normalized by DQN" implied the discrete-action environments, but that was quite indirect.)

 

The paper also needs a clear exposition and discussion of the main ingredients of the approach in the main text. As one example, in section 3, on line 206, the paper mentions a "meta-MDP", with no further explanation of definition.

The "meta-MDP where experts are actions" means everything is the same as the original MDP, except that instead of the original action space, each action chooses an expert policy to act in the original MDP's action space. We've now clarified that sentence in the paper. Could you point out anything else in particular that remains unclear?

 

Some claims and explanations in the paper are unwarranted. For instance, on line 228, the authors state, "SIO can automatically use BC only when it is increasing returns". This is too strong a claim and is not supported by evidence.

Our paper said, in our "Intuition" paragraph: "unlike approaches that naively train with BC and RL together, SIO can automatically use BC only when it is increasing returns". The next sentence introduces a "simplified view" of two phases for this. We then describe phase two as: "once the selector has learned to stitch the experts and the student has cloned this behavior, the student will be selected with higher probability". We've changed "can" to "attempts to" to make this slightly clearer.

Figure 1 and our ten expert-selection figures in the appendix provide some evidence for this. For example, Figure 1 shows that SIO quickly avoids BC of random experts, and Figure 27 shows that SIO tends to use more BC for a "great" expert, compared to e.g. the random experts in Figures 1 and 24. Could you suggest other claims or explanations that might be unwarranted?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.