- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes Stitch-Imitate-Optimize (SIO), a framework for learning a deployable policy from multiple imperfect experts. SIO jointly trains a selector and a student: the selector learns to choose an expert or the student at each state, while the student uses behavior cloning on selected expert actions and reinforcement learning on its own actions. The paper also introduces two variants, AMSIO and ISSIO, to improve robustness. Theoretical analysis and experiments across classic control and MuJoCo environments demonstrate the benefits of combining expert stitching with policy optimization.
Critical strengths
-
Intuitive and practical framework. Learning a state-dependent selector to combine imperfect experts is an appealing idea. The combination of selective behavior cloning and student-side RL also provides a natural way to benefit from expert actions without permanently restricting the student to them.
-
Thorough empirical and theoretical analysis. The experiments cover a wide range of expert qualities, complementary expert constructions, and both discrete and continuous control tasks. I also appreciate the selector ablations and the theoretical treatment of expert stitching and student improvement.
Critical weaknesses
-
Limited direct evidence of effective expert stitching. The complementary experts in Comp1–Comp4 are constructed from a shared strong policy, with random actions outside their designated regions. While the return improvements are convincing, the selection-frequency plots do not clearly establish whether the selector consistently identifies the appropriate expert at each state.
-
The imitation-to-RL transition is sensitive to the student-selection floor. Table 4 shows that ξ=0.5 improves performance with random experts but reduces returns on several good and complementary expert sets. This makes the claimed automatic balance between imitation and RL somewhat dependent on a manually chosen parameter.
-
Gap between theoretical and practical optimization. Theorem 2 assumes a state-dependent BC coefficient tied to the student-selection probability, whereas Algorithm 1 uses a fixed coefficient and an additional selection floor. Although the authors acknowledge the idealized setting, the theoretical guarantees do not directly characterize the implemented algorithm.
Questions for authors
-
Could the authors clarify how accurately the selector identifies the appropriate expert within the designated regions of Comp1–Comp4? Comparing the selector's stitched behavior with the final student policy would also help establish how effectively the student inherits this behavior.
-
Table 4 suggests that the preferred student-selection floor depends strongly on expert quality. How should ξ be chosen when the relative quality of the experts is unknown?
-
Theorem 2 relies on a state-dependent BC coefficient, while the practical implementation fixes β=1. Which aspects of the theoretical argument are expected to carry over to the implemented SIO objective?
-
There has been a significant amount of similar work recently (such as 2602.17038); could the authors provide a comparison with these existing studies?
3 comments
Part 2 of 2
Questions:
Could the authors clarify how accurately the selector identifies the appropriate expert within the designated regions of Comp1–Comp4?
(See our discussion on this in Part 1 of this reply.)
Table 4 suggests that the preferred student-selection floor depends strongly on expert quality. How should be chosen when the relative quality of the experts is unknown?
If you do not know how good the experts are, we suggest using , the value shown in our main experiment tables. Our main experiment tables and Table 4 show this gives generally high returns.
Theorem 2 relies on a state-dependent BC coefficient, while the practical implementation fixes . Which aspects of the theoretical argument are expected to carry over to the implemented SIO objective?
Just to emphasize, Theorem 1 does not require a state-dependent . And, again, we view the theory as a verification of our basic SIO idea, rather than a practical performance guarantee. Our empirical results indeed suggest that works well in practice.
There has been a significant amount of similar work recently (such as 2602.17038); could the authors provide a comparison with these existing studies?
"Experts" in Mixture-of-Experts methods (including that method) are parts of a single neural network, trained along with their analog of our selector, and used at deployment time. In contrast: our "experts" are more abstract (similar to MaxAgg, our experts could be e.g. Python programs); used as static inputs to our algorithm; and unused at deployment time. Investigating the effects of that method's additional temporal consistency might be an interesting direction for future work.
I feel that the authors' response has not addressed most of my concerns. Many of these issues could have been resolved through further experimental validation, but since I do not see the results I was looking for, I will maintain my current score.
Part 1 of 2
Thank you for your review.
Strengths: This reviewer notes our intuitive and practical framework, and thorough empirical and theoretical analysis.
Weaknesses:
A fine-grained empirical analysis would be an interesting direction for future work.
On the theoretical side, Appendix B.2, "Optimal Stitching Counterexample", provides evidence of effective expert stitching (through SIO's selector, under some assumptions). The counterexample shows that, under some assumptions, MaxFollowing and MaxAgg-NoSI correspond to policies with linear regret compared to optimal stitching, but Theorem 1 proves our theoretical SIO's selector is no-regret against optimal stitching of the experts.
We also agree that the return improvements are indeed quite convincing. To emphasize that: among the combinations of four
Compexpert sets and discrete-action vs. continuous-action tasks, Tables 1 and 2 show AMSIO gets higher returns than PPO on all but one of the eight combinations. Moreover, in half of those settings, AMSIO gets over double the returns of PPO. Additional evidence includes the combination of: much lower AMSIO returns on the "random" expert sets, the constructions of theCompexpert sets, and the fact that the AMSIO returns on theCompexpert sets often match the returns on the "great" expert set.We agree this is a weakness to an extent. However, note that:
We agree that our theory does not exactly match our empirical algorithm, though we want to emphasize that Theorem 1 does not require the state-dependent BC coefficient, that some amount of theoretical-empirical gap is not uncommon, and that we think our empirical results are already strong by themselves. As we say in our paper, we view the analysis as a verification of our basic SIO idea, rather than a practical performance guarantee.