- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes a method using one action-free mentor expert trajectory to guide the RL process. The paper proposes mentor baseline implicit imitation learning, formulating a bridge between expert state-only demo and the actual observer trajectories with a defined neighborhood. The paper then derives a baseline injection technique for downstream RL. The paper provides theoretical analysis on the proposed method and evaluates the proposed method against imitation learning and RL baselines in MuJoCo environments, demonstrating superior performance in most setups.
Critical strengths
-
Good presentation and strong motivation: The paper's presentation is overall understandable. Figure 1 in the paper describes adequately of the motivation and the key insights of the proposed method.
-
Theoretical analysis: The paper proposes a theoretical analysis on the baseline computed in their method in Theorem 1 and Appendix C.2. The analysis is overall solid and demonstrates that the performance loss after baseline injection is bounded by the discounted cumulative amount of .
-
Empirical failure analysis: The paper conducts a failure analysis in Section 5.3, to examine how the neighborhood definition affects failures of MB-I2L. The results demonstrate that without mentor-observer correspondence given and only one single mentor trajectory, the algorithm may learn wrong correspondence, which may lead to wrong direction of baseline injection. Although this is a limitation, but explicitly analyzing it provides a better understanding to the proposed method.
Critical weaknesses
-
Additional discussions on related works: There is one closely related work [1] on leveraging expert demonstrations to guide reinforcement learning process. The paper should provide additional discussions and also leverage [1] as baseline for the proposed method.
-
More complex tasks: The paper conducts the empirical validations on MuJoCo environments. Evaluations on some harder tasks such as tasks in Robomimic [2] and Meta-World [3] should strengthen the empirical analysis in the paper.
[1] Dong, P., Lessing, A. M., Chen, A. S., & Finn, C. (2025). Reinforcement learning via implicit imitation guidance. arXiv preprint arXiv:2506.07505.
[2] Mandlekar, A., Xu, D., Wong, J., Nasiriany, S., Wang, C., Kulkarni, R., ... & Martín-Martín, R. (2021). What matters in learning from offline human demonstrations for robot manipulation. arXiv preprint arXiv:2108.03298.
[3] Yu, T., Quillen, D., He, Z., Julian, R., Hausman, K., Finn, C., & Levine, S. (2020, May). Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning (pp. 1094-1100). PMLR.
Questions for authors
See weakness section.
2 comments
Thanks for your response. My concerns have mostly been addressed. I'll keep my positive score.
Thank you for your comments and your positive assessment of our work. Below, we address your comments on comparisons with related work and evaluation on more complex tasks.
Relationship to and comparison with DGN (Weakness 1). Data-Guided Noise (DGN) by Dong et al. (2025), which you brought to our attention, is closely related to our work in that it uses demonstrations to support exploration in online RL and learns a policy through RL. We will therefore add a discussion of DGN to the main text. However, the methods differ in the following respects. DGN learns exploration noise from demonstration state–action pairs recorded in the same state and action spaces as those of the learner (Section 4.1, Eq. 1). In our setting, mentor actions are not provided, and the mentor and observer may have different state and action spaces. Therefore, the information required by the original formulation of DGN is not available in our setting, and DGN cannot be directly used as a baseline.
Evaluation on more complex tasks (Weakness 2). We agree that evaluating our method on manipulation tasks in Robomimic and Meta-World would be valuable for further examining its scope of applicability. However, our current experiments already address challenging sparse-reward learning that requires complex control and exploration. In particular, AntMaze requires the observer to simultaneously learn locomotion for an Ant with multiple joints and navigation within the maze. The AntMaze observation consists of the Ant’s own 105-dimensional observation, the two-dimensional center-of-mass position, the two-dimensional goal position, and a one-dimensional step count normalized to [0, 1], yielding a very high-dimensional observation with 110 dimensions in total. We will add this information to Appendix G.2.2. In addition to the challenges above, in our setting, the episode terminates when the Ant falls, so the observer must learn to reach the goal while maintaining its posture. The mentor data consist of a single action-free trajectory from a ball mentor and contain no demonstrations of the Ant’s actions or joint movements. Under the same training budget of 5M environment steps, the normalized AUC of the evaluation success rate was 0.00±0.00 for standard SAC and 0.53±0.054 for MB-I2L (mean ± SD, Table 1).