acceptodds
Under review as a conference paper at ICLR 2027

Mentor-Baseline Implicit Imitation Learning from a Single Action-Free Mentor Trajectory

Abstract

In online reinforcement learning (RL), we investigate whether a learning agent (observer) can benefit from experience acquired by another agent (mentor) in addition to its own interaction with the environment. The mentor need not be an expert, the two agents may have different state and action spaces, and the observer is given only a single action-free mentor trajectory of states and rewards. In this setting, we propose Mentor-Baseline Implicit Imitation Learning (\mbiil), which uses the mentor's return-to-go and approximate mentor–observer state correspondence to form state baselines and propagates them to guide the observer's action-value learning. We further formalize mentor-derived baseline optimism, which can drive exploration but can also bias value learning, and bound the resulting policy performance loss. In sparse-reward environments, we evaluate \mbiil using a simple ball as the mentor and observers with substantially different embodiment and control requirements, including a high-degree-of-freedom Ant for maze navigation and a Hopper for obstacle crossing. Our results show that a single action-free mentor trajectory can facilitate learning across substantial embodiment differences, with \mbiil being the only compared method to learn to reach the goal in AntMaze. They also show that the observer can achieve a higher return than the mentor, while mentor–observer correspondence too coarse to distinguish states of different values can induce overly optimistic baselines and impede learning, consistent with our theoretical analysis.

Primary area:
reinforcement learning
Keywords:
Implicit Imitation Learning, Reinforcement Learning, Action-Free Demonstrations, Cross-Embodiment Learning, Sparse-Reward

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_oESiOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a method using one action-free mentor expert trajectory to guide the RL process. The paper proposes mentor baseline implicit imitation learning, formulating a bridge between expert state-only demo and the actual observer trajectories with a defined neighborhood. The paper then derives a baseline injection technique for downstream RL. The paper provides theoretical analysis on the proposed method and evaluates the proposed method against imitation learning and RL baselines in MuJoCo environments, demonstrating superior performance in most setups.

Critical strengths
  • Good presentation and strong motivation: The paper's presentation is overall understandable. Figure 1 in the paper describes adequately of the motivation and the key insights of the proposed method.

  • Theoretical analysis: The paper proposes a theoretical analysis on the baseline computed in their method in Theorem 1 and Appendix C.2. The analysis is overall solid and demonstrates that the performance loss after baseline injection is bounded by the discounted cumulative amount of Γ\Gamma.

  • Empirical failure analysis: The paper conducts a failure analysis in Section 5.3, to examine how the neighborhood definition affects failures of MB-I2L. The results demonstrate that without mentor-observer correspondence given and only one single mentor trajectory, the algorithm may learn wrong correspondence, which may lead to wrong direction of baseline injection. Although this is a limitation, but explicitly analyzing it provides a better understanding to the proposed method.

Critical weaknesses
  • Additional discussions on related works: There is one closely related work [1] on leveraging expert demonstrations to guide reinforcement learning process. The paper should provide additional discussions and also leverage [1] as baseline for the proposed method.

  • More complex tasks: The paper conducts the empirical validations on MuJoCo environments. Evaluations on some harder tasks such as tasks in Robomimic [2] and Meta-World [3] should strengthen the empirical analysis in the paper.

[1] Dong, P., Lessing, A. M., Chen, A. S., & Finn, C. (2025). Reinforcement learning via implicit imitation guidance. arXiv preprint arXiv:2506.07505.

[2] Mandlekar, A., Xu, D., Wong, J., Nasiriany, S., Wang, C., Kulkarni, R., ... & Martín-Martín, R. (2021). What matters in learning from offline human demonstrations for robot manipulation. arXiv preprint arXiv:2108.03298.

[3] Yu, T., Quillen, D., He, Z., Julian, R., Hausman, K., Finn, C., & Levine, S. (2020, May). Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning (pp. 1094-1100). PMLR.

Questions for authors

See weakness section.

2 comments

Thank you for your comments and your positive assessment of our work. Below, we address your comments on comparisons with related work and evaluation on more complex tasks.

Relationship to and comparison with DGN (Weakness 1). Data-Guided Noise (DGN) by Dong et al. (2025), which you brought to our attention, is closely related to our work in that it uses demonstrations to support exploration in online RL and learns a policy through RL. We will therefore add a discussion of DGN to the main text. However, the methods differ in the following respects. DGN learns exploration noise from demonstration state–action pairs recorded in the same state and action spaces as those of the learner (Section 4.1, Eq. 1). In our setting, mentor actions are not provided, and the mentor and observer may have different state and action spaces. Therefore, the information required by the original formulation of DGN is not available in our setting, and DGN cannot be directly used as a baseline.

Evaluation on more complex tasks (Weakness 2). We agree that evaluating our method on manipulation tasks in Robomimic and Meta-World would be valuable for further examining its scope of applicability. However, our current experiments already address challenging sparse-reward learning that requires complex control and exploration. In particular, AntMaze requires the observer to simultaneously learn locomotion for an Ant with multiple joints and navigation within the maze. The AntMaze observation consists of the Ant’s own 105-dimensional observation, the two-dimensional center-of-mass position, the two-dimensional goal position, and a one-dimensional step count normalized to [0, 1], yielding a very high-dimensional observation with 110 dimensions in total. We will add this information to Appendix G.2.2. In addition to the challenges above, in our setting, the episode terminates when the Ant falls, so the observer must learn to reach the goal while maintaining its posture. The mentor data consist of a single action-free trajectory from a ball mentor and contain no demonstrations of the Ant’s actions or joint movements. Under the same training budget of 5M environment steps, the normalized AUC of the evaluation success rate was 0.00±0.000.00 \pm 0.00 for standard SAC and 0.53±0.0540.53 \pm 0.054 for MB-I2L (mean ±\pm SD, Table 1).

Thanks for your response. My concerns have mostly been addressed. I'll keep my positive score.

Reviewer_4gyEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Mentor-Baseline Implicit Imitation Learning (MB-I2L), which uses a single action-free mentor trajectory to guide online reinforcement learning when the mentor and learner may have different state and action spaces. It matches states through predefined shared features and transfers the mentor’s return-to-go through bridges formed by the learner’s own experience to construct additional critic targets. The method is implemented with SAC, alongside an analysis of performance loss from optimistic mentor-derived values. Experiments use a ball mentor in PointMaze, AntMaze, and two HopperHurdle variants. The reported results are positive in PointMaze, AntMaze, and HopperHurdle-Short, while the default method largely fails in HopperHurdle-Long.

Critical strengths
  • The paper addresses an interesting question of whether a learner can benefit from a single action-free demonstration produced by an agent with different capabilities. This is a worthwhile setting to study when demonstrations from the learner’s own embodiment are difficult to obtain.

  • The theoretical analysis makes the potential mismatch between mentor-derived returns and the learner’s achievable value explicit. It relates this mismatch to possible performance loss in an idealized setting, providing a formal description of an important limitation of the approach.

Critical weaknesses
  • The evidence for transfer across embodiments is limited to a small set of controlled tasks with directly specified coordinate correspondence. PointMaze uses a ball for both mentor and learner, leaving AntMaze and the two Hopper variants as the cross-embodiment evaluations. All demonstrations come from the same type of mentor, and the shared features are position or position combined with the sign of vertical velocity. The authors acknowledge the need for prespecified correspondence in their setting. My concern is that the experiments do not establish how useful the method remains beyond these simple coordinate-based matches and limited task families.

  • The default method largely fails on HopperHurdle-Long. Table 1 reports a normalized AUC of 0.005, compared with 0.25 for count-based exploration and 0.13 for PWIL. Although this remains a single-hurdle task, the longer mentor trajectory includes an additional bounce before the hurdle and places greater demands on repeated movement. The failure categories show that the learner commonly jumps without clearing the hurdle, but do not sufficiently isolate the proposed posture-related explanation. Adaptively disabling baseline injection partially improves performance, with the aggressive setting reaching 0.29 ± 0.31 while causing AntMaze to fail completely. The recovered mean is close to count-based exploration at 0.25 ± 0.28, with substantial variability. These results do not establish a robust remedy for the failure.

  • Completing a bridge establishes that the learner reached its endpoint, but does not establish that the mentor’s remaining return is achievable from the learner’s posture, velocity, and joint configuration there. Clipping limits individual target increases, and the variant in Appendix G.8 adaptively disables injection using bridge counts and increment sizes. These mechanisms can reduce the effect of excessive optimism, but neither directly assesses the compatibility of the borrowed return with the learner’s full endpoint state. This leaves an important source of misleading credit insufficiently controlled.

  • The contribution relative to closely related methods is not sufficiently established. Section 4 identifies different state spaces as a distinction from prior implicit imitation, but the comparison with Chrysomallis et al. (2025), which introduces HA-DIIQN, remains brief. That work already includes feasibility assessment and bridges connecting learner capabilities to mentor trajectories. MCAC also uses return-to-go estimates to raise ordinary critic targets. The PBRS comparison, which uses the same state baseline, and the bridge ablation provide useful component comparisons. However, they do not resolve what is added relative to these closely related approaches. A more precise comparison of assumptions and mechanisms, together with an explanation of which prior methods can reasonably be adapted, is needed.

  • The practical usefulness of the performance bound is not established. The main bound depends on the learner’s unknown optimal value. Corollary 2 provides a more conservative reference-policy alternative whose state values can, in principle, be estimated from rollouts, but neither version is evaluated experimentally. For the goal-only reward setting, returns lie between 0 and 1. With the experimental discount of 0.99, the bound already allows a loss of 1 when its occupancy-averaged positive propagated gap reaches 0.01. A quantitative evaluation in a controlled setting would help establish whether the bound meaningfully restricts performance loss or mainly provides a qualitative description of the risk.

  • The method’s dependence on discovering useful bridges is insufficiently characterized, particularly in AntMaze and HopperHurdle. Figures 6 and 8 provide information about positive target increases and their magnitude in PointMaze, but do not establish how quickly useful bridges become available in the harder environments. Reporting the time to obtain useful bridges, their lengths, and the fraction of experience receiving baseline injection would help explain when the method improves learning and when bridge discovery becomes a bottleneck.

  • The zero-performing baselines require stronger validation. Every reported competitor has zero normalized AUC in AntMaze, while HER and PBRS score zero in all four task settings, including PointMaze. PBRS also uses the same mentor-derived state baseline as the proposed method. These outcomes do not establish an implementation problem, but they make implementation checks, reward scaling, and tuning coverage particularly important. Appendix G.1 reports a small parameter search for ORCA on PointMaze, but does not report comparable tuning studies for the other baselines. Stronger evidence that the comparison methods are appropriately configured is needed to assess the reported advantage.

  • The presentation makes the central mechanism unnecessarily difficult to reconstruct. Figure 5 in Appendix E illustrates a three-step bridge and its sampled baseline, so an example is present. However, it remains symbolic and does not connect a concrete mentor state and full learner state to the matching features and resulting critic target. Such an example would make clearer why reaching a matched endpoint does not establish that the borrowed continuation is achievable. Section 3.2 also contains an unresolved “Appendix ??” reference attached to the claim about faster mentors.

  • Computational cost is reported through environment interaction budgets without hardware specifications or wall-clock measurements. A brief comparison of training time and overhead relative to SAC would make the practical cost clearer.

Questions for authors
  • What explains the HopperHurdle-Long failure, and why does aggressive adaptive disabling help there but cause AntMaze to fail?

  • What is the specific advance over Chrysomallis et al. (2025) and MCAC, and what comparisons support it?

  • How were baseline implementations and tuning validated, given the uniformly zero AntMaze results and HER/PBRS failures across all tasks?

  • Can either performance bound be evaluated in a controlled setting to demonstrate a nontrivial restriction on performance loss?

4 comments

Thank you for your comments on our work. We address your questions below.

Analysis of failure causes (Q1). In HopperHurdle-Long, the observer needs to land before the hurdle and jump again, requiring body-specific movements that are not demonstrated by the ball mentor (Sec. 5.2). Fig. 11 (in the current version) likewise shows that the predominant failure type is Hop, namely, failure after takeoff and before clearing the hurdle. Meanwhile, because ψ\psi does not include posture or joint state, completing a bridge does not necessarily mean that the observer can realize the mentor's remaining return from the bridge endpoint. Since a positive baseline can be assigned even to states from which the goal is unreachable (corresponding to kO=∞k_O=\infty in the theoretical analysis in Sec. 3.2), we believe that this excessive optimism hinders the observer's learning of body control. This interpretation is supported by interventions on the strength and duration of baseline injection. Table 7 shows that relaxing the cap on the baseline-induced increment per critic update degrades performance in HopperHurdle-Short, while Table 10 shows that MB-I2L$^{\prime}$, which adaptively disables injection, improves performance in both Short and Long. However, the result of 0.29±0.310.29\pm0.31 for (10,10)(10,10) on Long is comparable to CE's 0.25±0.280.25\pm0.28 and exhibits substantial variability. We therefore position MB-I2L$^{\prime}$ as a diagnostic intervention for investigating how the duration of injection affects failure, rather than as a robust solution. In AntMaze, Timeout accounts for a larger proportion of failures under (10,10)(10,10) than under Default and (50,50)(50,50) (Fig. 11), and success was also rarely observed during training rollouts. Disabling injection involves a trade-off between the benefit of suppressing optimism and the loss of exploration guidance. In AntMaze, with its large state space, losing guidance before sufficient learning has taken place makes it difficult to acquire and maintain behavior that follows the mentor's path. In HopperHurdle, by contrast, our interpretation is that the observer can more easily return to the mentor's path through its own experience after injection is disabled, so the benefits of suppressing optimism outweigh the loss of guidance (App. G.9).

Relationship to HA-DIIQN and MCAC (Q2). HA-DIIQN extends the DQN-based DIIQN to heterogeneous action settings and assumes that the mentor and observer share the same state space. Mentor states are also fed into the critic, and mentor rewards are not used. Among our tasks, it can be applied to PointMaze, where the state space is shared, by discretizing the actions. In contrast, MB-I2L can also handle different state spaces. It defines shared features and neighborhoods to relate states across these spaces and injects the mentor's return-to-go into the critic target through the observer's own trajectory segments, which serve as bridges. Furthermore, HA-DIIQN performs a 1-step TD update using the first feasible transition of a bridge, whereas MB-I2L adjusts the critic target as described above. MCAC is formally similar to MB-I2L in that it raises the critic target by taking the maximum of the TD target and the MC return. However, its MC returns are computed from the observer's own trajectories and demonstrations with actions from the same MDP. Consequently, it cannot use the mentor data provided in the standard MB-I2L setting, namely, action-free data from a different MDP. Furthermore, in our setting, the MC return remains 00 until the first success and therefore provides no information about the goal. MB-I2L uses the mentor's return-to-go from a different MDP to provide a value signal even before the observer has ever reached the goal. However, this value is not necessarily achievable by the observer. Theorem 1 addresses the performance loss caused by this optimism. We evaluated MB-I2L's ability to use mentor information across different bodies and state spaces in AntMaze and HopperHurdle-Short. We also examined the contribution of propagation along bridges in Appendix G.5.1.

Comparison methods (Q3). For HER and PBRS, we identified and corrected implementation issues and reran the experiments. Please see our general response to all reviewers. The other comparison methods (CE, CE$^{\prime}$, PWIL, and ORCA) use the same SAC implementation, features, and distance computations as MB-I2L, and exhibit nonzero performance on at least one of our PointMaze and HopperHurdle tasks. We therefore believe that their uniformly zero performance on AntMaze is attributable to the difficulty of the task, with its 5,000-step episode horizon, termination upon falling, and reward provided only upon reaching the goal, rather than to implementation issues. As described in App. G.1, λ\lambda was set so that the total contribution of intrinsic rewards would be comparable in scale to the extrinsic reward. We also confirmed that the comparison methods obtained almost no extrinsic reward even during training rollouts in AntMaze. Specifically, among these methods, successes during training rollouts were observed only for PWIL and ORCA. PWIL succeeded in only two episodes in total across ten seeds, each trained for 5M steps, while ORCA succeeded in only one episode in total across five seeds, each trained for 5M steps. Consequently, the reward is almost entirely λrint\lambda r^{\mathrm{int}}. Our SAC implementation automatically adjusts the entropy coefficient α\alpha for all methods, and we also confirmed that policy entropy was close to its target value by approximately 50,000 steps. In the entropy-constrained formulation underlying this automatic adjustment, multiplying the entire reward by a positive constant does not change the optimal policy. Thus, before extrinsic rewards are obtained, λ\lambda has little effect on the policy apart from transient learning dynamics, and we consider it unlikely that the choice of λ\lambda is responsible for the uniformly zero performance on AntMaze. Once extrinsic rewards begin to be obtained, λ\lambda determines the relative weighting of intrinsic and extrinsic rewards.

Nontrivial bound (Q4). In our goal-only reward setting, a reward of 11 is received only once, upon success, so the trivial upper bound on policy loss is 11. We have added a finite MDP example in which the performance-loss bound is nontrivial even for γ=0.99\gamma=0.99 to Appendix C.4 of the revised manuscript; please refer to that appendix for details. In this MDP, the mentor succeeds with certainty in two steps, while the observer has a three-step path that succeeds with certainty and a two-step path that succeeds with probability pp. The mentor-derived baseline leads the greedy policy to select the latter. For p=0.98p=0.98, the actual policy loss is 0.00990.0099, and the bound in Theorem 1 is 0.01980.0198, well below the trivial upper bound of 11.

We report the results of additional experiments, analyses, and timing measurements.

Bridge statistics (Weakness 6). We added three measures of bridges during training to Fig. 8 of the revised manuscript (Appendix G.5.1). (i) Mentor Progress is the largest mentor index reached at the endpoint of any bridge completed in a given training episode, normalized by the final mentor index. If no bridge was completed, we use the mentor index at the bridge start point. We exclude episodes in which the observer never entered a mentor neighborhood. (ii) Pushed Fraction is the fraction of transitions sampled for critic updates whose critic targets were raised by baseline injection (Lt>0L_t > 0). (iii) Steps to Endpoint is the mean number of steps from a sampled transition to the bridge endpoint used to compute its baseline, averaged over sampled transitions that have a bridge endpoint.

Because it is difficult to determine directly whether an individual bridge contributed to later success, we examined when bridges reached the midpoint of the mentor trajectory. In HopperHurdle-Short and AntMaze, the 100-episode moving average of Mentor Progress first reached 0.5 at 0.17M and 0.38M environment steps, respectively (medians across 10 seeds). In HopperHurdle-Short, Pushed Fraction reached about 40% early in training and remained roughly constant thereafter. In AntMaze, it increased to about 60% as training progressed. For each seed, we averaged Steps to Endpoint over the entire training run. The median of these per-seed means was 25 steps in Short, 43 in AntMaze, and 9 in Long. The median for Long was computed across all 10 seeds and primarily reflects the six seeds in which progress stalled near the start.

In the original HopperHurdle-Long environment, Mentor Progress stalled near the start in 6 of the 10 seeds and before the hurdle in the remaining four. In both groups, Pushed Fraction reached roughly 30% or more within the first 0.2M environment steps. Thus, baseline injection was already raising critic targets early in training in Long, even though Mentor Progress remained stalled at these positions. This indicates that the stagnation in Long cannot be explained by an absence of baseline injection; we discuss possible causes below.

Failure in HopperHurdle-Long (follow-up to Q1). In our earlier response, we reported that the most common failure in the original Long environment was Hop, in which the episode ended after the Hopper’s foot left the ground but before it cleared the hurdle (Fig. 11(d) of the revised manuscript). Distinguishing the two positions at which progress stalled reveals a difference that this classification alone did not capture.

  • Six seeds in which progress stalled near the start. In videos from all six seeds, the Hopper extended its leg to raise its body immediately after the episode began, without first crouching. This movement allowed it to reach a neighborhood along the mentor’s path quickly, but it did not jump high enough, lost its balance, and failed near the start. Because its foot left the ground, this failure was also classified as Hop.
  • Four seeds in which progress stalled before the hurdle. The bridges reached the position where the mentor made ground contact immediately before bouncing over the hurdle (mentor index 80, Mentor Progress approximately 0.33), but did not extend to the subsequent ascent. This pattern is consistent with the possibility that the observer had difficulty timing its next jump from this position to form a bridge matching the mentor’s subsequent ascent.

(Continued in the next comment.)

Based on these results, we hypothesize that baseline injection changed the relative priority of actions in the original Long environment and that the observer settled into a local optimum, repeatedly selecting actions that did not necessarily lead to task success. For example, when the mentor returns-to-go at two endpoints are similar, reaching an endpoint in fewer observer steps discounts the corresponding return-to-go less and therefore yields a larger baseline. The baseline may therefore have favored reaching a later mentor neighborhood quickly in a single large jump over landing and adjusting the takeoff position for the next jump. In videos of training episodes in the original Long environment, we also observed the Hopper’s center of mass entering the forbidden region after such a large jump.

We also ran additional experiments with a physical hurdle that could collide with any part of the Hopper’s body (see our response to Reviewer WPKD for the setup). In the original Long environment, mean normalized AUC was 0.0050 for MB-I2L and 0.25 for CE. In the physical-hurdle variant of Long, the corresponding values were 0.70 ± 0.40 and 0.62 ± 0.24 (mean ± SD across five seeds). The increase in mean AUC was larger for MB-I2L than for CE. MB-I2L also had the highest mean AUC among the methods evaluated with the physical hurdle in Long, although its advantage over CE was small relative to the variability across seeds. In the original setting, the episode ends if the Hopper’s center of mass enters the forbidden region during a large jump or if its momentum carries it into the region after landing. With a physical hurdle, the Hopper can continue moving after contact. Similar jumps may therefore be less likely to lead directly to failure, which may partly explain the difference in performance between the two settings. In the one seed in which MB-I2L failed to learn to reach the goal with the physical hurdle, we observed the same leg extension immediately after the episode began as in the original Long environment. Appendix G.8 describes the setup, and Table 9 reports the results for all methods evaluated in the physical-hurdle variants.

Range of tasks and mentors (Weakness 1). In addition to the physical-hurdle experiments, we varied the ball mentor’s bounce height in HopperHurdle-Short so that its trajectory passed through positions that were harder for the observer to reach. We kept the observer, task, shared features, and neighborhood definition fixed. Let Δz\Delta z denote the height of the ball’s center above the top of the hurdle at the apex of its bounce. At Δz=0.2\Delta z = 0.2, 0.40.4 (default), and 0.60.6, the mean normalized AUC of MB-I2L was 0.68, 0.63, and 0.42, respectively. Mean AUC decreased as bounce height increased, but at all three heights it exceeded that of both SAC (0.00) and CE (0.077), neither of which uses a mentor (Appendix G.7.2, Table 8, and Table 1). In the PointMaze speed experiment, we varied the mentor’s maximum speed while keeping its body shape fixed. In the fastest setting, the mentor’s maximum speed exceeded the observer’s maximum attainable speed. We clarified in Appendix G.7.1 that this experiment examines differences in movement capability. Together, these experiments broaden the range of hurdle contact conditions and mentor speeds and bounce heights while retaining a correspondence based on shared coordinates.

Correction to the section reference (Weakness 8). In Section 3.2, we corrected the reference to the PointMaze experiment with faster mentors. The results are shown in Fig. 10(a) in Appendix G.7.1.

Computational cost (Weakness 9). We added the wall-clock training time for one million environment steps in PointMaze to Appendix G.11. With evaluation rollouts and heatmap output disabled, we ran SAC and MB-I2L concurrently, using one seed per method, on the same machine equipped with an Intel Core i7-12700K CPU, an NVIDIA RTX A4000 GPU, and 64 GB of RAM. Under these conditions, training took approximately 11,000 seconds (3.1 hours) for SAC and 19,900 seconds (5.5 hours) for MB-I2L. MB-I2L thus took about 1.8 times as long as SAC. Its additional computations include searching for mentor neighborhoods, tracking bridges, and computing baselines.

Reviewer_WPKDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Mentor-Baseline Implicit Imitation Learning (MB-I2L), which uses a single action-free mentor trajectory containing states and rewards to guide an observer’s online reinforcement learning. The mentor and observer may have different embodiments, state spaces, and action spaces. Using predefined shared features, the method associates observer states with mentor states and transfers mentor return-to-go through “bridges” formed by the observer’s own experience. These transferred values increase critic targets when they exceed standard RL targets. The paper analyzes the potential performance loss caused by optimistic baselines and evaluates the method on PointMaze, AntMaze, and two HopperHurdle variants.

Critical strengths
  1. An interesting mechanism for transferring task-level information across embodiments. Rather than recovering or reproducing mentor actions, MB-I2L combines observer-executed trajectory segments with mentor-derived continuation values. This allows the observer to learn its own control strategy and potentially outperform the mentor. The bridge mechanism provides a concrete way to exploit a successful trajectory despite incompatible action spaces.

  2. Evidence of useful exploration guidance, together with analysis of its limitations. In AntMaze, MB-I2L is the only compared method that learns to reach the goal from a ball mentor’s trajectory. Ablations support the contribution of bridge-based propagation. The paper also explicitly examines how inappropriate state correspondences can create harmful optimism, connecting its theoretical analysis with empirical failures.

Critical weaknesses
  1. The practical scope of cross-embodiment value transfer remains unclear. The method relies on predefined shared features to associate states across embodiments. However, similar shared features need not imply similar continuation values when omitted body-specific variables determine feasibility. For obstacle crossing, posture, joint velocities, body dimensions, and contact configuration can determine whether an agent can clear an obstacle and land without falling. The concern is therefore that the method may be useful primarily for tasks in which these embodiment-dependent differences have limited influence on the transferred value, rather than broadly supporting cross-embodiment learning. The theory characterizes the consequences of value mismatch but does not provide a mechanism that reliably resolves it.

  2. The experiments provide limited validation under physically consequential embodiment differences. In HopperHurdle, the obstacle is implemented as a forbidden region for the agent’s center of mass, without collision geometry interacting with the rest of the body. Consequently, the experiment does not test whole-body obstacle clearance, a central difficulty in realistic cross-embodiment transfer. The default method also largely fails on HopperHurdle-Long. Although the AntMaze result is promising, the current evidence does not sufficiently establish that the approach remains beneficial when embodiment differences determine which demonstrated progress is physically achievable.

Questions for authors
  1. Scope of applicability. What conditions on the shared features and mentor–observer dynamics make transferred continuation values informative? In particular, how can a practitioner assess whether omitted body-specific variables will cause harmful value overestimation before committing to extensive observer training?

  2. Physically realistic cross-embodiment evaluation. Have you evaluated MB-I2L on obstacle-crossing tasks with whole-body collision geometry and fall termination? Such results would materially clarify whether the benefits persist when body dimensions, posture, and contact configuration determine the feasibility of following the mentor’s progress.

  3. Robustness to embodiment mismatch. Do you have a controlled comparison varying the degree of embodiment mismatch while keeping the task and correspondence procedure comparable? Identifying when mentor guidance helps, becomes ineffective, or harms learning relative to observer-only RL would help establish the practical scope of the method.

3 comments

Thank you for your comments on our work. We would like to answer your questions below.

Assessing applicability in practice (Q1). When assessing applicability, it is useful to check whether the correspondence is consistent with known task constraints, in addition to considering proximity in the shared feature space. Section 5.3 shows that performance recovers when matches across walls are excluded without changing the neighborhood radius. However, our current method does not provide a general diagnostic procedure for predicting, before training, the risk of overestimation arising from posture or contact configuration. As the mentor speed experiment discussed in our response to Q3 below also shows, the observer’s ability to reproduce the mentor’s motion is not a prerequisite for obtaining useful guidance.

Physically realistic tasks (Q2). As you point out, our current experiments do not include a hurdle with collision geometry that can interact with the Hopper’s body. However, of the variables you mention, posture, joint velocities, and contact configuration can already determine feasibility in the current HopperHurdle setting. (i) The hurdle region is forbidden to the center of mass. To clear this region, the Hopper must use its own joint dynamics and ground contact to raise its center of mass above the hurdle. Whether it can do so depends on its posture and joint velocities. (ii) We retain the original Hopper termination condition for unhealthy states (Appendix G.2.3). If the Hopper falls after a failed landing, the episode terminates. The current setting therefore includes the fall termination mentioned in Q2. (iii) The shared features consist only of the x and z coordinates of the center of mass and the sign of its vertical velocity. They omit posture and joint states. This is the situation you describe, in which omitted body-specific variables can determine whether the Hopper can clear the hurdle even when the shared features match.

Controlled comparison of embodiment mismatch (Q3). Figure 10 in Appendix G.7.1 reports a controlled comparison in PointMaze. For each of two paths, we varied the mentor’s maximum speed across four levels while keeping the observer and the procedure for constructing the mentor–observer correspondence fixed. Although the mentor and observer share the same body geometry, conditions in which their attainable speed ranges differ constitute cross-embodiment settings in terms of movement capability. MB-I2L outperformed standard SAC even when the mentor’s maximum speed exceeded the observer’s maximum attainable speed. This shows that the observer’s ability to reproduce the mentor’s actions is not necessarily important, and that a traversable path can provide guidance to the observer. We will revise Appendix G.7 to clarify this point.

We report the results of our additional experiments below.

Evaluation with a hurdle that can collide with any part of the Hopper’s body (Q2). We conducted additional experiments in PhysicalHopperHurdle, where we implemented the hurdle as a physical obstacle that can collide with any part of the Hopper’s body. We removed the forbidden region for the center of mass and retained the original Hopper termination condition for unhealthy states. Contact with the hurdle does not itself constitute failure, and the Hopper can step on the hurdle to get across. The Hopper’s center of mass is at a height of approximately 0.6 when standing, and the top of the original forbidden region was at 0.7. We set the physical hurdle height to 0.1 to match this difference. We used the same mentor data, shared features, and neighborhood definition as in the original setting.

Normalized AUC of the evaluation success rate (mean ± SD over 5 seeds, Table 9).

EnvironmentMB-I2LSACCEPWILOri. Rew.
PhysicalHurdle-S0.98 ± 0.0250.20 ± 0.450.83 ± 0.110.45 ± 0.470.53 ± 0.11
PhysicalHurdle-L0.70 ± 0.400.00 ± 0.000.62 ± 0.240.35 ± 0.330.54 ± 0.21

In Short, MB-I2L achieved a higher mean AUC than all comparison methods. In Long, MB-I2L also achieved the highest mean AUC, although the difference from CE was small relative to the variability across seeds. Several methods achieved higher mean AUCs than in the original setting, suggesting that the task was easier with the physical hurdle. In Long, the increase in mean AUC was larger for MB-I2L (from 0.0050 to 0.70) than for CE (from 0.25 to 0.62).

The original and physical-hurdle settings differ only in how the hurdle is implemented and how contact is handled. In Appendix G.5.1, we hypothesize that learning in the original Long setting stalled at a local optimum because baseline injection favored actions that brought the observer quickly to later mentor neighborhoods. For example, the baseline may favor a single large forward jump over landing and adjusting the takeoff position for the next jump. We observed such large jumps, followed by entry into the forbidden region, in videos of training episodes in the original Long setting. If the observer reaches a later mentor neighborhood during the large jump, baseline injection can raise critic targets along the bridge ending there, even if the observer subsequently fails. For comparable mentor returns-to-go at the endpoints, reaching an endpoint in fewer observer steps discounts the mentor return less and therefore yields a larger baseline. In the original setting, the episode ends if the Hopper’s center of mass enters the forbidden region during the jump or if its momentum carries it into the region afterward. With a physical hurdle, the Hopper can continue moving after contact. This difference may help explain why MB-I2L performed better in the physical-hurdle variant of Long.

We also observed a second pattern of behavior in both Long settings. At the start of the episode, the Hopper extended its leg without first crouching. This brought it quickly into a neighborhood along the mentor’s path, but it gained too little height, tipped backward, and failed near the starting position. Because this behavior occurs before the Hopper reaches the hurdle, it does not depend on how the hurdle is implemented. With the physical hurdle, this behavior occurred in the one seed (of five) in which MB-I2L failed to learn to reach the goal. In the original setting, Mentor Progress (the furthest mentor index reached by the bridges in each episode; Appendix G.5.1) stalled near the start of the trajectory in six of ten seeds, and we observed the same behavior in videos from all six. We report these results and their interpretation in Appendix G.8.

Controlled comparison of embodiment mismatch (Q3). In addition to the mentor speed experiment discussed in our previous response (Appendix G.7.1, Fig. 10(a) of the revised manuscript), we varied the bounce height of the ball mentor in HopperHurdle-Short. We held the observer, task, shared features, and neighborhood definition fixed. We varied only the height Δz\Delta z of the ball’s center above the top of the hurdle at the apex of its bounce, comparing the default value of 0.4 with 0.2 and 0.6. In the mentor speed experiment, the observer can pass through the same positions as the mentor even if it moves more slowly. Increasing the bounce height instead allows us to examine positions that are difficult for the Hopper to reach, making it harder for the observer to enter the corresponding mentor neighborhoods.

Normalized AUC of MB-I2L in HopperHurdle-Short (mean ± SD, Table 8).

Δz\Delta z0.20.4 (default)0.6
Normalized AUC0.68 ± 0.320.63 ± 0.270.42 ± 0.42
Seeds5105
Mentor steps638195

Mean AUC decreased as the mentor’s bounce height increased. The difference between 0.2 and 0.4 was small relative to the variability across seeds, and at 0.6, two of the five seeds had an AUC of zero. In videos of the two failed seeds at Δz=0.6\Delta z = 0.6, the Hopper extended its leg to raise its body and then fell, which is the same behavior described above for the Long settings. At all three heights, mean AUC exceeded that of both SAC (0.00) and CE (0.077), neither of which uses a mentor (Table 1).

MB-I2L allows the observer to skip intermediate states along the mentor trajectory and form a bridge to a later neighborhood without having to reach every mentor state. We believe this property partly explains why learning could succeed even when the bounce height was varied. Changing the height also changed the number of steps the mentor took to reach the goal. Both the accessibility of the mentor neighborhoods and the magnitude of the mentor return-to-go may therefore have contributed to the decrease in AUC.

Examples of conditions under which mentor guidance can hinder learning include neighborhoods that allow matches across walls (Sec. 5.3) and the original HopperHurdle-Long setting, where MB-I2L performed worse than CE (Table 1). We report the bounce-height experiment and discuss these two possible explanations in Appendix G.7.2 (Table 8).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to All Reviewers: Baseline Implementation Corrections and Validation

We thank the reviewers for their feedback on our work. While checking our baseline implementations, we identified issues with the handling of terminal transitions in HER and PBRS. Because these issues bear on the reliability of our comparisons, we report them and the corresponding corrections here.

For PBRS, we had omitted setting the next-state potential to zero on terminal transitions. We corrected this and reran PBRS under the original experimental settings. The normalized AUC of the evaluation success rate, averaged over five seeds, was 0 on all four tasks (PointMaze, AntMaze, HopperHurdle-Short, and HopperHurdle-Long), matching the values reported in the original submission. Thus, MB-I2L still achieved higher mean normalized AUC than PBRS when both methods used the same mentor information. HER relabels goals in previously collected transitions. In this study, we used the Stable-Baselines3 (SB3) implementation of HER. In our setting, reaching the goal terminates the episode, so a transition that reaches a relabeled goal should also be treated as terminal. Our adaptation of the standard SB3 implementation did not consistently apply this rule. We corrected the terminal handling and reran HER. The corrected implementation likewise yielded a normalized AUC of 0 on all four tasks, matching the values in the original submission.

As an additional check of the implementations, we ran SAC, PBRS, and HER on PointMaze using the same U-shaped maze layout as in the paper. We set the cell width to one quarter of that used in the PointMaze experiments reported in the paper. All three methods achieved an evaluation success rate of 1.0. These results suggest that task difficulty contributed to the low success rates of these methods on the original tasks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.