acceptodds
Under review as a conference paper at ICLR 2027

Learning Transferable Skills using Goal-Conditioned Bisimulation

Abstract

Unsupervised skill discovery has emerged as a promising approach for leveraging reward-free datasets to pretrain general-purpose policies. However, current skill discovery methods either require access to expert data or exhibit limited generalization, failing to transfer effectively to previously unseen layouts. A key challenge is to learn representations that capture the temporal structure of the environment while remaining robust to variations across layouts. To address this issue, we present an objective for learning action-aware temporal representations that satisfy the functional equivariance property while preserving the local temporal structure of the environment. Building upon this embedding, we further propose unsupervised skill discovery using bisimulation, which learns transferable skills by conditioning the behavior of skills exclusively on the subset of state features that directly affect their execution. This enforces invariant behavior across different layouts, enabling skills to transfer effectively to other configurations. Finally, through comprehensive empirical evaluations, we show that skills learned in a given environment can be effectively applied to solve downstream tasks in various environment layouts, demonstrating strong out-of-distribution generalization.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Unsupervised Skill Discovery, Representation Learning, Goal-Conditioned Bisimulation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Pwn5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the problem of learning skills that can be reused in environments with different structures after being discovered in one environment. To this end, inspired by the bisimulation literature, it learns state representations ψ\psi that reflect action similarity and trains a network ϕ\phi that captures temporal distance based on these representations. It then learns a policy based on HILP [1] that takes latent states β(s)\beta(s) as input to reduce dependence on specific maze layouts. Experiments evaluate the method through high-level policy learning and test-time planning in maze environments, demonstrating improved transfer performance to new mazes.

[1] Park, Seohong, et al. “Foundation Policies with Hilbert Representations.” 2024.

Critical strengths
  • The motivation is clear: existing offline RL and offline skill discovery methods, such as HILP, struggle to transfer learned skills.
  • The paper is easy to read, and the proposed method is well-designed to address the problem it aims to solve.
Critical weaknesses
  • Although described in the limitations, the lack of manipulation experiments raises questions about the method’s scalability. In maze environments, the task-specific state typically consists of only the two-dimensional “xy-coordinates” within the full state space (e.g., 29 dimensions for Ant), making the ϕ\phi network relatively easy to learn. In contrast, manipulation requires jointly considering object positions, the end-effector’s position and orientation, and other factors, making the ϕ\phi network harder to learn. Although the algorithm does not rely on maze-specific assumptions, its performance may be due to the proposed loss being particularly easy to optimize in maze environments.
  • The experimental setup for layout generalization in Section 5.3 does not seem fair. HILP uses the planning method presented in its original paper [1], whereas the proposed algorithm uses a graph-based method. This makes it difficult to fully distinguish whether the improvement in test-time performance comes from planning or from the learned representations. Since both methods learn state embeddings that can be used for graph-based planning, it seems reasonable to compare them using the same planning method.
  • In Equation 3, using the L1 distance between actions to learn action-aware representations seems to have limited scalability. For example, when a robot is controlled through torques, the resulting motion depends heavily on the dynamics and its current pose. Thus, actions that are close in L1 distance may produce very different motions when applied in different states. In maze environments, this may not be a major issue because a single action has only a small effect on the task-relevant state, namely the xy-coordinates. However, it could make a substantial difference in manipulation, where precise contact is important.

For these reasons, I currently lean toward rejection.

[1] Park, Seohong, et al. “Foundation Policies with Hilbert Representations.” 2024.

Questions for authors

Most of my questions stem from the weaknesses discussed above. The main questions are as follows:

  • Does the performance advantage in Section 5.3 remain when HILP and the proposed method use the same graph-based planner?
  • What evidence supports the method’s applicability beyond mazes, particularly to manipulation, where task-relevant states are more complex and the L1 distance between actions may not reflect their effects due to differences in dynamics and the current pose?
  • In Equation 6, aka_k is sampled uniformly from the action space, whereas Algorithm 1 samples it from the dataset. Could you clarify which sampling distribution is used? If actions are sampled uniformly from the action space, this may frequently produce actions outside the dataset distribution and lead to unstable training. Have you considered alternatives, such as sampling from a policy network?
3 comments
AuthorsOpenReview ↗

Response to Reviewer Pwn5

We thank the reviewer for the valuable feedback.

Q1: We agree with the reviewer that using different planning methods may introduce a potential source of unfairness in the comparison. Therefore, following the reviewer's suggestion to evaluate HILP with graph-based planning, we report the layout generalization results in the table below. The results show that incorporating graph-based planning improves HILP's performance. Although HILP achieves more promising results on the highly noisy explore dataset, consistent with the findings from the hierarchical evaluation in Table 1, USB+ATR still significantly outperforms HILP on average.

MethodHILP with Graph-based PlanningUSB+ATR with Graph-based Planning
pointmaze-medium-navigate0.0 ± 0.036.9 ± 9.0
pointmaze-large-navigate0.0 ± 0.042.0 ± 9.4
pointmaze-medium-stitch0.0 ± 0.033.7 ± 9.3
pointmaze-large-stitch3.5 ± 3.552.5 ± 9.8
antmaze-medium-navigate16.2 ± 4.375.7 ± 2.2
antmaze-large-navigate22.6 ± 6.766.3 ± 3.9
antmaze-medium-stitch10.8 ± 4.268.6 ± 3.0
antmaze-large-stitch35.7 ± 7.254.8 ± 4.3
antmaze-medium-explore87.6 ± 2.369.8 ± 4.1
antmaze-large-explore87.3 ± 2.821.6 ± 4.5
Average26.3752.19

Q2: To further address the generalizability of our method beyond locomotion, we have conducted additional experiments and reported the results in a separate comment. These experiments further demonstrate the applicability of our method to other domains, particularly manipulation tasks that involve more complex skills.

Q3: Thank you for your detailed review. As you mentioned, to avoid sampling out-of-distribution actions, we sample actions from the dataset in practice.

Following your suggestion, we further ablate the choice of action distribution. Specifically, in this new experiment, when calculating d(si;sj)d(s_i; s_j) in Equation 6, we sample only actions that have been taken at either sis_i or sjs_j in the dataset. This experiment is closely related to the alternative you mentioned, namely, sampling actions from a policy network, as these actions are sampled from the goal-conditioned policy, starting from either sis_i or sjs_j and conditioned on arbitrary goals.

We evaluate this variant using test-time planning in the antmaze-large-navigate environment and report the results averaged over 5 random seeds:

Evaluation SetupActions Sampled from States sis_i or sjs_jActions Sampled from Entire Dataset (Ours)
Task Generalization43.2 ± 4.564.6 ± 4.8
Layout Generalization39.4 ± 3.666.3 ± 3.9
Reviewer_Pwn5OpenReview ↗

Response to Authors

We appreciate your time and effort in conducting the additional experiments. Your new results have addressed most of my concerns.

However, regarding manipulation tasks (see Weaknesses 1 and 3), although USB+ATR achieves the highest average performance, the improvement over the baselines is not substantial, and the experiments are limited to single-cube manipulation tasks. Therefore, I think that we should be somewhat cautious about claiming that the method generally performs well on manipulation tasks.

I understand that the rebuttal period is limited, so I do not expect additional experiments at this stage. However, I believe that demonstrating generalization to multi-block manipulation tasks would further strengthen the paper.

Overall, I appreciate your efforts in addressing my concerns. At this stage, I will maintain my original score.

AuthorsOpenReview ↗

Response to Reviewer Pwn5

Dear Reviewer Pwn5,

Thank you for your thoughtful feedback and for acknowledging our efforts to address your concerns. We appreciate your constructive suggestions, which have helped us further strengthen our evaluation.

We agree that evaluating our method on more challenging manipulation tasks would provide stronger evidence of its generalization capabilities. Motivated by your suggestion, we conducted additional preliminary experiments on the OGBench cube-double environment, which involves manipulating multiple objects and requires more complex interactions than the single-cube setting.

In this environment, USB+ATR achieves a preliminary success rate of 79.2 ± 4.2% with high-level policy training (2 random seeds), compared to 100.0 ± 0.0% on the corresponding cube-single-noisy environment. These results are encouraging and suggest that our method retains its ability to learn useful skills in more complex manipulation settings. Furthermore, we observed interesting learned behaviors involving interactions between the two cubes, providing additional qualitative evidence of the method's ability to discover meaningful manipulation skills.

We acknowledge that these results are preliminary, and we are conducting further evaluations with additional random seeds and more comprehensive baseline comparisons. We intend to include these results and analyses in the final version of the manuscript.

We sincerely appreciate your time, constructive feedback, and careful consideration of our work.

Reviewer_iJGkOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper tackles the generalizability bottleneck of existing temporal representation for skill discovery. To mitigate this problem, the paper combines two approaches, action-aware temporal representation and unsupervised skill discovery using bisimulation. The experiments demonstrate promising results on the proposed approach. Overall, the approach is promising, as long as existing concerns are addressed.

Critical strengths
  • The problem is real and critical. Using naive temporal representation relies too much on the layout’s global temporal geometry, limiting its generalizability.

  • The solution is reasonable and straightforward. Action-aware temporal representation induces the discovered skill vectors to have similar representational direction to action sequence. To fully ensure the generalization, the paper also combines bisimulation, inducing the policy to utilize the representation \beta that considers similar dynamics rather than similar states.

  • The main experiment results are promising, outperforming the baselines on average.

Critical weaknesses
  • Trade-off does exist. From the in-distribution settings (Appendix B.2), the proposed approach is not the best in most settings, while quite robust on average.

  • A concern remains whether replacing the “layout-specific” temporal structure is a good inductive bias for general skill discovery setting. In maze configuration, the discrepancy between global temporal structure and locally reusable action is obvious. On the other hand, layout could be a critical information, for example, in manipulation settings, the object position or obstacle might change the proper action to estimate.

  • The experiments are all done in maze configuration, where the discussed problem matters a lot. However, the paper does not conduct experiments on general skill discovery settings, such as OG Bench [1] manipulation environment.

Questions for authors
  • The paper states that the action-temporal representation is not a universal temporal representation. But which temporal properties does it retain? For example, how well do distances in ψ\psi correlate with true temporal distances?

  • Following Weaknesses 2 and 3, it is still unclear whether the current inductive bias is still necessary in general settings. Could the authors conduct additional experiments on diverse environments?

3 comments
AuthorsOpenReview ↗

Response to Reviewer iJGk

Thank you for your careful review and constructive feedback on our paper. We are pleased to hear that you found our work interesting.

Q1: To clarify what temporal distances ATR retains, we explain what we mean by ATR being a local but not universal temporal representation. By saying that ATR is a local temporal representation, we mean that the temporal distances between adjacent states are constrained by Equation 5. As a result of this constraint, temporally close states will eventually have nearby representations in the ATR space.

On the other hand, by saying that ATR is not a universal temporal representation, we mean that there is no guarantee that the representations of any two arbitrary states will correlate with their temporal distance. Instead, ATR introduces an inductive bias whereby starting from any state and moving in the same direction results in similar sequences of actions, which facilitates the learning of monotonic skills.

Nevertheless, this inductive bias still allows ATR to retain some temporal distance information for more distant states. For instance, consider a special case in which we can reach state B from state A using a certain set of actions, and then reach state C from state B using the same set of actions. Assuming that there are no other shorter paths between A and C that interfere with this relationship, we can infer that:

∥ϕ(A)−ϕ(C)∥=∥ϕ(A)−ϕ(B)∥+∥ϕ(B)−ϕ(C)∥.\|\phi(A) - \phi(C)\| = \|\phi(A) - \phi(B)\| + \|\phi(B) - \phi(C)\|.

This illustrates how temporal distance information can also emerge for states that are farther apart.

To illustrate this with empirical results, we draw the reviewer's attention to Figure 3. In this figure, although nearby states have close representations, this correlation does not necessarily hold for arbitrary pairs of points (e.g., points (4, 24) and (12, 24), which are temporally far apart). However, our embedding preserves the intended objective: displacements along any fixed vector in this representation correspond to an almost monotonic sequence of actions.

Q2: Regarding the question of how well our method generalizes beyond locomotion tasks, we have conducted additional experiments, the results of which are presented in a separate author comment. These results demonstrate that our method is also effective in other domains, including manipulation tasks involving more complex skills.

AuthorsOpenReview ↗

Follow-Up on Our Rebuttal

Dear Reviewer iJGk,

We would like to kindly follow up on our rebuttal. We believe that we have addressed the major concerns raised through our responses and the clarifications provided, and we would be grateful for any further thoughts or feedback you may have.

If our responses have adequately addressed your concerns, we would sincerely appreciate it if you could consider revisiting your ratings. Any additional feedback would also be greatly appreciated and would help us further strengthen the final version of our work.

Thank you again for your time, thoughtful feedback, and careful consideration of our submission.

Sincerely,
The Authors

I thank the authors for the detailed response. The manipulation results are encouraging and address my concern about the applicability. I also appreciate the clarification regarding the local temporal representation. However, while Equation (5) constrains the representations of adjacent states, it remains unclear to what extent the learned distances correlate with actual temporal distances. A quantitative analysis may help to validate it. In addition, the equality in the response,

∣ϕ(A)−ϕ(C)=∣ϕ(A)−ϕ(B)∣+∣ϕ(B)−ϕ(C)∣|\phi (A)-\phi(C)=|\phi (A) - \phi(B)| + |\phi(B)-\phi(C)|,

requires an additional assumption. Equality holds when the two displacement vectors are aligned in the same direction, which isn't generally guaranteed and should be made explicit. Therefore I am leaning towards maintaining my current score.

Reviewer_9wH5OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper targets out-of-distribution (OOD) layout generalization (which, in my opinions, is the novel env generalization, not the type of novel task generalization) for unsupervised skill discovery in offline goal-conditioned RL. It introduces two coupled components: (i) an action-aware temporal representation (ATR) that leverages a goal-conditioned bisimulation-style embedding ψ(s,g)\psi(s,g) based on action similarity (rather than rewards) to shape a temporal embedding ϕ\phi that preserves local temporal structure while enforcing functional equivariance; and (ii) a bisimulation-based state encoder β(s)\beta(s) that clusters states with similar transition dynamics by comparing ATR displacements under shared actions (using a learned dynamics model to create matched next-states in the offline setting). Conditioning skills on β\beta and training with directional rewards in ϕ\phi yields monotonic, layout-agnostic skills that transfer to unseen maze layouts. Empirically, on OGBench mazes with randomized layouts, the method outperforms chosen baselines under both hierarchical control and test-time planning.

Critical strengths
  1. Separating the representation defining behavioral diversity from the representation necessarily used for control is an interesting extension of HILP [1].
  2. High-level motivation, desiderata, and method overview are well articulated.
Critical weaknesses
  1. The evaluation establishes navigation-layout transfer, but not broader compositional skill generalization. Layout transfer is meaningful in its own right. Nevertheless, all evaluations remain within a goal-reaching navigation task family. They do not test new task semantics, object interactions, or new compositions of heterogeneous primitives. Moreover, directional locomotion is a particularly favorable setting for action-consistent embeddings, where you just need to extract four primitive control skill over 4 directions (up, down, left, right). Experiments on more complex manipulation setting would strengthen the broader contribution.

  2. Severe technical limitations or concerns exist. I list as follows:

  • The ATR construction is defined via matching local differences to ψ(s,s’)\psi(s,s’) there is no explicit guarantee of global integrability or triangle inequality beyond an informal appeal to metric space constraints. Path-inconsistencies around loops could degrade global geometry; discussion or diagnostics for this would strengthen the claims.
  • β\beta relies on a learned forward model ff to synthesize matched next states for shared actions. Model bias can affect β\beta, especially if actions are sampled outside the dataset support; there is little quantitative analysis of f’s fidelity or robustness to model misspecification.
  • The “actions as metric” choice in ψ\psi may be sensitive to action scaling across dimensions (e.g., different torque ranges in Ant); normalization choices are not described and could materially affect equivariance quality.
  • The deterministic MDP assumption simplifies losses but narrows applicability.
Questions for authors
  1. How far does the proposed abstraction generalize beyond layout changes within navigation? Both evaluated domains admit a relatively compact family of directional behaviors, whereas the motivation also discusses manipulation. What evidence distinguishes learning broadly reusable skills from reducing dependence on absolute position or the training map? An evaluation requiring recombination of learned skills for new manipulation tasks, where different behaviors depend on different state features, would make the practical contribution substantially clearer.

  2. Why does USB+ATR perform substantially worse than GCIQL and QRL in all four PointMaze planning settings? Is the problem the geometry of the ϕ\phi graph, information discarded by β\beta, or execution of the planned subgoals? A controlled comparison on identical layouts and goals, holding the planner and subgoal sequence fixed while varying the controller, would help distinguish these explanations.

  3. What precisely is meant by “maximally spread” in Equation (4)? Global centering fixes the arbitrary origin of ψ\psi and changes displacement directions, but leaves pairwise target distances unchanged. Why is the mean self-pair embedding an appropriate reference? Please explain the 0.0/0.4% versus 66.3% gap in Table 5, for example by measuring cycle inconsistency of the centered target displacements and the fitting residual in Equation (5).

  4. What temporal structure does ATR retain after Equation (5)? In Table 3, on antmaze-large-navigate, replacing HILP’s representation with ATR reduces task success from 71.7% to 57.6% and layout success from 44.9% to 25.1%. Does this reflect a loss of useful temporal-distance structure or another effect of the representation change? Please directly assess the relationship between latent distance and temporal distance.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 9wH5

Thank you for your detailed and thoughtful review.

W1 and Q1: To further evaluate the generalizability of our method beyond locomotion, we conducted additional experiments, with the results reported in a separate comment. These experiments provide further evidence of the method’s applicability to other domains, particularly manipulation tasks.

W2-1 and Q3: The choice of origin can significantly affect the pairwise distances between targets. In fact, constructing an ATR requires a fixed center. To illustrate, consider a state with two different goals associated with different ψ\psi values. If we use a fixed center, after normalization, the two goals will necessarily point in different directions. However, if we use a dynamic center that depends on gg, the two goals may end up pointing in the same direction, making them appear close to each other. This claim is further supported by the empirical results in Table 5.

W2-3: In general, input scaling is an inherent challenge in machine learning. Similar to our work, several influential studies have used actions as the metric, including the work by Agarwal et al. (2021). In the OGBench environments we use, actions generally have similar scales and are clipped to the range [−1,1][-1, 1]. However, in environments with substantially different action scales, an appropriate normalization method may be necessary.

W2-4: The deterministic assumption is not unique to our work, as it has also appeared in many prior works on goal-conditioned RL (Wang et al., 2023; Park et al., 2023, 2024; Ma et al., 2023; Ahn et al., 2025).

Although extending current methods to stochastic environments would be a valuable direction, the performance of existing methods on deterministic benchmarks suggests that deterministic environments remain challenging. Additionally, skill transferability, despite its importance, remains relatively underexplored in the literature. Therefore, focusing on deterministic environments is a reasonable and meaningful starting point for investigating this problem, particularly given the challenges that existing methods continue to face even in deterministic settings.

Q2: As discussed in the manuscript (lines 436–441), planning over raw states in the pointmaze environment favors methods that disregard the temporal structure of the environment, since the 2D state space of pointmaze already provides sufficient information for optimal planning.

However, methods such as HILP and USB+ATR attempt to reconstruct this temporal representation in every environment, regardless of the information available in their observations. This is further reflected in the significant performance drop of GCIQL in the antmaze environment.

Q4: Actually, by replacing HILP's representation with ATR, we sacrifice some of the universal temporal information in favor of skill monotonicity. Therefore, by training HILP skills on this temporal representation, we lose some of the universal temporal information without fully harnessing its advantages, as also reflected in the results.

A more detailed explanation of the distinction between local and universal temporal representations is provided in our response to Reviewer iJGk (Q1), which may also be insightful.

References

[1] Rishabh Agarwal, Marlos C. Machado, Pablo Samuel Castro, and Marc G Bellemare. Contrastive behavioral similarity embeddings for generalization in reinforcement learning. In International Conference on Learning Representations, 2021.

[2] Tongzhou Wang, Antonio Torralba, Phillip Isola, and Amy Zhang. Optimal goal-reaching reinforcement learning via quasimetric learning. In International Conference on Machine Learning, pages 36411–36430. PMLR, 2023.

[3] Seohong Park, Dibya Ghosh, Benjamin Eysenbach, and Sergey Levine. HIQL: Offline goal-conditioned RL with latent states as actions. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.

[4] Seohong Park, Tobias Kreiman, and Sergey Levine. Foundation policies with hilbert representations. In Proceedings of the 41st International Conference on Machine Learning, pages 39737–39761. PMLR, 2024.

[5] Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, and Amy Zhang. VIP: Towards universal visual reward and representation via value-implicit pre-training. In The Eleventh International Conference on Learning Representations, 2023.

[6] Hongjoon Ahn, Heewoong Choi, Jisu Han, and Taesup Moon. Option-aware temporally abstracted value for offline goal-conditioned reinforcement learning. In The Thirty-ninth Conference on Neural Information Processing Systems, 2025.

AuthorsOpenReview ↗

Follow-Up on Our Rebuttal

Dear Reviewer 9wH5,

We would like to kindly follow up on our rebuttal. We believe that we have addressed the major concerns raised through our responses and the clarifications provided, and we would be grateful for any further thoughts or feedback you may have.

If our responses have adequately addressed your concerns, we would sincerely appreciate it if you could consider revisiting your ratings. Any additional feedback would also be greatly appreciated and would help us further strengthen the final version of our work.

Thank you again for your time, thoughtful feedback, and careful consideration of our submission.

Sincerely,
The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Additional Evaluation on Robotic Manipulation

Following the helpful suggestion of all reviewers, we conducted an additional experiment on a robotic manipulation task to demonstrate the generality of our approach beyond navigation environments. Specifically, we evaluated our method on the OGBench cube-single task, where a robotic arm is required to manipulate a cube on a tabletop workspace.

To introduce the notion of layout generalization, we collected a new dataset, with the same size as the original OGBench dataset, in which the cube is manipulated exclusively within the left half of the workspace. Consequently, all states in which either the robotic arm or the cube occupies the right half of the workspace are completely unseen during training.

At evaluation time, methods might be required to solve tasks that involve moving the cube into these unseen regions. Specifically, the seen layout evaluation consists of a vertical displacement within the observed region (Task 2), whereas the unseen layout evaluations require horizontal (Task 1) or diagonal (Tasks 4 and 5) cube movements into previously unseen areas.

We conduct experiments on two datasets: (1) a play dataset, which resembles the high-quality navigation-style datasets provided by OGBench, and (2) a noisy dataset, which provides lower-quality demonstrations but with higher state coverage.

Hierarchical evaluation results on the in-distribution tasks are averaged over 5 random seeds, while results on the layout generalization setting are averaged over 3 unseen layouts, each evaluated with 5 random seeds.

Task Generalization

EnvironmentSACOPALHILPUSB+ATR
cube-single-play86.7 ± 9.193.3 ± 6.780.0 ± 10.7100.0 ± 0.0
cube-single-noisy86.7 ± 9.1100.0 ± 0.0100.0 ± 0.0100.0 ± 0.0
Average86.796.6590.0100.0

Layout Generalization

EnvironmentSACOPALHILPUSB+ATR
cube-single-play42.2 ± 7.471.1 ± 6.842.2 ± 7.453.3 ± 7.5
cube-single-noisy42.2 ± 7.473.3 ± 6.782.2 ± 19.397.8 ± 2.2
Average42.272.262.275.55

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.