acceptodds
Under review as a conference paper at ICLR 2027

Exploiting Local Dynamics Regularity for Reusable Skills in Offline Hierarchical RL

Abstract

Hierarchical Reinforcement Learning (HRL) promises to solve long-horizon Reinforcement Learning (RL) tasks more efficiently than non-hierarchical counterparts by discovering and reusing temporally-extended skills. However, obtaining skills that are actually reusable remains an open challenge. Towards this end, we focus on abstractions that exploit the intuition of local dynamics: local transitions in different global contexts require similar kinds of action sequences. By aligning these contexts with the action sequences they require, we are able to learn which skills to reuse and where to reuse them. In principle, this information should benefit many HRL algorithms, where high-level policies have to reason about the low-level skills they use. The resulting algorithm CARL (ontrastive ction-based epresentations for Reusable ocal Control) shows both qualitative clustering of meaningful skills in complex humanoid environments and improved downstream performance on the OGBench benchmark when integrated with HIQL.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Offline Reinforcement Learning, Representation Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_aqahOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This work propose CARL that learns state-goal representation thru contrastive alignment with associated k-step action sequences. That is to say, if from s1 to g1 requires similar actions as s2 to g2, then (s1, g1) and (s2, g2) are represented similarly in the representation space. Building on the hypothesis that "behavioral similarity will naturally emerge when many states share local dynamics structure", the authors propose to use contrastive loss (InfoNCE) to train the embeddings on an offline dataset.

Several key assumptions are worth noting: (1) deterministic transition; (2) tasks are goal oriented and goals are a state in the state space; (3) offline dataset is collected by an expert policy follow optimal strategies (all optimal strategies equally).

Critical strengths
  1. The presentation is good visually, and in general easy to follow.
  2. The implementation details and justification choice of hyperparameter is provided, adding up to the credibility of the result. The ablation study is useful to understand the algorithm.
  3. Some detailed analysis is provided in Appendix.
Critical weaknesses

Major

  1. the paper is restricted to a limited, deterministic control setting, and subgoals are within the state space. I appreciate that the authors acknowledge that, though. But to me, many things can be simplified, as the transition probability is either zero or one, and many things may be reduced to a trivial/highly restrictive version. Although the authors propose to generalize to a stochastic setting, the proposed method may not work well with the general stochastic case--where the same action sequence from one state may lead to diverse outcomes. The proposed method might be brittle if any of those assumptions break.
  2. Sec 3: The k-ball MDP is not well-defined: consider some state s'' that is k+1 steps reachable from s. That means there is some state, say s', is k-step reachable from s. Then there is a possibility of taking some action to transition from s' to s'' in the original MDP. But that state is not even in your k-ball MDP's state space. Then, the transition kernel should not be the original one. So please specify the transition function and reward.
  3. The definition of Dynamics Bisimilarity is also strange to me. I don't understand what is the quotient set S1/BS_1/B when B is on S1×S2S_1\times S_2. Usually a quotient set is defined on an equivalence relation on one set. Then, if I'm not missing something, definition 3.3 is broken.
  4. Overall, I think section 3.2 should go through a major revision. To be more constructive, some examples with a few states throughout the how section could help.
  5. Sec.4: the connection of the proposed method is indirectly related to what is proposed in sec.3. There are few direct references to any definitions in sec.3. In sec.3 definitions, it is not related to goals either. To summarize a bit, sec.3 compares states through local transition structure, while sec.4 compares state-goal pairs through dataset action distributions. Then why is the story not directly "if the state-goal shares a similar (optimal) control action sequence (inverse dynamics), then they share similar embeddings"?
  6. line 216: "we can recover a bisimulation metric" I'm doubtful about this -- it is not true if you mean the bisimulation metrics by Fern et al., 2004. In general, "if they share the same optimal action sequences" -> "recover bisimulation metric" is false: you can have two trajectories from two states s1 and s2, that have the same optimal actions but totally different rewards. In this case, s1 and s2 are not bisimilar (and the metric > 0). It's closer to the work by Hansen-Estruch et al., 2022, but you need more justification and theory to make such claims. Please also define the metrics formally. Even if the theory goes through, many recent papers show the difficulty of "recovering the exact metrics" (Kemertas and Aumentado-Armstrong, 2021; Zang et al., 2023; Luo et al., 2025). So at least it should be toned down.
  7. experiment: In sec.5.1 and fig.2, I can only see a sketch of the diagnostic experiment. If I'm not missing any description in the Appendix, then to me it's not sufficient to understand the exact setting of the experiment. And any of those rooms only requires memorizing the successful action sequence -- many methods in offline skill discovery (see the survey, sec.5, by Klissarov et al., 2025) may have the same promise, but the comparison is only limited to HIQL, which mainly focuses on credit assignment. It does not seem enough for generalization, like, in many scenarios we requires agent can generalize to different maps with different successful paths. And, as a diagnostic setting, I would expect to see more diagnostic results other than just "who succeeds or performs better".
  8. the scaled result is largely limited to OGBench. The baselines are limited to HIQL/HGCBC, with insufficient justification for why to choose those methods. There are many related HRL works, or representation learning methods (like methods in sec.5, by Klissarov et al., 2025, or related work in the work by Park et al., 2024) that work on this setting and are worth comparison. There is also no standalone CARL (alg.1) result in the experiments either (then why you write alg.1).

Minor:

  1. 3.1 should define the dataset D as well.
  2. inconsistency in presentation still exists, such as mixing "state-goal" and "state--goal"
  3. the citation of previous work (in HRL, offline skill discovery, action chunking, metric learning, contrastive representation learning, inverse dynamics, even for state/state-action abstraction) is not comprehensive enough. E.g., for bisimulation, at least the work by Givan et al., 2003 should be cited. Talking about related work, I also have some concerns about the novelty of the work, although this is subjective. For succinctness, I'll not expand on this point.
  4. And many description in terms of bisimulation isn't accurate, e.g.: "Prior work adapting bisimulation to RL measures similarity between MDPs through sequences of rewards or single-step controllability metrics under identical action sequence" which is not true as many those cited works consider behavioral distance of states in one same MDP.
  5. line 229: " sqeuence" -> "sequence". I didn't go through a thorough check, but please fix those typos.
  6. Sec.5.4 only shows some working examples but without statistical significance. I suggest merging this section into sec.5.3 to save space.

Reference

Castro et al. (2021). MICo: Improved representations via sampling-based state similarity for Markov decision processes. NeurIPS.

Ferns et al. (2004). Metrics for finite Markov decision processes. UAI.

Ferns et al. (2011). Bisimulation metrics for continuous Markov decision processes. SIAM Journal on Computing.

Givan et al. (2003). Equivalence notions and model minimization in Markov decision processes. Artificial Intelligence.

Hansen-Estruch et al. (2022). Bisimulation makes analogies in goal-conditioned reinforcement learning. ICML.

Kemertas and Aumentado-Armstrong (2021). Towards robust bisimulation metric learning. NeurIPS.

Klissarov et al. (2025). Discovering temporal structure: An overview of hierarchical reinforcement learning. arXiv:2506.14045.

Luo et al. (2025). Understanding behavioral metric learning: A large-scale study on distracting reinforcement learning environments. Reinforcement Learning Journal.

Park et al. (2024). Foundation policies with Hilbert representations. ICML.

Zang et al. (2023). Understanding and addressing the pitfalls of bisimulation-based representations in offline reinforcement learning. NeurIPS.

Questions for authors
  1. As addressed above, the definition of k-ball MDP and Dynamics Bisimilarity seems handwavy. Can you explain more or fix the definition to minimize my abovementioned concerns?
  2. Please clarify the connection between the contrastive loss and the definitions in sec.3.
  3. Please justify the choice of baseline methods.
  4. Seem like you are using the InfoNCE loss to train action and s,g embedding simultaneously. Could the encoders minimize the loss by learning arbitrary matching codes rather than capturing behavioral similarity? What prevents similar action sequences from being mapped far apart?
  5. Could you provide more details of sec.5.1 experiment? Do you use the same hyperparameter settings as in later sections?
  6. In fig.3, what are the goals exactly? The direction, or some state in the state space? Where are the states sampled from?
Reviewer_VBtMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CARL, a contrastive representation learning method for reusable skills in offline hierarchical reinforcement learning. CARL aligns state-goal representations with the action sequences that achieve them, encouraging similar local behaviors to share representations across different global contexts. Integrated with HIQL and HGCBC, CARL improves performance on a broad range of OGBench tasks. The paper also provides ablations and visualizations to examine the learned representations.

Critical strengths
  1. Meaningful research direction. Learning representations that support skill reuse across different contexts addresses an important problem in hierarchical RL.

  2. Strong empirical results. CARL consistently improves HIQL and HGCBC across many OGBench tasks, with particularly substantial gains in long-horizon navigation and manipulation.

  3. Useful ablations. The comparisons with single-action and regression-based objectives, along with analyses of representation dimensionality, skill horizon, and dataset coverage, help establish the effectiveness of the approach.

Critical weaknesses
  1. The motivation does not sufficiently establish the limitations of existing goal-conditioned HRL methods.

The introduction argues that goal-conditioned low-level policies lose temporal abstraction and consistent skill reusability. However, these properties are not inherently absent from goal-conditioned HRL, and the paper does not clearly explain why existing methods such as HIQL fail to capture reusable local behaviors. The authors should identify the specific limitation their method addresses, ideally with a comparison showing how HIQL and CARL represent the same behavior across different global contexts.

  1. The theoretical justification overstates what the contrastive objective guarantees.

Section 4.1 concludes that two state-goal pairs have identical embeddings if and only if they induce the same conditional action distribution. However, identical InfoNCE scores do not necessarily imply identical embeddings, since distinct embeddings may produce the same similarities with all action embeddings. The authors should clarify the additional assumptions required for this equivalence or weaken the claim to equivalence of the induced score functions.

More broadly, the connection between dynamics bisimilarity and behavioral similarity needs clarification. The former concerns environment dynamics, while the latter depends on the action distribution in the offline dataset. A discussion of when these two notions agree would strengthen the theoretical motivation.

  1. The training objective is not fully consistent with the definition of behavioral similarity.

Definition 4.1 defines behavioral similarity through the distribution of action sequences required to reach a goal. However, Appendix A pairs intermediate goals (s_{t+m}) with the full (k)-step action sequence, including actions taken after the goal is reached. These subsequent actions may introduce information unrelated to the intended skill. The authors should explain this design choice and compare it against using action sequences truncated at (m) steps.

  1. The experiments do not directly establish that CARL enables skill reuse.

The downstream results demonstrate the usefulness of CARL, but improved task performance and qualitative clustering do not necessarily establish that the learned representations support reuse of the same skills across contexts.

The gridworld experiment (§5.1) is relevant, but CARL's encoder is trained using data from all rooms, including those held out from downstream policy training. This distinction should be made clearer when describing zero-shot generalization. A more direct evaluation could train a low-level policy in one context and measure whether the same learned skill representation produces the intended behavior in other contexts, without additional policy training.

  1. The empirical conclusions could be more carefully qualified.

The reported win rates include small improvements on tasks where confidence intervals overlap, and several challenging environments remain near zero success. In addition, the main HIQL comparisons use previously reported baseline results rather than matched experimental runs. The authors should distinguish numerical wins from statistically supported improvements and clarify whether the comparisons use identical evaluation protocols. The stronger gains on selected tasks deserve more emphasis than the aggregate win-rate counts.

Questions for authors

What specific failure of existing goal-conditioned representations motivates CARL beyond the intuition that local behaviors should be reusable?

Under what assumptions does equality of contrastive scores imply equality of state-goal embeddings?

Why are full action sequences used for intermediate goals, rather than truncating sequences at the goal-reaching step?

Can the authors measure skill reuse directly, rather than inferring it from downstream success rates and embedding visualizations?

Reviewer_PYBSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CARL, a contrastive learning method designed to learn a representation of local dynamics from offline datasets. CARL captures similarities in local transition structures by aligning state-goal pairs with their required action sequences to construct a state-goal representation. This representation generalizes across state spaces with comparable dynamics to enable the reuse of skills across different global contexts. When integrated with existing hierarchical reinforcement learning (HRL) methods, such as HIQL and HGCBC, CARL improves downstream performance on diverse OGBench tasks.

Critical strengths
  • The motivation is clear. The paper introduces behavioral similarity as a practical heuristic for understanding when low-level skills can be reused, and uses this idea to align state–goal pairs with action sequences. Additionally, the qualitative visualizations effectively show that pairs associated with similar behaviors cluster together in the learned representations.

  • The paper is clearly written and easy to follow. It first defines dynamics bisimilarity, then introduces behavioral similarity as a practical relaxation for identifying reusable skills. This makes the motivation for CARL easy to understand.

Critical weaknesses
  1. The proposed method aims to identify and reuse local behaviors, which is conceptually related to offline skill learning approaches such as OPAL [1] and HILP [2]. Although the experiments show improvements over HIQL and HGCBC, they do not compare CARL with these related approaches. As a result, it remains unclear whether reusing local behaviors offers an advantage over learning temporally extended skills from offline data.

  2. CARL is designed to identify local behaviors that can be reused across different contexts. The evaluation could be strengthened by including OGBench’s locomotion stitch tasks, which require agents to combine short trajectory segments to reach distant goals. Results on these tasks could help assess whether CARL can reuse local behaviors in this setting and whether its gains over HIQL and HGCBC persist.

  3. On page 5, HGCBC is introduced as a baseline, but the citation points to Park et al. (2023), which also uses it as a baseline. Please cite the original source of HGCBC.

[1] Ajay, Anurag, et al. "OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning" 2021.

[2] Park, Seohong, et al. “Foundation Policies with Hilbert Representations.” 2024.

Questions for authors
  1. How does CARL compare with an offline skill learning method that learn temporally extended skills under a comparable evaluation setting?

  2. Could the authors evaluate CARL on OGBench’s locomotion stitch datasets to assess whether it can reuse local behaviors to connect short trajectory segments and reach distant goals, and whether its gains over HIQL and HGCBC persist? Results on the AntMaze stitch or PointMaze stitch datasets would directly address this question.

  3. The paper attributes CARL’s limited gains on antmaze-teleport to stochastic portal dynamics. However, CARL’s reported mean performance is higher on pointmaze-teleport, which also includes stochastic portals. Could the authors discuss what might explain this difference?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.