- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This work propose CARL that learns state-goal representation thru contrastive alignment with associated k-step action sequences. That is to say, if from s1 to g1 requires similar actions as s2 to g2, then (s1, g1) and (s2, g2) are represented similarly in the representation space. Building on the hypothesis that "behavioral similarity will naturally emerge when many states share local dynamics structure", the authors propose to use contrastive loss (InfoNCE) to train the embeddings on an offline dataset.
Several key assumptions are worth noting: (1) deterministic transition; (2) tasks are goal oriented and goals are a state in the state space; (3) offline dataset is collected by an expert policy follow optimal strategies (all optimal strategies equally).
Critical strengths
- The presentation is good visually, and in general easy to follow.
- The implementation details and justification choice of hyperparameter is provided, adding up to the credibility of the result. The ablation study is useful to understand the algorithm.
- Some detailed analysis is provided in Appendix.
Critical weaknesses
Major
- the paper is restricted to a limited, deterministic control setting, and subgoals are within the state space. I appreciate that the authors acknowledge that, though. But to me, many things can be simplified, as the transition probability is either zero or one, and many things may be reduced to a trivial/highly restrictive version. Although the authors propose to generalize to a stochastic setting, the proposed method may not work well with the general stochastic case--where the same action sequence from one state may lead to diverse outcomes. The proposed method might be brittle if any of those assumptions break.
- Sec 3: The k-ball MDP is not well-defined: consider some state s'' that is k+1 steps reachable from s. That means there is some state, say s', is k-step reachable from s. Then there is a possibility of taking some action to transition from s' to s'' in the original MDP. But that state is not even in your k-ball MDP's state space. Then, the transition kernel should not be the original one. So please specify the transition function and reward.
- The definition of Dynamics Bisimilarity is also strange to me. I don't understand what is the quotient set when B is on . Usually a quotient set is defined on an equivalence relation on one set. Then, if I'm not missing something, definition 3.3 is broken.
- Overall, I think section 3.2 should go through a major revision. To be more constructive, some examples with a few states throughout the how section could help.
- Sec.4: the connection of the proposed method is indirectly related to what is proposed in sec.3. There are few direct references to any definitions in sec.3. In sec.3 definitions, it is not related to goals either. To summarize a bit, sec.3 compares states through local transition structure, while sec.4 compares state-goal pairs through dataset action distributions. Then why is the story not directly "if the state-goal shares a similar (optimal) control action sequence (inverse dynamics), then they share similar embeddings"?
- line 216: "we can recover a bisimulation metric" I'm doubtful about this -- it is not true if you mean the bisimulation metrics by Fern et al., 2004. In general, "if they share the same optimal action sequences" -> "recover bisimulation metric" is false: you can have two trajectories from two states s1 and s2, that have the same optimal actions but totally different rewards. In this case, s1 and s2 are not bisimilar (and the metric > 0). It's closer to the work by Hansen-Estruch et al., 2022, but you need more justification and theory to make such claims. Please also define the metrics formally. Even if the theory goes through, many recent papers show the difficulty of "recovering the exact metrics" (Kemertas and Aumentado-Armstrong, 2021; Zang et al., 2023; Luo et al., 2025). So at least it should be toned down.
- experiment: In sec.5.1 and fig.2, I can only see a sketch of the diagnostic experiment. If I'm not missing any description in the Appendix, then to me it's not sufficient to understand the exact setting of the experiment. And any of those rooms only requires memorizing the successful action sequence -- many methods in offline skill discovery (see the survey, sec.5, by Klissarov et al., 2025) may have the same promise, but the comparison is only limited to HIQL, which mainly focuses on credit assignment. It does not seem enough for generalization, like, in many scenarios we requires agent can generalize to different maps with different successful paths. And, as a diagnostic setting, I would expect to see more diagnostic results other than just "who succeeds or performs better".
- the scaled result is largely limited to OGBench. The baselines are limited to HIQL/HGCBC, with insufficient justification for why to choose those methods. There are many related HRL works, or representation learning methods (like methods in sec.5, by Klissarov et al., 2025, or related work in the work by Park et al., 2024) that work on this setting and are worth comparison. There is also no standalone CARL (alg.1) result in the experiments either (then why you write alg.1).
Minor:
- 3.1 should define the dataset D as well.
- inconsistency in presentation still exists, such as mixing "state-goal" and "state--goal"
- the citation of previous work (in HRL, offline skill discovery, action chunking, metric learning, contrastive representation learning, inverse dynamics, even for state/state-action abstraction) is not comprehensive enough. E.g., for bisimulation, at least the work by Givan et al., 2003 should be cited. Talking about related work, I also have some concerns about the novelty of the work, although this is subjective. For succinctness, I'll not expand on this point.
- And many description in terms of bisimulation isn't accurate, e.g.: "Prior work adapting bisimulation to RL measures similarity between MDPs through sequences of rewards or single-step controllability metrics under identical action sequence" which is not true as many those cited works consider behavioral distance of states in one same MDP.
- line 229: " sqeuence" -> "sequence". I didn't go through a thorough check, but please fix those typos.
- Sec.5.4 only shows some working examples but without statistical significance. I suggest merging this section into sec.5.3 to save space.
Reference
Castro et al. (2021). MICo: Improved representations via sampling-based state similarity for Markov decision processes. NeurIPS.
Ferns et al. (2004). Metrics for finite Markov decision processes. UAI.
Ferns et al. (2011). Bisimulation metrics for continuous Markov decision processes. SIAM Journal on Computing.
Givan et al. (2003). Equivalence notions and model minimization in Markov decision processes. Artificial Intelligence.
Hansen-Estruch et al. (2022). Bisimulation makes analogies in goal-conditioned reinforcement learning. ICML.
Kemertas and Aumentado-Armstrong (2021). Towards robust bisimulation metric learning. NeurIPS.
Klissarov et al. (2025). Discovering temporal structure: An overview of hierarchical reinforcement learning. arXiv:2506.14045.
Luo et al. (2025). Understanding behavioral metric learning: A large-scale study on distracting reinforcement learning environments. Reinforcement Learning Journal.
Park et al. (2024). Foundation policies with Hilbert representations. ICML.
Zang et al. (2023). Understanding and addressing the pitfalls of bisimulation-based representations in offline reinforcement learning. NeurIPS.
Questions for authors
- As addressed above, the definition of k-ball MDP and Dynamics Bisimilarity seems handwavy. Can you explain more or fix the definition to minimize my abovementioned concerns?
- Please clarify the connection between the contrastive loss and the definitions in sec.3.
- Please justify the choice of baseline methods.
- Seem like you are using the InfoNCE loss to train action and s,g embedding simultaneously. Could the encoders minimize the loss by learning arbitrary matching codes rather than capturing behavioral similarity? What prevents similar action sequences from being mapped far apart?
- Could you provide more details of sec.5.1 experiment? Do you use the same hyperparameter settings as in later sections?
- In fig.3, what are the goals exactly? The direction, or some state in the state space? Where are the states sampled from?