- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper under review introduces an contrastive learning approach called angular reachability representation (ARR) that learns representations for goal-conditioned RL. Specifically, goal and state are embedded onto a unit sphere and their cosine-simularity is used as state-goal compariability metric in an InfoNCE-type loss. The method is benchmarked on selected navigation tasks from OGBench.
Critical strengths
- Generally interesting idea to learn representations for navigation problems
- Well written and mathematically solid paper with technical details, inuition, and motivation laid out transparently.
Critical weaknesses
- My major concern is about the scope and applicability of the proposed method. Although I find the idea interesting, I'm concerned that it is too adjusted for navigation-tasks. The paper provides no theoretical nor empirical indications why ARR should generalize to more and other RL tasks. Smaller experiments on
scene-play-v0indicate that the method will not generalize. - I miss a structured theoretical analysis of the method. For instance, what are assumptions on the environment for ARR to learn the optimal policy? Is it possible to derive regret bounds when focusing on navigation tasks? There, one may also be able to derive condidtions on the environment in order for ARR to work (or to fail).
- The breadth of experiments is too narrow and focused on navigation to assess the impact of the method beyond navigation in an antmaze.
- Although the angular embedding idea appears novel to me, the theoretical heavy lifting stems from InfoNCE and thus the theoretical contribution seems only modest to me.
Questions for authors
I find the paper is not ready for publication yet. The following questions are indented to help the authors to strengthen the paper for a future submission:
- What are theoretical conditions on the environment for ARR to be successful?
- Although I do not believe ARR to generalise beyond maze navigation, can ARR generalise beyond the given maze? How many mazes does ARR need to generalize to any maze?
- ARR could be beneficial under high-dimensional image observations where the problem underneath requires steering in some vector field, like in the problems studied in [1] or [2].
[1] Sorokin et al, Interferobot: aligning an optical interferometer by a reinforcement learning agent, NeurIPS'20
[2] Schmähling et al., Trajectory-Level Data Augmentation for Offline Reinforcement Learning, ICML'26
Thank you for your constructive suggestions and guidance. We would welcome further discussion, regardless of the final acceptance decision, as we believe your feedback will be valuable for our future research.
Although our paper provides a theoretical framework, ARR does not achieve the best performance on every task, including
scene-play-v0, and we have reported these limitations transparently. As discussed in our paper, ARR learns reachability induced by the offline data. This objective does not, by itself, identify a policy that reaches goals in the minimum time or along the shortest path. Nevertheless, Monte Carlo contrastive methods can offer stability and long-horizon learning benefits, while TD-based methods can suffer from compounding errors over long horizons, as discussed by Myers et al. [1]. ARR’s effectiveness also depends on the quality and coverage of the offline data. We consider these factors a possible explanation for its stronger performance on navigation tasks, although further investigation is needed. Our future work aims to extend ARR to learn optimal temporal-distance representations, for example by imposing triangle-inequality constraints. We also plan to explore integrating TD learning and Monte Carlo contrastive learning within the ARR framework.Different mazes have different obstacle layouts: identical coordinates can correspond to very different traversable distances. We therefore believe that generalizing from a single maze to unseen mazes is challenging for offline GCRL without additional data or information about the new layouts. Although aspects of the underlying dynamics may transfer, changes in the environment’s connectivity make direct generalization difficult. So, we also view access to diverse, carefully curated, high-quality data as an important bottleneck for embodied learning.
We believe ARR may have potential for tasks with high-dimensional image observations. Exploring this direction would involve additional considerations, including visual feature extraction, network architecture, and the data augmentation techniques you suggested. We appreciate you bringing the recent augmentation work to our attention, as we had previously overlooked it. These considerations broaden the experimental scope, and visual observations were therefore not the initial focus of our study.
We find it interesting to give learned representations an explicit geometric interpretation and reuse that structure for test-time planning, and we appreciate your recognition of this direction. We believe that both ARR’s representation-learning stage and its policy-extraction stage offer opportunities for further investigation and improvement, as discussed in Limitations and Future Work. We believe that no single reinforcement learning algorithm can excel across all tasks and settings, which helps explain why the field remains active and continues to evolve. Different tasks and environments often require different algorithms and reward designs. Developing a broadly generalizable foundation policy is therefore a shared goal for many researchers, including ourselves. Therefore, while ARR leaves some room for further exploration, we believe that its strong performance in specific settings and its approach to linking representation geometry, value functions, and test-time planning constitute a worthwhile contribution.
[1] Offline Goal-Conditioned Reinforcement Learning with Quasimetric Representations. NeurIPS 2025.