- Rating
- 4
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper introduces hyperbolic distance metrics into contrastive reinforcement learning (CRL), providing a strong inductive bias for goal-reaching tasks. The benefits of this extension are particularly pronounced in maze-like environments, where long-horizon goal-reaching problems exhibit tree-like structures, as demonstrated in the paper. Interestingly, hyperbolic CRL also outperforms standard Euclidean CRL on the majority of other evaluated tasks, including those without obvious maze-like structures.
Although the paper's scope is limited to contrastive RL, it provides strong quantitative and qualitative evidence supporting its central claims. Specifically, the experiments demonstrate that (1) hyperbolic CRL generally outperforms Euclidean CRL, (2) hyperbolic geometry is particularly effective in maze environments because it explicitly introduces a tree-like inductive bias, and (3) hyperbolic distances improve experience stitching, enabling more effective reuse of previously collected trajectories.
The experimental evaluation is comprehensive, covering two important goal-conditioned RL benchmarks: JaxGCRL and OGBench. The strong results achieved under the standard evaluation settings of these widely used benchmarks provide compelling evidence of the proposed method's effectiveness. The paper is also exceptionally well presented, with a clear narrative and carefully designed figures that communicate the main insights effectively. In particular, the visualizations are sufficiently informative that even readers who only skim the figures can gain a substantial understanding of the paper's contributions.
Critical strengths
- The paper convincingly demonstrates that a simple modification—replacing Euclidean distances in CRL’s contrastive objective with geodesic distances in hyperbolic space—leads to substantial performance improvements across a wide range of environments in both online and offline goal-conditioned RL.
- The paper’s clear narrative and well-designed, intuitive figures make its main claims and supporting evidence easy to understand.
Critical weaknesses
None
Questions for authors
- While the paper examines only contrastive RL, the title is more general (GCRL) and perhaps a bit vague. While it's not a reason to reject the support, I recommend rewriting the title or providing additional experiments to provide evidence that hyperbolic distances improve not only GCRL
- Perhaps it would be helpful to visualize how Euclidean space differs from hyperbolic space for readers who are not familiar with hyperbolic geometry. Maybe add such a visualization to Appendix B
- Can the sectional curvature, −c, be turned into a trainable parameter?
2 comments
I thank the authors for addressing my questions and will maintain my positive rating of the paper.
Response to Reviewer 66Kw
Dear Reviewer 66Kw,
Thank you for your very positive and thoughtful review. Changes in response to your comments are marked in green in our revised manuscript.
Concrete steps we took based on your feedback:
Q1: The title is more general than the CRL scope
We agree and changed the title to "Outperforming 1000 Euclidean Layers With Hyperbolic Contrastive RL" in the revised manuscript. It now matches the abstract, which already frames our method as hyperbolic CRL. We now also use CRL instead of goal-conditioned RL in the introduction's "central question" and the related work.
Q2: Visualizing Euclidean vs. hyperbolic space
Thank you for the suggestion. We added Appendix B, Figure 12, which shows how the shortest path between the same two points in the Euclidean and hyperbolic planes differs, and how the magnitude of c affects the geometry.
Q3: Can the curvature be learned?
The curvature can be learned, but this is challenging in deep RL, as we mentioned in our limitations. We follow [1], who learn the curvature for contrastive image-text alignment by optimizing logc as an unconstrained parameter. We clamp the curvature c to [0.01,1]. We initialize it with 0.5 and 0.02, and share it between the critic's state-action and goal encoders. On both mazes and Humanoid, the learned c decreases and settles near or at the lower bound. At such small curvature, the agent performs like the Euclidean baseline because the space is nearly flat. On Arm Push Easy, the learned c is noisy and varies across seeds. On the mazes, learnable curvature removes the gain of the fixed c=0.5, although our curvature ablation (Figure 9) shows that maze performance improves with c larger than the learned value. While this suggests that an environment-dependent curvature would improve performance, standard gradient descent fails to find it. We added the experiments to Appendix E.5 and refined our limitations. The table below shows the mean of the last 10 evaluations ± standard error.
[1] Yoshikawa and Matsubara. PHyCLIP: ℓ1-Product of Hyperbolic Factors Unifies Hierarchy and Compositionality in Vision-Language Representation Learning. ICLR 2026.