acceptodds
Under review as a conference paper at ICLR 2027

Outperforming 1000 Euclidean Layers With Hyperbolic Goal-Conditioned RL

30 views

Abstract

Scaling self-supervised contrastive reinforcement learning (CRL) to 1000-layer Euclidean neural networks has enabled state-of-the-art goal-reaching performance, yet wall-clock time scales linearly with depth. We find that changing the geometry achieves stronger performance with far shallower networks. This matters in complex navigation environments, where walls can make route distance deviate substantially from Euclidean distance. In a U-shaped maze, we show that the longer the route, the more it deviates from Euclidean distance. Hyperbolic CRL addresses this mismatch by replacing Euclidean distances in CRL's contrastive objective with geodesic distances in hyperbolic space. On hard navigation tasks, inductive bias beats scale: depth-64 hyperbolic agents outperform 1024-layer-deep Euclidean networks while training more than faster and using about fewer parameters. We observe that hyperbolic agents behave qualitatively better in hard humanoid mazes, where they follow corridors to reach goals, whereas Euclidean agents try straight-line shortcuts over walls. Our results show that matching the representation geometry to the task can substitute for substantial network depth.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, contrastive reinforcement learning, representation learning, hyperbolic space, hyperbolic deep learning

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 4.00 over 3 reviews, above 99% of the others.
Reviewer_66KwOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces hyperbolic distance metrics into contrastive reinforcement learning (CRL), providing a strong inductive bias for goal-reaching tasks. The benefits of this extension are particularly pronounced in maze-like environments, where long-horizon goal-reaching problems exhibit tree-like structures, as demonstrated in the paper. Interestingly, hyperbolic CRL also outperforms standard Euclidean CRL on the majority of other evaluated tasks, including those without obvious maze-like structures.

Although the paper's scope is limited to contrastive RL, it provides strong quantitative and qualitative evidence supporting its central claims. Specifically, the experiments demonstrate that (1) hyperbolic CRL generally outperforms Euclidean CRL, (2) hyperbolic geometry is particularly effective in maze environments because it explicitly introduces a tree-like inductive bias, and (3) hyperbolic distances improve experience stitching, enabling more effective reuse of previously collected trajectories.

The experimental evaluation is comprehensive, covering two important goal-conditioned RL benchmarks: JaxGCRL and OGBench. The strong results achieved under the standard evaluation settings of these widely used benchmarks provide compelling evidence of the proposed method's effectiveness. The paper is also exceptionally well presented, with a clear narrative and carefully designed figures that communicate the main insights effectively. In particular, the visualizations are sufficiently informative that even readers who only skim the figures can gain a substantial understanding of the paper's contributions.

Critical strengths
  • The paper convincingly demonstrates that a simple modification—replacing Euclidean distances in CRL’s contrastive objective with geodesic distances in hyperbolic space—leads to substantial performance improvements across a wide range of environments in both online and offline goal-conditioned RL.
  • The paper’s clear narrative and well-designed, intuitive figures make its main claims and supporting evidence easy to understand.
Critical weaknesses

None

Questions for authors
  • While the paper examines only contrastive RL, the title is more general (GCRL) and perhaps a bit vague. While it's not a reason to reject the support, I recommend rewriting the title or providing additional experiments to provide evidence that hyperbolic distances improve not only GCRL
  • Perhaps it would be helpful to visualize how Euclidean space differs from hyperbolic space for readers who are not familiar with hyperbolic geometry. Maybe add such a visualization to Appendix B
  • Can the sectional curvature, −c, be turned into a trainable parameter?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 66Kw

Dear Reviewer 66Kw,

Thank you for your very positive and thoughtful review. Changes in response to your comments are marked in green\color{#1B7A1B}{\textbf{green}} in our revised manuscript.

Concrete steps we took based on your feedback:

  • Changed the title to "Outperforming 1000 Euclidean Layers With Hyperbolic Contrastive RL" and now use CRL instead of goal-conditioned RL in the central question and the related work (title and Section 3 in green\color{#1B7A1B}{\text{(title and Section 3 in green}}, central question in blue)\color{#0033CC}{\text{central question in blue)}}
  • Added a figure that contrasts Euclidean and hyperbolic geometry and shows the effect of the curvature magnitude cc (Appendix B, Figure 12, green coloring)\color{#1B7A1B}{\text{(Appendix B, Figure 12, green coloring)}}
  • Ran experiments with a learnable curvature and refined our limitations accordingly (Appendix E.5, Figure 21, Section 6, green coloring)\color{#1B7A1B}{\text{(Appendix E.5, Figure 21, Section 6, green coloring)}}

Q1: The title is more general than the CRL scope

We agree and changed the title to "Outperforming 1000 Euclidean Layers With Hyperbolic Contrastive RL" in the revised manuscript. It now matches the abstract, which already frames our method as hyperbolic CRL. We now also use CRL instead of goal-conditioned RL in the introduction's "central question" and the related work.


Q2: Visualizing Euclidean vs. hyperbolic space

Thank you for the suggestion. We added Appendix B, Figure 12, which shows how the shortest path between the same two points in the Euclidean and hyperbolic planes differs, and how the magnitude of cc affects the geometry.


Q3: Can the curvature be learned?

The curvature can be learned, but this is challenging in deep RL, as we mentioned in our limitations. We follow [1], who learn the curvature for contrastive image-text alignment by optimizing log⁡c\log c as an unconstrained parameter. We clamp the curvature cc to [0.01,1][0.01, 1]. We initialize it with 0.5 and 0.02, and share it between the critic's state-action and goal encoders. On both mazes and Humanoid, the learned cc decreases and settles near or at the lower bound. At such small curvature, the agent performs like the Euclidean baseline because the space is nearly flat. On Arm Push Easy, the learned cc is noisy and varies across seeds. On the mazes, learnable curvature removes the gain of the fixed c=0.5c = 0.5, although our curvature ablation (Figure 9) shows that maze performance improves with cc larger than the learned value. While this suggests that an environment-dependent curvature would improve performance, standard gradient descent fails to find it. We added the experiments to Appendix E.5 and refined our limitations. The table below shows the mean of the last 10 evaluations ±\pm standard error.

EnvironmentEuclideanFixed c=0.5c = 0.5 (paper)Learned ccLearned cc at end of training
Ant U4-Maze (far goals)211.7 ±\pm 36.0421.7 ±\pm 52.0211.0 ±\pm 26.80.010 (lower bound)
Ant Hardest Maze363.1 ±\pm 15.9502.8 ±\pm 8.9392.0 ±\pm 17.60.010 (lower bound)
Humanoid530.5 ±\pm 12.1280.9 ±\pm 14.9530.1 ±\pm 39.10.018
Arm Push Easy814.7 ±\pm 13.8790.7 ±\pm 8.6792.6 ±\pm 30.10.055

[1] Yoshikawa and Matsubara. PHyCLIP: ℓ1\ell_1-Product of Hyperbolic Factors Unifies Hierarchy and Compositionality in Vision-Language Representation Learning. ICLR 2026.

I thank the authors for addressing my questions and will maintain my positive rating of the paper.

Reviewer_7xz1OpenReview ↗
Rating
4(+2)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper outlines a drop-in modification to contrastive reinforcement learning (CRL) that replaces the Euclidean distance in the CRL critic with geodesic distance in hyperbolic space. The authors demonstrate that maze route distances are approximately Euclidean at short range and increasingly tree-like at long range, which hyperbolic space can capture more faithfully than Euclidean space. This modification is evaluated on online JaxGCRL and offline OGBench environments, and the authors show that hyperbolic agents outperform Euclidean agents at equal depth on most tasks.

Critical strengths

The main strengths of this paper are its empirical results: a small drop-in modification to CRL yields large performance gains on long-horizon navigation tasks, where depth-64 hyperbolic agents outperform much deeper Euclidean agents while training much faster. The evaluation is broad, spanning many environments and offline tasks, and the authors make a reasonable attempt to explain these improvements through a case study on the Ant U5-Maze, behavioural examples, and a curvature ablation.

Critical weaknesses

The central question of whether Euclidean distance is the appropriate geometry for goal-reaching has been studied before (Gustafson & Daw, 2011, geodesic generalisation for RL; Dayan, 1993, successor representation). The second key contribution, that maze route distances are approximately Euclidean at short range and increasingly tree-like at long range, appears insufficiently distinguished from a general property of corridor-based environments. The analysis shows that route and straight-line distances differ in the Ant U5-Maze, but the novelty of this finding remains unclear. What remains is replacing the Euclidean distance in the CRL critic with a hyperbolic geodesic distance. Hyperbolic distances in contrastive objectives are established in vision (Ge et al., 2023), and hyperbolic representations have already been introduced to deep RL (Cetin et al., 2023). I am therefore not convinced that this is a large enough step beyond existing work to constitute a significant methodological novelty.

The authors report time at goal for the online experiments, following Wang et al. (2025), but not success rate as in Bortkiewicz et al. (2025). These metrics measure different things: time at goal also reflects stability at the goal location, not only goal-reaching. This matters for the main result in the Humanoid maze, that hyperbolic agents “follow corridors to reach goals, whereas Euclidean agents try straight-line shortcuts over walls. The reported performance gap could partly reflect the Euclidean agents falling on reaching the goal region, which ends the episode, rather than differences in navigation ability as the paper argues.

Questions for authors
  1. Does the improvement over Euclidean agents track how tree-like each environment is? Hyperbolicity is computed only for the Ant U5-maze, but gains are reported across many environments.

  2. If the Humanoid was restricted from vaulting over the walls, would the Euclidean agents then learn to navigate around the maze? This would help separate goal-reaching from locomotion stability in this more complex task.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 7xz1 (Part 1)

Dear Reviewer 7xz1,

Thank you for your thoughtful review. Changes in response to your comments are marked in blue\color{#0033CC}{\textbf{blue}} in the revised manuscript.

Concrete steps we took based on your feedback:

  • Added Gustafson & Daw [1] and Dayan [2] to the related work, sharpened the discussion of [3, 4], and formulated the central question and our second key contribution more precisely (Sections 1 and 3, blue coloring)\color{#0033CC}{\text{(Sections 1 and 3, blue coloring)}}
  • Repeated the route distance analysis on Ant Hardest Maze, whose corridors form loops (Section 4, Appendix E.1, Figure 16, blue coloring)\color{#0033CC}{\text{(Section 4, Appendix E.1, Figure 16, blue coloring)}}
  • Reported success rate for the main experiments and added a detailed analysis of goal-reaching on the Humanoid U-Maze and Big Maze (Appendices E.2 and E.4, Figures 17 to 20, blue coloring)\color{#0033CC}{\text{(Appendices E.2 and E.4, Figures 17 to 20, blue coloring)}}
  • Related the tree-likeness of all six navigation environments to the performance gain (Section 5.2, Figure 5, Appendix E.6, blue coloring)\color{#0033CC}{\text{(Section 5.2, Figure 5, Appendix E.6, blue coloring)}}
  • Prevented vaulting on Humanoid U-Maze and Big Maze, retrained both agents, and reported steps at goal and success rate (Appendix E.3, Table 3, blue coloring)\color{#0033CC}{\text{(Appendix E.3, Table 3, blue coloring)}}

W1a: The central question has been studied before [1, 2]

Thank you for these references. We added [1, 2] to Section 3, sharpened our discussion of [3, 4], and formulated our contribution more precisely. The central question now reads: "Is Euclidean distance the right geometry for contrastive reinforcement learning?"

[1, 2] establish that value generalization should follow route distance [1] or future state occupancy [2] instead of straight-line distance. CRL is consistent with this view, since its critic estimates a continuous analog of the discounted future occupancy in [2] (Equation 1). Our question concerns the learned embedding space in which the critic represents this occupancy as a distance. [1, 2] fit linear value functions on hand-built [1] or tabular [2] features and do not use a learned embedding distance. In CRL, the actor optimizes this distance directly, so its geometry affects the agent. To our knowledge, we are the first to use hyperbolic representations in CRL (Section 3).


W1b: Tree-like route distances are a general property of mazes with corridors

Starting from this property, Section 4 shows that a Euclidean critic captures little of the tree-like structure of mazes, whereas a hyperbolic critic of the same depth captures it far more accurately. [5] compensates for this with much deeper networks.

For goal pairs with similar straight-line distance but different route distance, the Euclidean critic ranks the longer route as farther in 59.4% of comparisons, close to chance (50%). In contrast, the hyperbolic critic ranks 82.5% correctly (Figure 2c). The effect is also not specific to U-shaped corridors. In Ant Hardest Maze, whose corridors form loops, the same comparison favors hyperbolic (95.6%) critics over Euclidean (67.4%) ones (Appendix E.1).

Across all six navigation environments, the performance gain grows with the tree-likeness of the route distances (see Q1). Even the two mazes with loops show gains. To our knowledge, neither result has been reported before.


W1c: Replacing the distance is not a large enough methodological step beyond [3, 4]

In CRL, the choice of distance determines what the actor optimizes. Our critic is the negative hyperbolic distance, and the actor directly maximizes this quantity (see W1a). This distinguishes our method from both [3], which has no policy, and [4], which learns a policy and a value function from hyperbolic features.

Changing only the distance preserves CRL's contrastive objective, actor-critic framework, and theory, and lets us attribute the gains to the critic's geometry (Section 5.5). Section 4 and the empirical results show that this choice can substitute for much of the network depth used in [5]. On Humanoid U-Maze, our depth-64 hyperbolic agent outperforms the 1024-layer Euclidean agent while training 12x faster (Figure 1 and Table 5).


[1] Gustafson & Daw. Grid cells, place cells, and geodesic generalization for spatial reinforcement learning. PLoS Comput. Biol., 2011.

[2] Dayan. Improving generalization for temporal difference learning: The successor representation. Neural Computation, 1993.

[3] Ge et al. Hyperbolic contrastive learning for visual representations beyond objects. CVPR 2023.

[4] Cetin et al. Hyperbolic deep reinforcement learning. ICLR 2023.

[5] Wang et al. 1000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilities. NeurIPS 2025.

AuthorsOpenReview ↗

Response to Reviewer 7xz1 (Part 2)

W2: Time at goal vs. success rate

Appendix E.2 now reports success rate for our main experiments. The ranking of the two agents changes only on Humanoid, where the hyperbolic deficit is now a tie. The hyperbolic agents are still substantially better on both Humanoid mazes. Because success rate ignores everything after arrival, the gap between the Euclidean and hyperbolic agents cannot come from falling at the goal.

Appendix E.4 breaks down the success rate by goal distance. The table below shows success rate over 10 seeds at depth 16, with 20 evaluation episodes per goal and seed. Each row averages the per-goal success rates of Appendix E.4. We group goals into equal thirds of each maze's maximum route distance from the start. Both agents reach most close goals. In the intermediate and far thirds, the Euclidean agent succeeds in at most 10% of episodes. In contrast, the hyperbolic agent still reaches intermediate goals in more than a third. On Humanoid U-Maze, every Euclidean success beyond three cells is a vault over the wall.

MazeGoals (route distance in cells)EuclideanHyperbolic
Humanoid U-MazeClose (1–2)60.0 %85.0 %
Humanoid U-MazeIntermediate (3–4)10.0 %36.5 %
Humanoid U-MazeFar (5–6)5.0 %0.0 %
Humanoid Big MazeClose (1–3)72.5 %87.5 %
Humanoid Big MazeIntermediate (4–7)3.7 %37.8 %
Humanoid Big MazeFar (8–11)0.0 %4.0 %

Q1: Does the improvement track how tree-like each environment is?

Yes. Thank you for this insightful question. In response, we added a new analysis to the main text (Section 5.2), which is now part of our second key contribution. We now compute the relative δ\delta-hyperbolicity δrel\delta_{\mathrm{rel}} of the route distances (lower is more tree-like) for all six navigation environments and compare it with the performance gain in our main results (depth 16, 10 seeds), as shown in the table below. We exclude Ant Ball and the three Arm environments because their goals are object positions. How far such a goal is depends on the agent's position and the object, which route distance cannot express.

We measure performance gain as the hyperbolic mean minus the Euclidean mean. The Spearman rank correlation between δrel\delta_{\mathrm{rel}} and the gain is ρ=−0.94\rho = -0.94 for both steps at goal and success rate. This is significant, with p=0.017p = 0.017 for a two-sided permutation test. Thus, the more tree-like the environment, the larger the performance gain of the hyperbolic agent over the Euclidean agent (Figure 5, Appendix E.6).

EnvironmentLayoutδrel\delta_{\mathrm{rel}}Gain in steps at goalGain in success rate (points)
Ant U5-Mazecorridor maze0.08+248+41.4
Ant U4-Mazecorridor maze0.16+210+29.1
Humanoid U-Mazecorridor maze0.21+219+17.1
Ant Hardest Mazemaze with loops0.41+140+26.1
Humanoid Big Mazemaze with loops0.44+80+16.3
Humanoidopen arena0.58−250−1.0

Q2: Would Euclidean agents learn to navigate the corridors on Humanoid mazes without the ability to vault?

No, they do not learn proper navigation. We raised the walls of both Humanoid mazes from 1 m to 3 m and retrained both agents using 5 seeds. Episodes terminate once the torso position is above 2 m, so these walls cannot be crossed. The Euclidean success rate does not rise once vaulting is impossible, so vaulting does not explain the gap. Regarding locomotion stability, the Euclidean agent's steps at goal rise slightly on U-Maze. The table below shows mean ±\pm standard error, which we added in Appendix E.3.

MazeAgentSteps at goal (1m)Steps at goal (3m)Success rate (1m)Success rate (3m)
U-MazeEuclidean8.0 ±\pm 3.328.0 ±\pm 14.422.9 ±\pm 4.816.4 ±\pm 1.5
U-MazeHyperbolic244.9 ±\pm 49.5224.2 ±\pm 59.944.9 ±\pm 5.444.1 ±\pm 8.7
Big MazeEuclidean5.8 ±\pm 1.43.8 ±\pm 1.015.6 ±\pm 3.011.4 ±\pm 2.8
Big MazeHyperbolic118.7 ±\pm 40.8160.0 ±\pm 37.335.4 ±\pm 5.637.8 ±\pm 3.4

Dear Authors,

Thank you for the rebuttal. The additional analyses have definitely improved the manuscript. I would ask for one more addition: to include an extra figure in E1/Figure 16 which shows the environment layout of Ant Hardest Maze just so readers can visually compare both 1) with Ant U5 Maze and 2) with different route distances in this maze.

The authors have addressed my concerns sufficiently, so I'll raise the score to a 4.

Dear Reviewer 7xz1,

Thank you for the follow-up and for raising your score. Your review helped to further improve our paper. The cross-environment analysis in Section 5.2, which tests whether the hyperbolic gains track tree-likeness, originated from your comments. We consider it a central part of the revised manuscript.

Further, we have added the layout of Ant Hardest Maze to Figure 16 in Appendix E.1, next to the layout of Ant U5 Maze. Readers can now compare the two mazes directly and relate the route distances to the layout.

Reviewer_BiBsOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work investigates whether changing the geometry of learned representations from Euclidean to Hyperbolic can reduce the extreme network depth reportedly required by contrastive goal-conditioned reinforcement learning (CRL). The authors argue that Euclidean embedding distance is poorly matched to navigation environments with obstacles as it reflects straight-line proximity whereas successful control depends on route distance (which is increasingly tree-like over long horizons). They introduce Hyperbolic CRL, a simple drop-in modification to CRL that maps state-action and goal embeddings to a hyperboloid and replaces negative Euclidean distance in the contrastive critic with negative hyperbolic geodesic distance. Across locomotion and maze-navigation tasks, hyperbolic agents generally outperform Euclidean agents at matched depth.

Critical strengths
  • Impact: The paper proposes a minimal modification to the critic’s representation and distance function that is both intuitive to understand and appears substantially more compute-efficient than scaling Euclidean models to hundreds or thousands of layers.
  • Strength of empirical analysis: The method is evaluated across several agents and tasks spanning locomotion, navigation, and manipulation, with 10 seeds and bootstrap confidence intervals.
Critical weaknesses
  • The 512/1024 layer Euclidean curves in Figure 1 are taken from Wang et al. 2025 rather than re-run; Appendix C.3 and Table 1 document protocol differences (Brax version/timestep, num_epochs, num_eval_envs). Baselines should be trained under the same protocol before the title claim can be supported.

  • Curvature c is swept per environment but Euclidean baseline gets no analogous per-task tuning

Questions for authors
  • how was c = 0.5 chosen as the default?
1 comment
AuthorsOpenReview ↗

Response to Reviewer BiBs

Dear Reviewer BiBs,

Thank you for your positive review. We have updated the paper in line with your comments. Changes are marked in brown\color{#8B4513}{\textbf{brown}} in the revised manuscript. Below, we also address the other comments in your review.

Concrete steps we took based on your feedback:

  • Retrained both Euclidean baselines of Figure 1 under our protocol with 5 seeds (Figure 1, Section 5.1, Reproducibility statement, brown coloring)\color{#8B4513}{\text{(Figure 1, Section 5.1, Reproducibility statement, brown coloring)}}
  • Changed Figure 1 to plot steps at goal against measured training time in H100 hours (Figure 1, brown coloring)\color{#8B4513}{\text{(Figure 1, brown coloring)}}
  • Trained depth-64 Euclidean and hyperbolic agents for 400M steps on all Humanoid mazes (Section 5.5, Figure 10, brown coloring)\color{#8B4513}{\text{(Section 5.5, Figure 10, brown coloring)}}
  • Stated that the curvature magnitude c=0.5c = 0.5 is fixed for all online tasks and documented how we selected it (Section 5, Appendix D.1, brown coloring)\color{#8B4513}{\text{(Section 5, Appendix D.1, brown coloring)}}

W1: Figure 1 baselines are taken from Wang et al. [1]; training protocol

We retrained both Figure 1 baselines using our protocol and the step budgets of [1], with 5 seeds each. Figure 1 now plots steps at goal against measured training time in H100 hours, with 95% bootstrap confidence intervals for both agents. We report the mean over the final 10 evaluation epochs. The Brax timestep difference does not affect the Humanoid mazes (Appendix C.3), and the mentioned parameters pertain to evaluation, not training (Table 1).

On Humanoid U-Maze, the retrained 1024-layer Euclidean agent reaches 36.4 steps at goal, compared to 286.5 for our depth-64 hyperbolic agent, which trains about 12x faster (Table 5). The title claim therefore holds under our protocol, and also holds against the 81.5 reported in [1].

On Humanoid Big Maze, our depth-64 hyperbolic agent matches the retrained 512-layer Euclidean agent while training 7x faster (Figure 1, Table 5). At 100M steps, both agents spend under 1% of each episode at the goal, so we also trained Euclidean and hyperbolic agents at matched depth 64 for 400M steps on the Humanoid mazes (Figure 10).

Hyperbolic agents outperform Euclidean ones in all Humanoid mazes (table below, mean ±\pm standard error over 5 seeds). We could not train the 512- and 1024-layer Euclidean agents to 400M steps because five seeds of each would require 121 H100-days in total.

MazeEuclideanHyperbolic
Humanoid U-Maze62.5 ±\pm 51.8465.9 ±\pm 25.9
Humanoid Big Maze87.7 ±\pm 33.1288.5 ±\pm 20.0
Humanoid Hardest Maze3.8 ±\pm 0.735.2 ±\pm 17.9

W2 & Q: Curvature tuning and the choice of c=0.5c = 0.5

We do not tune the curvature magnitude cc per task. All online results use c=0.5c = 0.5 for every task, and the offline results likewise use a single fixed curvature for every task (Table 2). The revised manuscript states this explicitly now. The Euclidean and the hyperbolic agents use identical hyperparameters (Table 1), which prior work already tuned for the Euclidean agent: [2] tuned the energy function, contrastive objective, and update-to-data ratio, and [1] tuned the depth, batch size, and actor vs. critic depth. We did not re-tune this configuration for the hyperbolic agent. We selected c=0.5c = 0.5 from an initial 5-seed grid search over c∈{0.1,0.3,0.5}c \in \lbrace 0.1, 0.3, 0.5 \rbrace on Ant U4-Maze, Humanoid, Ant Ball, and Arm Push Hard, and then fixed it for all tasks. Appendix D.1 now describes this procedure. We ran the sensitivity analyses in Figures 9 and 23 only after making this choice. In them, a value other than c=0.5c = 0.5 reaches more steps at goal on Ant U4-Maze, Humanoid, and Arm Push Easy. Therefore, a task-specific choice would have selected different values. On Ant U4-Maze, Ant Hardest Maze, and Ant Ball, the hyperbolic agent outperforms the Euclidean agent at every cc from 0.2 to 1.0 (Figure 23).


[1] Wang et al. 1000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilities. NeurIPS 2025.

[2] Bortkiewicz et al. Accelerating Goal-Conditioned RL Algorithms and Research. ICLR 2025.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Our Findings and Revisions

Dear AC and Reviewers,

Thank you for your time, thoughtful feedback, and constructive reviews.

Our central finding is that aligning the representation geometry to the task structure can substitute for substantial network depth. Wang et al. [1] showed that scaling contrastive RL to 1000 layers enables new goal-reaching capabilities. We show that changing the geometry of the critic's distance metric achieves even stronger results at a fraction of the depth: With a hyperbolic critic, a depth-64 agent outperforms their 1024-layer agent on Humanoid U-Maze and trains about 12x faster. How much the hyperbolic critic helps depends on the geometry of the environment. Across six navigation environments, the more tree-like the environment, the larger the performance gain (Spearman rank correlation ρ=−0.94\rho = -0.94). To our knowledge, we are the first to measure tree-likeness directly in RL environments and the first to relate it to agent performance.

The revised manuscript addresses every point the reviewers raised. In particular, we:

  • Reviewer BiBs: Retrained the deepest Euclidean baselines under our protocol. The title claim holds. When training longer at matched depth 64, the hyperbolic agent outperforms the Euclidean agent on all three Humanoid mazes.
  • Reviewer 7xz1: Added the analysis relating tree-likeness to performance gains and reported the success rate.
  • Reviewer 66Kw: Scoped the title and the central question to contrastive RL.

[1] Wang et al. 1000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilities. NeurIPS 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.