acceptodds
Under review as a conference paper at ICLR 2027

ANGULAR REACHABILITY REPRESENTATIONS FOR OFFLINE GOAL-CONDITIONED REINFORCEMENT LEARNING

Abstract

Offline goal-conditioned reinforcement learning (GCRL) provides a promising framework for learning generalizable goal-reaching policies from large-scale offline trajectory datasets. However, existing approaches often suffer from inadequate modeling of state-goal relationships, leading to suboptimal policy learning and limited deployment accuracy. In this paper, we introduce a novel contrastive representation learning framework for offline GCRL based on unit hyperspherical state-goal embeddings. Our approach learns structured representations of states and goals on two unit hyperspheres, where the reachability between a state and a goal is characterized by their angular relationship. Specifically, this reachability measure can be interpreted as a goal-conditioned value function defined by the future-state occupancy measure, and is equivalently represented by the cosine similarity between the corresponding unit embeddings. We refer to these embeddings as Angular Reachability Representations (ARR). Building upon the learned representations, we propose a novel offline reinforcement learning algorithm that performs policy optimization based on ARR, improving data utilization and mitigating distribution shift. Furthermore, we introduce a test-time planning strategy that leverages ARR to refine action selection during deployment without additional training. Experiments on OGBench demonstrate the effectiveness of the proposed representation, policy-learning method, and test-time planner across offline goal-conditioned control tasks.

Primary area:
reinforcement learning
Keywords:
Offline GCRL, Contrastive Learning, Unit Hyperspherical Representations

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_8uqwOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper under review introduces an contrastive learning approach called angular reachability representation (ARR) that learns representations for goal-conditioned RL. Specifically, goal and state are embedded onto a unit sphere and their cosine-simularity is used as state-goal compariability metric in an InfoNCE-type loss. The method is benchmarked on selected navigation tasks from OGBench.

Critical strengths
  • Generally interesting idea to learn representations for navigation problems
  • Well written and mathematically solid paper with technical details, inuition, and motivation laid out transparently.
Critical weaknesses
  • My major concern is about the scope and applicability of the proposed method. Although I find the idea interesting, I'm concerned that it is too adjusted for navigation-tasks. The paper provides no theoretical nor empirical indications why ARR should generalize to more and other RL tasks. Smaller experiments on scene-play-v0 indicate that the method will not generalize.
  • I miss a structured theoretical analysis of the method. For instance, what are assumptions on the environment for ARR to learn the optimal policy? Is it possible to derive regret bounds when focusing on navigation tasks? There, one may also be able to derive condidtions on the environment in order for ARR to work (or to fail).
  • The breadth of experiments is too narrow and focused on navigation to assess the impact of the method beyond navigation in an antmaze.
  • Although the angular embedding idea appears novel to me, the theoretical heavy lifting stems from InfoNCE and thus the theoretical contribution seems only modest to me.
Questions for authors

I find the paper is not ready for publication yet. The following questions are indented to help the authors to strengthen the paper for a future submission:

  • What are theoretical conditions on the environment for ARR to be successful?
  • Although I do not believe ARR to generalise beyond maze navigation, can ARR generalise beyond the given maze? How many mazes does ARR need to generalize to any maze?
  • ARR could be beneficial under high-dimensional image observations where the problem underneath requires steering in some vector field, like in the problems studied in [1] or [2].

[1] Sorokin et al, Interferobot: aligning an optical interferometer by a reinforcement learning agent, NeurIPS'20

[2] Schmähling et al., Trajectory-Level Data Augmentation for Offline Reinforcement Learning, ICML'26

1 comment

Thank you for your constructive suggestions and guidance. We would welcome further discussion, regardless of the final acceptance decision, as we believe your feedback will be valuable for our future research.

  1. Although our paper provides a theoretical framework, ARR does not achieve the best performance on every task, including scene-play-v0, and we have reported these limitations transparently. As discussed in our paper, ARR learns reachability induced by the offline data. This objective does not, by itself, identify a policy that reaches goals in the minimum time or along the shortest path. Nevertheless, Monte Carlo contrastive methods can offer stability and long-horizon learning benefits, while TD-based methods can suffer from compounding errors over long horizons, as discussed by Myers et al. [1]. ARR’s effectiveness also depends on the quality and coverage of the offline data. We consider these factors a possible explanation for its stronger performance on navigation tasks, although further investigation is needed. Our future work aims to extend ARR to learn optimal temporal-distance representations, for example by imposing triangle-inequality constraints. We also plan to explore integrating TD learning and Monte Carlo contrastive learning within the ARR framework.

  2. Different mazes have different obstacle layouts: identical coordinates can correspond to very different traversable distances. We therefore believe that generalizing from a single maze to unseen mazes is challenging for offline GCRL without additional data or information about the new layouts. Although aspects of the underlying dynamics may transfer, changes in the environment’s connectivity make direct generalization difficult. So, we also view access to diverse, carefully curated, high-quality data as an important bottleneck for embodied learning.

  3. We believe ARR may have potential for tasks with high-dimensional image observations. Exploring this direction would involve additional considerations, including visual feature extraction, network architecture, and the data augmentation techniques you suggested. We appreciate you bringing the recent augmentation work to our attention, as we had previously overlooked it. These considerations broaden the experimental scope, and visual observations were therefore not the initial focus of our study.

  4. We find it interesting to give learned representations an explicit geometric interpretation and reuse that structure for test-time planning, and we appreciate your recognition of this direction. We believe that both ARR’s representation-learning stage and its policy-extraction stage offer opportunities for further investigation and improvement, as discussed in Limitations and Future Work. We believe that no single reinforcement learning algorithm can excel across all tasks and settings, which helps explain why the field remains active and continues to evolve. Different tasks and environments often require different algorithms and reward designs. Developing a broadly generalizable foundation policy is therefore a shared goal for many researchers, including ourselves. Therefore, while ARR leaves some room for further exploration, we believe that its strong performance in specific settings and its approach to linking representation geometry, value functions, and test-time planning constitute a worthwhile contribution.

[1] Offline Goal-Conditioned Reinforcement Learning with Quasimetric Representations. NeurIPS 2025.

Reviewer_DViqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ARR learns separate state and goal embeddings on the unit sphere from offline trajectories, then freezes them to train a policy by progress-weighted behavior cloning. At deployment, it can select intermediate goals from a bank of dataset states. The paper relates contrastive scores to discounted future-state occupancy and compares Direct and Planner with nine baselines on four state-based OGBench datasets over eight seeds.

Critical strengths

The reported Direct-to-Planner gains are substantial across all three navigation environments: 43±7 to 99±1, 59±4 to 97±1, and 59±3 to 93±2. The frozen representation appears useful for waypoint selection, although the evaluation settings need clarification.

Critical weaknesses

1. The contribution over prior work and the calibration of progress scores remain unclear.

HILP (Park et al., ICML 2024) already uses the progress reward ⟨ϕ(s′)−ϕ(s),z⟩\langle\phi(s')-\phi(s),z\rangle with zz uniform on the unit sphere and selects minimax waypoints from dataset states (Eqs. 7 and 10). SoRB (Eysenbach et al., NeurIPS 2019) also plans through replay-buffer states. Neither is discussed. ARR uses asymmetric contrastive encoders and immediate progress to weight cloning, but the paper does not compare it with HILP-Plan. In Eq. (16), minimizing the maximum arccosine cost is equivalent to maximizing the minimum cosine similarity, so this rule alone does not establish a benefit of angular geometry. Direct is below CRL on all four datasets, so the gains do not establish a better representation without the planner.

Appendix B.1 sets a state-dependent offset c(s)c(s) to zero by convention (p. 17, lines 872–876). For an encoded goal command z=ψ(g)z=\psi(g), with V=VπDV=V^{\pi_D} and positive values, the ideal optimal-critic relation gives

pτ=log⁡Vg(s′)−log⁡Vg(s)+c(s′)−c(s).\frac{p}{\tau}=\log V_g(s')-\log V_g(s)+c(s')-c(s).

Equation (10) assumes a state-independent offset, but Algorithm 1 does not show how this is enforced. Unit normalization does not guarantee it. Because progress weights and Eq. (16) compare scores across states, this ambiguity can affect both policy training and waypoint selection. The alternative-loss ablation does not establish calibration across states for the selected row-wise objective.

2. The evaluation protocol and baseline results need clarification.

Table 3 lists an AntMaze final-goal distance of 2.0 and 1,000 steps for scene-play. OGBench uses an AntMaze XY success tolerance of 0.5 and a 750-step limit for scene. Internal planner parameters need not change the environment's success checks or time limits, but their relation is not explained.

On antmaze-large-navigate-v0, Table 1 reports HIQL at 76±3 and CGCIVL at 80±3; OGBench Table 2 and the CGCIVL paper's Table 1 report 91±2 for the respective methods. With Planner at 93±2, these baseline discrepancies need to be resolved before assessing its advantage. The published numbers alone are not a controlled comparison. I lean toward rejection because the evidence does not yet resolve these issues.

Questions for authors
  1. Do all methods use the standard OGBench success checks and time limits? Please distinguish these from Table 3's planner stopping parameters.

  2. How does ARR compare with HILP-Plan under matched state-bank sizes and deployment budgets? Does a matched unnormalized contrastive critic change the results?

  3. What fixes c(s)c(s) in practice, and how would correcting it affect progress weighting and planning?

  4. Which baseline configurations and checkpoint rules explain the lower HIQL and CGCIVL results? Does Figure 2's single-run pointmaze reference (88%) use the same training, checkpoint, and planner settings as Table 1's eight-seed result (99±1%)?

  5. How do Appendix D's 10,000-state banks (p. 20, lines 1076–1079) relate to Table 3's 100,000 states for AntMaze? Please specify the fallback, persistence, progress-margin, and XY exclusion rules. How much does the XY exclusion rule contribute to the PointMaze result?

2 comments

Thank you for these insightful questions. We address each question below.

Q1. Success and stopping rules.

All evaluations use official OGBench binary success and stop on environment termination or truncation. XY success tolerances are 1.0 for PointMaze and 0.5 for AntMaze. Navigation limits are 1,000 steps; Scene's 750-step environment limit overrides the larger outer-loop budget. AntMaze's distance 2.0 is a planner handoff threshold, not a success tolerance.

Q2. HILP and normalization.

The common planner uses a shared 10k bank, max–min selection, replanning interval 25, persistence 100, zero margins, and no XY exclusion or mean fallback. Planner success (%), mean ± sample SD across eight seeds:

\hline \text{Method}&\text{PointMaze-Medium}&\text{AntMaze-Large}\\\\ \hline \text{ARR, }\rho=1&92.7±2.8&89.9±1.2\\\\ \text{HILP, common planner}&71.7±19.8&20.9±7.0\\\\ \text{HILP, native recursive planner}&49.2±18.7&39.3±9.2\\\\ \text{ARR without embedding normalization}&0.0±0.0&0.9±1.3\\\\ \hline \end{array}$$ Native HILP with 50k candidates gives $72.7\pm16.7$%/$44.7\pm19.0$%. ARR/unnormalized use 100k+100k updates on PointMaze and 1M+1M on AntMaze; HILP uses 1M. Architectures/batches differ, so training compute is not matched. The normalization result applies only to this configuration. Arccosine minimax and cosine max–min rank identically; ARR's intended contribution is asymmetric representation reuse for weighted cloning and planning. **Q3. State-dependent offsets.** Row-wise InfoNCE does not fix $c(s)$; unit normalization cannot justify dropping it. The ideal relation for encoded goals is $p/\tau=\Delta\log V_g^{\pi_D}+\Delta c$, under the occupancy/sampling assumptions. CRL [1] analyzes binary NCE and regularized InfoNCE; Scaling CRL [2] uses InfoNCE. Our choice is empirical: eight-seed PointMaze Direct means are **55.6/35.9/47.7%** for InfoNCE/BNCE/symmetric InfoNCE. With $f(s,g)=\langle\phi(s),\psi(g)\rangle/\tau$, we tested $\bar f(s,g)=f(s,g)-\log\mathbb E_{G\sim q}e^{f(s,G)}$, using 4,096 training-marginal goals. This removes additive row shifts; approximation and sampling errors remain. $$\begin{array}{lrr} \hline \text{Task}&\text{Original progress}&\text{Corrected progress}\\\\ \hline \text{PointMaze-Medium}&55.6±11.1&44.5±11.8\\\\ \text{AntMaze-Large}&58.1±3.2&59.7±3.5\\\\ \hline \end{array}$$ Values are Direct success (%; mean ± sample SD). Diagnostics use $\rho=1$, unlike the submitted settings. Correction gives no consistent improvement, but does not establish that offsets are harmless or resolve their effect on planner selection. We retain the offset difference in the theoretical interpretation. **Q4. Baselines and 88% versus 99%.** HIQL and CGCIVL use official implementations, each trained for 1M updates with eight seeds. We evaluate each final 1M-update checkpoint (mean ± sample SD across seeds). HIQL's original paper did not evaluate on OGBench. Our CGCIVL PointMaze-Medium and AntMaze-Medium results closely match the published scores; the lower AntMaze-Large scores remain unexplained. Figure 2 varies one factor at a time, without failed-XY exclusion or the execution progress-margin check. Table 1 uses the full planner (revised Table 4). Thus 88% and $99\pm1$% differ in deployment settings as well as single-run versus eight-seed aggregation. **Q5. Banks and deployment heuristics.** AntMaze capacity is 10k; 100k was a typo. Nested 1k/10k/100k PointMaze banks (one checkpoint) give 85.6/89.6/84.8% success. Median query times with encoding: 2.80/3.10/5.07 ms on PointMaze and 2.83/3.07/5.08 ms on AntMaze (640 calls/setting, one Xeon 8558P thread). Embedding caches use 0.512/5.12/51.2 MB, plus equal normalized copies and raw states. Timing excludes actor/environment and bank preparation. PointMaze mean fallback retries rejected max-leg selection using average two-leg cost. The latent-only bundle uses replanning interval 25, persistence ≤50 steps, route/execution margins 0.02, and repeated-index exclusion. Execution progress targets the active waypoint. Handoff/failed-XY radii are 1/0.75. Eight new seeds ($\rho=1$), 250 paired episodes/seed; all six conditions reuse the same eight models. Values are success % (mean ± sample SD). $$\begin{array}{lr} \hline \text{Deployment}&\text{Success}\\\\ \hline \text{Direct}&41.5±11.4\\\\ \text{Latent-only bundle}&87.1±3.2\\\\ \text{+ final-goal handoff}&90.3±1.2\\\\ \text{+ handoff + failed-XY exclusion}&91.0±0.8\\\\ \text{+ handoff + mean fallback}&91.2±0.5\\\\ \text{+ handoff + both}&92.7±2.8\\\\ \hline \end{array}$$ Latent-only planning gains 45.6 points over Direct; all heuristics add 5.6 more. Failed-XY exclusion adds 0.7 over handoff. [1] *Contrastive Learning as Goal-Conditioned Reinforcement Learning*. NeurIPS 2022. [2] *1000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilities*. NeurIPS 2025.

Thank you for the clarifications and additional experiments. These resolve my questions about the evaluation protocol and state-bank capacity, and the new comparisons strengthen the empirical evidence.

However, two concerns remain: (1) the calibration tests cover only Direct deployment with ρ=1\rho=1, leaving their relevance to planner performance and the original mixed-target settings unclear; (2) the AntMaze-Large baseline discrepancy remains unexplained, while ARR and CRL achieve numerically similar results under the common planner.

I will maintain my current score of 2 (Weak rejection) for now.

Reviewer_QiE6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Angular Reachability Representations (ARR), a framework for offline goal-conditioned reinforcement learning that models state-goal reachability through angular relationships between unit-normalized embeddings. The proposed approach consists of three components. First, ARR learns separate state and goal encoders using an InfoNCE objective, relating the learned compatibility scores to discounted future-state occupancy under the behavior policy. Second, the authors introduce Offline Target-Progress Policy (OTPP), which uses changes in the learned state-goal compatibility scores to construct weights for behavior cloning. Third, an angular waypoint planner selects intermediate goals from an offline state bank during deployment without additional model training.

Critical strengths
  1. Strong empirical performance on long-horizon navigation tasks. A major strength of this work is the performance achieved by the proposed test-time waypoint planner. As reported in Table 1, ARR with planning achieves average success rates of 99%, 97%, and 93% on pointmaze-medium, antmaze-medium, and antmaze-large, respectively. In particular, the results on antmaze-large suggest that the proposed planning framework can effectively improve long-horizon navigation performance. These results are meaningful because long-horizon goal reaching remains challenging for offline reinforcement learning methods, particularly when direct goal-conditioned execution is unreliable.

  2. A unified and practically appealing framework for representation learning and deployment. The paper provides a coherent approach in which the learned state-goal representations are reused for both offline policy training and test-time waypoint selection. This avoids the need to train an additional task-specific planning model. The modular structure is practically appealing and could potentially facilitate integration with other offline reinforcement learning algorithms.

Critical weaknesses
  1. Insufficient theoretical justification. The InfoNCE critic is identifiable only up to a state-dependent additive term, which may affect the target-progress scores used by OTPP (Eq. 10). Although the paper acknowledges this limitation, its practical implications remain unclear.

  2. Insufficient ablation studies and limited generalization. The main performance gains come from test-time planning, but comparisons with existing methods using similar planners are missing. Moreover, the relatively weak performance on manipulation tasks raises concerns about the generality of the proposed approach.

Questions for authors
  1. Theoretical validity of target-progress estimation. How does the proposed training procedure address the state-dependent additive ambiguity of the row-wise InfoNCE critic discussed in Appendix B.1? Can the authors provide theoretical or empirical evidence that this ambiguity does not significantly affect the progress scores used by OTPP?

  2. Isolating the contribution of angular representations. Could the authors compare ARR against existing contrastive representations, such as CRL or Scaling CRL, when all methods use the same waypoint selection mechanism and comparable planning budgets? This comparison would help determine whether the observed navigation improvements arise from ARR specifically or from the introduction of test-time waypoint planning.

  3. Practical reliability of the planning guarantee. Appendix C establishes a decrease in the estimated angular cost under the assumption that the selected waypoint is reached exactly. How frequently is this assumption approximately satisfied in practice? Could the authors report waypoint-reaching success rates or analyze the sensitivity of the planner to unreachable or inaccurately evaluated waypoints?

  4. Generalization and reproducibility. What explains the substantial performance degradation on the scene-play manipulation benchmark? Additionally, Appendix D describes a checkpoint state-bank capacity of 10,000, whereas Table 3 reports 100,000 states for the antmaze planners. Could the authors clarify this difference and report how the state-bank size influences planning performance and computational cost?

1 comment

Thank you for these constructive questions. We have added the following clarifications and controls to the revised manuscript.

Q1. Calibration of target progress.

Row-wise InfoNCE does not fix c(s)c(s), and unit normalization alone cannot remove it. CRL [1] derives its main result using binary NCE, which identifies population-optimal logits under appropriate assumptions, as in our Appendix B.2. CRL also studies regularized InfoNCE, and Scaling CRL [2] uses InfoNCE. Our choice is empirical: eight-seed experiments (ρ=1\rho=1) give PointMaze-Medium Direct means of 55.6/35.9/47.7% for InfoNCE/BNCE/symmetric InfoNCE.

We tested corrected progress with f(s,g)=⟨ϕ(s),ψ(g)⟩/τf(s,g)=\langle\phi(s),\psi(g)\rangle/\tau and fˉ(s,g)=f(s,g)−log⁡EG∼qef(s,G)\bar f(s,g)=f(s,g)-\log\mathbb E_{G\sim q}e^{f(s,G)}, using 4,096 training-marginal reference goals. This removes additive row shifts; approximation and sampling errors remain.

\hline \text{Task}&\text{Original progress}&\text{Corrected progress}\\\\ \hline \text{PointMaze-Medium}&55.6±11.1&44.5±11.8\\\\ \text{AntMaze-Large}&58.1±3.2&59.7±3.5\\\\ \hline \end{array}$$ Values are Direct success percentages (mean ± sample SD). Correction gives no consistent improvement, but does not establish that the ambiguity is harmless. The ideal log-value interpretation retains $\Delta c$, subject to occupancy/sampling assumptions. Diagnostics use $\rho=1$, unlike the submitted settings. **Q2. Representation versus planning.** The common planner uses a shared 10k bank, max–min selection, replanning interval 25, persistence 100, zero margins, and no XY exclusion or mean fallback. Cells show Direct / Planner success (%; mean ± sample SD across eight seeds). $$\begin{array}{lrr} \hline \text{Method}&\text{PointMaze-Medium}&\text{AntMaze-Large}\\\\ \hline \text{ARR, }\rho=1&55.6±11.1\\;/\\;92.7±2.8&58.1±3.2\\;/\\;89.9±1.2\\\\ \text{CRL}&49.1±4.6\\;/\\;56.8±7.4&84.2±5.8\\;/\\;89.5±1.2\\\\ \text{Scaling port, BCE}&36.1±7.8\\;/\\;69.3±2.3&83.3±4.8\\;/\\;75.9±4.5\\\\ \text{Scaling port, InfoNCE+}\mathrm{LSE}^2&65.3±4.7\\;/\\;76.1±5.7&74.9±4.2\\;/\\;71.6±2.8\\\\ \hline \end{array}$$ Scaling variants are offline architecture ports, not the official online algorithm; the InfoNCE variant adds 0.1 $\mathrm{LSE}^2$. ARR's Planner gains are **37.1/31.8 percentage points**, supporting its usefulness for waypoint guidance. Architectures, batches, and training compute differ, so these gains cannot be attributed solely to angular geometry. **Q3. Inexact waypoint execution.** Define $D_g(s)=\arccos\langle\phi(s),\psi(g)\rangle$. For a waypoint satisfying $\max\\{D_w(s),D_g(w)\\}\le D_g(s)-\delta, \delta > 0$, the spherical triangle inequality gives $D_g(s^+)\le D_g(s)-\delta+e, e=\arccos\langle\phi(s^+),\phi(w)\rangle$. Thus $e < \delta$ suffices for estimated angular-cost descent. Here $e$ compares two state embeddings; the bound does not guarantee physical arrival or cover unrestricted mean fallback. In the eight-model PointMaze diagnostic, **2,656/4,255 assignments (62.4%)** reached XY distance ≤1 within their actual execution windows (at most 100 steps); **1,468/4,255 (34.5%)** satisfied $e < \delta$. Early-ending assignments remain included: these are not fixed-100-step reach probabilities or endpoint guarantees. Failure of the sufficient condition does not imply execution failure. A separate evaluation uses eight new training seeds ($\rho=1$), 250 episodes/seed: $$\begin{array}{lr} \hline \text{Deployment}&\text{Success (\\%; mean ± sample SD)}\\\\ \hline \text{Direct}&43.2±12.6\\\\ \text{Common planner}&87.1±4.0\\\\ \text{Planner, execution progress margin 0.02}&94.3±2.2\\\\ \hline \end{array}$$ The execution-margin variant has higher mean success. Its progress check toward the active waypoint differs from the route margin $\delta$ above. **Q4. Manipulation and bank capacity.** ARR's weak manipulation performance remains a limitation, as discussed in **Limitations and Future Work**. We corrected the extra zero in the AntMaze bank entries: the capacity is 10k. For one checkpoint ($\rho=1$), nested 1k/10k/100k PointMaze banks give 85.6/89.6/84.8% success. Median query times, including encoding, are 2.80/3.10/5.07 ms on PointMaze and 2.83/3.07/5.08 ms on AntMaze (640 calls/setting, one Xeon 8558P thread). Embedding caches occupy 0.512/5.12/51.2 MB, plus equally sized normalized copies and raw states. Timings exclude actor/environment execution and bank preparation. [1] *Contrastive Learning as Goal-Conditioned Reinforcement Learning*. NeurIPS 2022. [2] *1000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilities*. NeurIPS 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.