acceptodds
Under review as a conference paper at ICLR 2027

Hyperspherical Contrastive Representations Unlock Planning and Exploration

Abstract

We develop a self-supervised reinforcement learning algorithm that learns to build towers of up to eight cubes from scratch, without prior data, rewards, demonstrations, or pretrained models. Our method, Contrastive Planning via Spherical Linear Interpolation (C-Slerp), uses contrastive representations as reusable building blocks and repeatedly composes them to infer a sequence of intermediate waypoints. Under our hyperspherical parameterization, the learned representations allow intermediate waypoints to be inferred in closed form through spherical linear interpolation, without training a separate high-level policy or performing graph search. Importantly, the representation geometry also tells us how to adjust the number of waypoints based on the difficulty of each goal-reaching problem. Experiments show that our method succeeds on long-horizon navigation and robotic manipulation tasks, including settings where strong baselines fail to achieve a single success. Ablations further show that using a fixed number of waypoints substantially degrades performance, highlighting the importance of adapting the plan to task difficulty.

Primary area:
reinforcement learning
Keywords:
self-supervised learning, goal-conditioned reinforcement learning, planning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_qR1wOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes C-Slerp, an online, self-supervised, goal-conditioned RL method built on contrastive RL (CRL). It learns two scores: a state–state score fH(s,s+)=ψ(s)⊤R ψ(s+)f_H(s, s^+) = \psi(s)^\top R\, \psi(s^+) on the unit hypersphere, where RR is a learned orthogonal matrix (via a Cayley transform) that makes the score asymmetric; the standard CRL critic fL(s,a,g)=ϕ(s,a)⊤ψ(g)f_L(s, a, g) = \phi(s,a)^\top \psi(g). Waypoint inference is posed as a variational problem. Under a Bayes-optimality assumption and a uniformity assumption, the authors derive closed-form waypoint distributions: slerp from ψ(s)\psi(s) to RK+1ψ(g)R^{K+1}\psi(g), with each point mapped back by (R⊤)i(R^\top)^i, plus a small isotropic noise term. The number of waypoints KK is sampled from a softmax over the alignment scores ℓK(s,g)=ψ(s)⊤RK+1ψ(g)\ell_K(s,g) = \psi(s)^\top R^{K+1}\psi(g).

Critical strengths
  1. Simple, appealing idea. The planner adds no network parameters and does no graph search. It reuses the learned geometry both to choose how many stages to plan and to place the waypoints. Choosing the number of stages per problem is a well-motivated gap in hierarchical GCRL.

  2. Strong empirical results on hard sparse-reward tasks. Gripper-based stacking of 3–5 cubes with no curriculum, rewards or demonstrations, where every baseline gets zero success, is a notable result.

Critical weaknesses
  1. pγ,Kπ^(w1:K∣s,g)p^{\gamma,K}{\hat\pi}(w{1:K} \mid s, g) in Eq. 5 is described only in words. Sec. 3 defines the discounted occupancy conditioned on (s,a)(s, a), i.e., pπγ(s+∣s,a)p^\gamma_\pi(s^+ \mid s, a). Eq. 7 and App. A.2 use a state-only kernel pπ^γ(s+∣s)p^\gamma_{\hat\pi}(s^+ \mid s) that is never defined, i.e., which action marginalization and which policy or mixture it is taken over.

  2. The Markov factorization ignores the commanded goal. A.2 writes pπ^γ,K(w1:K,g∣s)=∏i=0Kpπ^γ(wi+1∣wi).p_{\hat\pi}^{\gamma,K}(w_{1:K}, g \mid s) = \prod_{i=0}^{K} p^\gamma_{\hat\pi}(w_{i+1} \mid w_i). But π^\hat\pi is goal-conditioned and hierarchical: its actions depend on the commanded goal, the active subgoal and the replanning phase. Treating these as hidden variables gives a mixture of Markov chains, p(w1:K,g∣s)=∫p(gcmd∣s)∏i=0Kp(wi+1∣wi,gcmd) dgcmd,p(w_{1:K}, g \mid s) = \int p(g_{\text{cmd}} \mid s) \prod_{i=0}^{K} p(w_{i+1} \mid w_i, g_{\text{cmd}})\, \mathrm{d}g_{\text{cmd}}, which is not Markov in states. The derivation also conflates the commanded goal with the reached endpoint gg. Lemma 2 (A.4) relies on the same assumption. Eysenbach et al. (2024) make a similar factorization, but with offline data from a single policy that is not goal-conditioned, where it is easier to justify.

Questions for authors
  1. Please give a formal definition of pπ^γ(s+∣s)p^\gamma_{\hat\pi}(s^+ \mid s) and pπ^γ,Kp^{\gamma,K}_{\hat\pi}, including the policy or mixture they are taken over. How do you justify the Markov factorization given that π^\hat\pi is goal-conditioned and hierarchical?

  2. Can you report empirical diagnostics for Assumption 2 (e.g., a uniformity loss or an estimate of the embeddings' entropy on Sd−1\mathbb{S}^{d-1}) and for the condition α(d−1)/(2cos⁡Δ)≪1\alpha(d-1)/(2\cos\Delta) \ll 1 under the learned α\alpha?

4 comments
AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer qR1w,

We are grateful for your careful reading of our paper and your valuable suggestions.

Our responses are provided below.


Q1. Formal definition of pπ^γ(s+∣s)p_{\hat{\pi}}^\gamma(s^+\mid s).

A1. Thank you for raising this point. To define pπ^γ(s+∣s)p_{\hat{\pi}}^\gamma(s^+\mid s), we consider the following process. We independently sample an initial state s0∼p0s_0 \sim p_0 and a commanded goal gcmd∼pgg_{\mathrm{cmd}} \sim p_g. Starting from s0s_0, we run the hierarchical goal-conditioned policy π^\hat{\pi} described in Sec. 4.1 with gcmdg_{\mathrm{cmd}} fixed throughout the rollout. Every hh steps, the planner takes the current state and gcmdg_{\mathrm{cmd}} as inputs and selects a waypoint representation zz. The low-level policy πθ\pi_\theta then takes actions conditioned on the current state and zz for the next hh steps. Let (s,a)=(st,at)(s,a)=(s_t,a_t) denote the state and action at time tt along the rollout. We independently draw T∼Geom(1−γ)T\sim\mathrm{Geom}(1-\gamma) on {1,2,…}\{1,2,\ldots\} and define s+=st+Ts^+=s_{t+T}, the state reached TT steps after ss.

This process induces the joint distribution pπ^γ(s,gcmd,a,s+)p_{\hat{\pi}}^\gamma(s,g_{\mathrm{cmd}},a,s^+). By marginalizing this joint distribution and conditioning on ss or (s,a)(s,a), we obtain

pπ^γ(s+∣s)=∫G∫Apπ^γ(s,gcmd,a,s+),da,dgcmdpπ^γ(s),p_{\hat{\pi}}^{\gamma}(s^+ \mid s) = \frac{\int_{\mathcal{G}}\int_{\mathcal{A}} p_{\hat{\pi}}^{\gamma}(s,g_{\mathrm{cmd}},a,s^+) \\,\mathrm{d}a \\,\mathrm{d}g_{\mathrm{cmd}}}{p_{\hat{\pi}}^{\gamma}(s)},

pπ^γ(s+∣s,a)=∫Gpπ^γ(s,gcmd,a,s+),dgcmdpπ^γ(s,a).p_{\hat{\pi}}^{\gamma}(s^+ \mid s,a) = \frac{\int_{\mathcal{G}} p_{\hat{\pi}}^{\gamma}(s,g_{\mathrm{cmd}},a,s^+) \\,\mathrm{d}g_{\mathrm{cmd}}}{p_{\hat{\pi}}^{\gamma}(s,a)}. Here, pπ^γ(s)p_{\hat{\pi}}^\gamma(s) and pπ^γ(s,a)p_{\hat{\pi}}^\gamma(s,a) are the corresponding marginals of the same joint distribution. Thank you again for pointing this out. We will make these definitions explicit in the final version.

Q2. Formal definition of pπ^γ,K(w1:K∣s,g)p_{\hat{\pi}}^{\gamma,K}(w_{1:K}\mid s,g).

A2. We appreciate your comment. As described in Sec. 4.1, we consider KK intermediate waypoints w1:K=(w1,…,wK)w_{1:K}=(w_1,\ldots,w_K) along a trajectory generated by π^\hat{\pi}. As in A1, we sample the commanded goal gcmd∼pgg_{\mathrm{cmd}}\sim p_g and hold it fixed throughout the rollout. Starting at w0=sw_0=s, we draw independent intervals Ti∼Geom(1−γ)T_i\sim\mathrm{Geom}(1-\gamma) for i=0,…,Ki=0,\ldots,K. Each wi+1w_{i+1} is the state reached TiT_i steps after wiw_i. We set g=wK+1g=w_{K+1}, the endpoint on which we condition. This endpoint gg need not equal the commanded goal gcmdg_{\mathrm{cmd}}.

This process defines the joint distribution pπ^γ,K(s,gcmd,w1:K,g)p_{\hat{\pi}}^{\gamma,K}(s,g_{\mathrm{cmd}},w_{1:K},g). We integrate out gcmdg_{\mathrm{cmd}} and condition on (s,g)(s,g) to obtain

pπ^γ,K(w1:K∣s,g)=∫Gpπ^γ,K(s,gcmd,w1:K,g),dgcmdpπ^γ,K(s,g).p_{\hat{\pi}}^{\gamma,K}(w_{1:K}\mid s,g) = \frac{\int_{\mathcal{G}} p_{\hat{\pi}}^{\gamma,K}(s,g_{\mathrm{cmd}},w_{1:K},g)\\,\mathrm{d}g_{\mathrm{cmd}}}{p_{\hat{\pi}}^{\gamma,K}(s,g)}.

Here, pπ^γ,K(s,g)p_{\hat{\pi}}^{\gamma,K}(s,g) is the joint marginal of the starting state s=w0s=w_0 and the reached endpoint g=wK+1g=w_{K+1}.

AuthorsOpenReview ↗

Official Comment by Authors

Q3. Markov factorization of the waypoint sequence.

A3-1. Thank you for raising this point. As you pointed out, the waypoint sequence generated by our hierarchical goal-conditioned policy π^\hat{\pi} need not be Markovian even when the environment is an MDP. For this reason, we treated the Markov property of the waypoint sequence as a separate modeling assumption in the main text and appendix. We will state this point more clearly in the final version.

A3-2. To examine the Markov assumption empirically, we tested whether p(wi∣wi−1)≈p(wi∣wi−1,wi−2)p(w_i\mid w_{i-1})\approx p(w_i\mid w_{i-1},w_{i-2}). We collected rollouts from the trained policies across a range of commanded goals, with 5,000 rollouts per task. For the main analysis, we sampled waypoint triples from trajectories using geometrically distributed time intervals.

Following the classifier-based approach in [1], we estimated the conditional mutual information (CMI) between wiw_i and wi−2w_{i-2} given wi−1w_{i-1}: I(wi;wi−2∣wi−1)=I(wi;wi−2,wi−1)−I(wi;wi−1).I(w_i;w_{i-2}\mid w_{i-1}) = I(w_i;w_{i-2},w_{i-1})-I(w_i;w_{i-1}).

We trained two binary classifiers with inputs [wi−1,wi−1,wi][w_{i-1},w_{i-1},w_i] and [wi−2,wi−1,wi][w_{i-2},w_{i-1},w_i], respectively. We duplicate wi−1w_{i-1} in the first classifier’s input so that both classifiers have the same input dimension and model size for a fair comparison. Each classifier estimated the probability that its input was a positive example. For positive examples, all waypoints came from the same trajectory. For negative examples, we replaced only wiw_i with a waypoint sampled from a different trajectory. The prediction target was the agent’s (x,y)(x,y) position in the mazes. In the manipulation tasks, the target included the gripper pose and the block positions. The results are shown below:

Mutual information (nats)I(wi;wi−1)I(w_i;w_{i-1})I(wi;wi−2,wi−1)I(w_i;w_{i-2},w_{i-1})Estimated CMI [95% CI]Null test [95% CI]
ant_big_maze2.476 ± 0.0422.441 ± 0.033[−0.043, −0.013][−0.081, −0.035]
ant_u5_maze1.946 ± 0.0211.919 ± 0.041[−0.024, −0.003][−0.025, −0.013]
3_gripper2.082 ± 0.0252.029 ± 0.042[−0.115, −0.005][−0.072, +0.039]
4_gripper1.828 ± 0.0281.741 ± 0.093[−0.186, +0.117][−0.069, +0.176]

We report mutual information (MI) as (mean ± standard deviation) over five classifier seeds. Although true CMI is nonnegative, its estimate can be negative because we use two separate classifier networks to estimate the two MI terms and then take their difference. To account for this estimation error, we compared the observed estimates with those from a null test in which we replaced only wi−2w_{i-2} with a waypoint sampled from a different trajectory. This gives us a baseline for interpreting the estimates when the true CMI is zero.

Across the four tasks, adding wi−2w_{i-2} did not increase the estimated mutual information, and the 95% confidence intervals for the observed and null CMI estimates overlapped in each task. On these trajectories, we found little evidence that wi−2w_{i-2} provided additional information about the next maze position or the next gripper pose and block positions in the manipulation tasks given wi−1w_{i-1}.

[1] Mukherjee et al. “CCMI: Classifier based conditional mutual information estimation.” 2020.

AuthorsOpenReview ↗

Official Comment by Authors

Q4. Empirical diagnostics for the uniformity assumption.

A4. Following your insightful suggestion, we added an empirical analysis of the uniformity assumption to the revised manuscript. Figure 13 shows that as the uniformity term of the symmetric InfoNCE loss decreases during training, the empirical distribution approaches the ideal distribution for points sampled uniformly on the unit hypersphere. These results provide empirical support for our uniformity assumption across all evaluated tasks.

Q5. Empirical diagnostics for the small-variance assumption.

A5. We appreciate you raising this point. Please see Figure 14 in the revised manuscript, where we plot the quantity α(d−1)2cos⁡Δ\frac{\alpha(d-1)}{2\cos\Delta} during training. Across all tasks, this quantity converges to a value much smaller than 1, providing empirical support for our small-variance assumption.


Thank you again for your valuable and insightful review. Also, please let us know if our responses have addressed your questions. If anything requires further clarification, please do not hesitate to let us know.

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer qR1w,

Thank you again for your time and thoughtful feedback on our paper. In response to your comments, we have conducted additional experiments and evaluations, and added new figures to strengthen our paper.

As the discussion period is coming to an end, we wanted to kindly check whether our responses have addressed your concerns. If you have any remaining questions or would like further clarification, please let us know. We would be happy to address them before the deadline.

Reviewer_eRsWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes C-Slerp, a goal-conditioned reinforcement learning method that uses hyperspherical contrastive representations for adaptive waypoint planning. The method selects the number of waypoints and infers intermediate subgoals through spherical interpolation. Experiments on navigation and manipulation benchmarks show strong improvements over existing methods.

Critical strengths
  1. The idea of adapting the number of waypoints to task difficulty is interesting. The geometric formulation also avoids training a separate high-level policy.
  2. The experimental results are convincing across several challenging tasks. The ablations on waypoint selection and the rotation matrix provide useful evidence for the proposed method.
Critical weaknesses
  1. The method selects the number of waypoints using alignment scores in the learned hyperspherical space. While Figures 4 and 5 show that adaptive planning outperforms fixed choices, they do not establish whether the predicted number of waypoints accurately reflects the actual task horizon or subgoal reachability. A quantitative analysis of this relationship would better support the central motivation.
  2. The method combines spherical representations, an additional contrastive objective, a learned rotation matrix, and adaptive waypoint selection. Although several components are ablated, the paper does not directly compare against an adaptive planner using Euclidean representations or alternative interpolation methods under matched conditions. Such comparisons would help establish whether the improvement specifically comes from the proposed hyperspherical formulation.Furthermore, all experiments are conducted in simulation with state-based observations. The applicability to visual inputs and physical robots remains untested, especially given the substantial environment interaction required during training.
Questions for authors
  1. The paper motivates hyperspherical representations through their ability to support closed-form waypoint inference using spherical linear interpolation. However, waypoint interpolation can also be performed in Euclidean and hyperbolic spaces. In particular, hyperbolic representations may be suitable for modeling hierarchical relationships and varying goal-reaching distances.Could the authors clarify what specific advantages hyperspherical geometry provides over Euclidean or hyperbolic representations for this task? Is the choice theoretically necessary for adaptive planning, or primarily motivated by mathematical convenience?
  2. The current ablations demonstrate the importance of adaptive waypoint selection, but they do not isolate the contribution of the underlying geometry. Could the authors compare hyperspherical interpolation with Euclidean linear interpolation or hyperbolic geodesic interpolation under comparable training and planning settings?Such comparisons would help establish whether the reported improvements arise from the hyperspherical representation itself or primarily from the adaptive planning strategy.
4 comments
AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer eRsW,

We sincerely thank you for the insightful feedback.

We have summarized the weaknesses and questions you raised. In our responses below, we present the results of two additional experiments that we conducted to address your concerns.


Q1. A quantitative analysis of the predicted number of waypoints.

A1. We appreciate your comment. Regarding the predicted number of waypoints, we refer the reviewer to Figures 15 and 16 in the appendix. Across all tasks, the predicted number of waypoints KK generally decreases as the agent approaches the goal, suggesting that KK reflects the remaining task horizon. We will also mention this result in the main text of the final version so that readers can find it more easily.

Q2. A quantitative analysis of estimated subgoal reachability.

A2. In Figure 21, we provide a qualitative visualization of our reachability score ρi=ϕ(s,ai)⊤zi\rho_i=\phi(s,a_i)^\top z_i. Following your valuable suggestion, we extended this analysis by comparing the score with empirical subgoal success rates. Across all tasks, we randomly sampled state-goal pairs (s,g)(s,g) from the visited states and computed the reachability score ρi\rho_i for each pair using zi=ψ(g)z_i=\psi(g). We then grouped the pairs by score into intervals of width 0.04 and randomly selected up to 200 pairs from each interval for evaluation. We then evaluated whether the trained policy π(a∣s,g)\pi(a\mid s,g) successfully reached gg from ss within 200 steps, using each benchmark’s success criterion.

We added Figures 17 and 18 to the revised manuscript to show how the success rate varies with the reachability score. The results show that higher scores generally correspond to higher success rates, suggesting that ρi\rho_i reflects subgoal reachability. Thank you for this suggestion!

Q3. Advantages of hyperspherical representations for planning.

A3. Thank you for your question. As described in the paper, hyperspherical representations are essential to the key components of our method, including adaptive planning, closed-form waypoint inference, and waypoint selection for the low-level policy.

A3-1. First, the hyperspherical representation supports high-level planning. Our planning requires applying RR repeatedly, expressed as RK+1R^{K+1}. Since RK+1R^{K+1} preserves unit norms, we can compare different numbers of waypoints KK through alignment scores on the same hypersphere. Without this norm preservation, repeated matrix multiplications can amplify or shrink representations, making comparisons across KK sensitive to scale differences across tasks. Furthermore, the unit hypersphere allows us to use a von Mises-Fisher likelihood to derive a softmax distribution for selecting KK needed between the current state and the goal. Together, these properties make hyperspherical geometry essential to our formulation for selecting the number of waypoints, without training a separate high-level policy.

A3-2. Furthermore, we would like to emphasize that our hyperspherical representations allow us to reformulate our objective into a tractable form. Specifically, under the uniformity assumption on the unit hypersphere, we can reformulate the KL terms in Eq. 9 as entropy regularization, allowing us to model waypoint stochasticity. Moreover, since the learned orthogonal matrix RR preserves the hyperspherical geometry, we can derive our closed-form waypoint solution from the reformulated objective in Eq. 10 using only spherical linear interpolation.

A3-3. Hyperspherical geometry also bounds the reachability scores ρi=ϕ(s,ai)⊤zi\rho_i=\phi(s,a_i)^\top z_i in Eq. 15 within [−1,1][-1,1], since both ϕ(s,ai)\phi(s,a_i) and ziz_i lie on the unit hypersphere. Without constraining representations to the unit hypersphere, these scores would also depend on representation norms, which can vary across environments and require separate tuning of the reachability threshold ϵ\epsilon for each task. In contrast, our hyperspherical formulation provides a common scale, allowing us to use the same value, ϵ=0.4\epsilon=0.4, across all 12 benchmark environments, achieving the highest success rates on most tasks.

We will emphasize the role of hyperspherical representations more clearly in the final version.

AuthorsOpenReview ↗

Official Comment by Authors

Q4. Comparison with an adaptive planner using Euclidean representations.

A4-1. Thank you for your suggestion. For the reasons outlined in A3, we would like to emphasize that hyperspherical geometry is essential to our derivation. Our method relies on this geometry to infer waypoints in closed form and to select their number KK adaptively using the same learned representations. If we change only the geometry of the representations, the same derivation no longer applies.

A4-2. Given this difference in geometry, we followed your valuable suggestion and constructed a Euclidean variant to match our setting as closely as possible. We made the following changes:

  • Hyperspherical representations → representations in Rn\mathbb{R}^n.
  • Orthogonal transformation RR → linear transformation AA.
  • Spherical linear interpolation → Euclidean linear interpolation.

We evaluated the Euclidean variant in five benchmark environments using the same training budget and network size as our method. As shown below, the Euclidean variant achieved lower success rates (averaged over five seeds) than our method in all five environments.

Success rate3-tower (gripper)8-towerAnt big MazeAnt U4 MazeAnt U5 Maze
Hyperspherical (Ours)0.879±0.0210.369±0.0270.925±0.0080.954±0.0070.848±0.023
Euclidean under matched conditions0.000±0.0000.000±0.0000.129±0.0220.068±0.0850.000±0.000

We observed that the norm of AA continued to grow during training despite regularization and that the norms of the learned representations varied across tasks. We believe that these differences in scale may have contributed to the poor performance of the Euclidean variant. Since its performance did not stabilize, we report its peak evaluation success rate during training for each task. These results suggest that adaptive waypoint selection alone does not fully explain the performance of our method on these tasks.

A4-3. Regarding hyperbolic representations, such as those used in [1], we agree that they can be well suited to goal-reaching tasks because they can capture the tree-like structure inherent in these tasks. However, we would like to note that both our closed-form waypoint solution and our distribution over the number of waypoints KK rely on hyperspherical geometry (please see A3 for details). That said, we believe that using hyperbolic representations for adaptive planning by repeatedly applying a learned matrix to select the planning horizon is a promising direction for future work.

[1] Klein et al. “Outperforming 1000 Euclidean Layers With Hyperbolic Goal-Conditioned RL.” 2026.

Q5. Comparison with alternative waypoint interpolation methods.

A5-1. Thank you for your suggestion. While we agree that comparing our method with alternative interpolation methods strengthens the paper, we would like to emphasize that our use of spherical linear interpolation (SLERP) is not a heuristic choice. Under the stated assumptions, we derive SLERP as part of the closed-form solution to the waypoint inference objective in Eq. 10.

A5-2. With this distinction in mind, we followed your valuable suggestion and evaluated the following two alternatives under conditions that match our planning formulation as closely as possible.

  • Direct SLERP between the state and goal representations: ui=slerp⁡(ψ(s),ψ(g),ti)u_i = \operatorname{slerp}\bigl(\psi(s), \psi(g), t_i\bigr)
  • Normalized linear interpolation between the state and goal representations: ui=((1−ti)ψ(s)+tiψ(g))/∥(1−ti)ψ(s)+tiψ(g)∥2u_i = \bigl((1-t_i)\psi(s) + t_i\psi(g)\bigr) / \bigl\lVert (1-t_i)\psi(s) + t_i\psi(g) \bigr\rVert_2
Success rate3-tower (gripper)8-towerAnt big MazeAnt U4 MazeAnt U5 Maze
SLERP with R (Ours)0.879±0.0210.369±0.0270.925±0.0080.954±0.0070.848±0.023
Direct SLERP0.214±0.0690.000±0.0000.217±0.0100.000±0.0000.000±0.000
Normalized linear interpolation0.248±0.0590.000±0.0000.229±0.0230.000±0.0000.000±0.000

To isolate the effect of the waypoint interpolation method, we kept the representation space hyperspherical across all three methods. As shown in the table above, our method achieved higher success rates (averaged over five seeds) than both alternatives on each of the five tasks. These results suggest that waypoints derived from our objective are more effective for goal reaching than those obtained by directly interpolating between the state and goal representations.

AuthorsOpenReview ↗

Official Comment by Authors

Q6. Applicability to physical robots and visual inputs.

A6. We appreciate your comment. As discussed in the limitations section, we agree that, like other unsupervised RL approaches, our method requires a large amount of interaction with the environment because it learns online without prior data, pretrained models, or predefined rewards. The amount of interaction required is therefore a practical consideration when extending our method to physical robots and visual inputs. We believe that using recent advances in large-scale GPU-accelerated simulation and sim-to-real transfer to extend our method to physical robots and visual inputs is a promising direction for future work.


Thank you again for your thoughtful review. We hope that the additional experiments and clarifications address your concerns. We would welcome the opportunity to discuss any remaining questions.

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer eRsW,

Thank you again for your time and thoughtful feedback on our paper. In response to your comments, we have conducted additional experiments and evaluations, and added new figures to strengthen our paper.

As the discussion period is coming to an end, we wanted to kindly check whether our responses have addressed your concerns. If you have any remaining questions or would like further clarification, please let us know. We would be happy to address them before the deadline.

Reviewer_EAmvOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper solves the problem of planning in the presence only of a goal reward without ANY prior data in the form of demonstrations, or pretrained models. The key is the intrepolation of waypoints in a hyperspherical representation. The approach is online RL without offline data. A critic is learnt for the Q-function self-supervised without external rewards. The main challenge is how to learn the waypoints. A first step with a hierarchical policy offers K waypoints. Their separation is governed by a geometric distribution. A temporal relationship between hyperspherical representation is defined as the inner product between a rotates last state and current state. The hyperspherical representation is also used to score future states by learning an O(n) matrix in order to represent temporal asymmetry. The approach assumes Bayes optimality and uniformity. Also, the distribution of waypoints is factorized to make it tractable and the waypoints are approximated in closed form.

Critical strengths
  • The idea of contrastive representations for long-term planning is novel.

  • The hyperspherical representation allows for waypoint prediction without training an extra high-level policy.

  • The stochasticity in the number of waypoints p(K∣ψ(s),ψ(g))p(K|\psi(s),\psi(g)) and the simplicity of the resulting softmax (p. 19)

  • The reliance on latent representations without having to reconstruct the actual state.

Critical weaknesses
  • My biggest problem is the use of an orthogonal matrix to capture the temporal transitions. The authors are careful to use only the inner product of ψ(s′)Rψ(s)\psi(s')R\psi(s) and not claiming a world model ψ(s′)=Rψ(s)\psi(s')=R\psi(s). Nevertheless, an orthogonal matrix has unit eigenvalues and cannot model contractions or expansions. It has approximately half the degrees of freedom of a linear model in Rn\mathbb{R}^n. So I am wandering how many actual parts of the state can be reached.

  • My second concern is the independence of waypoints. In a markov chain A,B,C,D the distribution becomes p(D|C)p(C|B)p(B|A). What is the approximation error paid for this? If the distribution over trajectories is bimodal, what will happen? Is the coupling of the means sufficient to resolve this?

Questions for authors
  • Need explanation on why hyperspherical. Does spherical geometry really matter?

  • Explain under which assumptions on the dynamics RKR^K is expressive.

  • A simulation experiment with a linear model in R^n showing only the expressivity of ψ(s′)Aψ(s)\psi(s')A\psi(s) would help.

  • A clarification on closed vs approximately closed (what assumptions were made to make it closed).

  • Explain assumption of uniformity of representation

  • Explain effects of factorization of waypoints

4 comments
AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer EAmv,

We sincerely appreciate your positive assessment of our work and your constructive comments, which have helped us strengthen the paper.

We have carefully read your review and prepared our responses below:


Q1. Expressivity of orthogonal RR for modeling temporal transitions.

A1-1. Thank you for raising this point. We understand your concern about expressivity. However, in our method, the orthogonal matrix RR is used to represent rotations on the unit hypersphere rather than arbitrary linear transformations. These rotations preserve the unit norms of the learned representations, allowing us to model their temporal relationships within the same hyperspherical space.

Moreover, we would like to note that in our formulation fH(s,s+)=ψ(s)⊤Rψ(s+)f_H(s,s^+)=\psi(s)^\top R\psi(s^+), RR operates in the learned representation space of ψ\psi rather than directly in the original state space. When ψ\psi sufficiently captures the nonlinearity of the underlying dynamics, RR can be effective in modeling temporal relationships.

A1-2. Consistent with this interpretation, Figure 11 shows that once the matrix dimension nn (i.e., the representation dimension) is sufficiently large, further increasing it provides no consistent performance benefit. We hypothesize that this limited benefit arises because ψ\psi already provides sufficient nonlinear expressivity for RR to model temporal relationships effectively.

Our results on long-horizon manipulation and navigation tasks further demonstrate that this parameterization enables agents to reach a range of goal states across the environments. We would like to refer the reviewer to Figure 21 for visualizations of the waypoint representations.

A1-3. Furthermore, we would like to emphasize that the orthogonality of RR is not merely a design choice, but is essential to our derivation. Our method maps the current state and goal onto the unit hypersphere and infers intermediate waypoints through spherical linear interpolation that accounts for RR. Orthogonality preserves the unit norms of the waypoint representations. Contractions or expansions would generally break this property, so the same derivation of our closed-form waypoint inference would no longer apply.

Q2. Concern about the factorization of the waypoint distribution.

A2-1. Thank you for your insightful comment. Based on your valuable suggestion, we conducted additional experiments in various maze layouts, including those with two routes connecting the same start and goal, to examine whether the inferred waypoint sequence follows a consistent, valid route under the factorized distribution.

We added these experiments to Figure 22 in the main paper. Our method solves all tested mazes and produces valid waypoint sequences across the different layouts. In particular, in the two-route bimodal mazes, the inferred waypoints for each state-goal pair follow one route to the goal without switching to the other. These results provide empirical support for the effectiveness of the coupled means in maintaining waypoint consistency under the factorized distribution.

A2-2. The factorization also allows us to handle different numbers of waypoints KK within a single formulation, without learning separate representations or high-level policy for each KK. As described in the paper, the waypoint factorization q(w1:K∣s,g)=∏i=1Kqi(wi∣s,g)q(w_{1:K}\mid s,g)=\prod_{i=1}^{K}q_i(w_i\mid s,g) allows us to optimize qq for different KK under the same objective in Eq. 10. Thus, the factorization is a modeling choice that makes inference in the learned representation space more tractable and supports our closed-form derivation.

A2-3. From the perspective of policy execution, the policy π(a∣s,z)\pi(a\mid s,z) pursues a single waypoint zi∗z_{i^*} at each planning step. The planner selects this waypoint from the sampled candidates z1:Kz_{1:K} between the current state representation ψ(s)\psi(s) and the goal representation ψ(g)\psi(g) based on their estimated reachability, as described in Eq. 15. The policy therefore does not need to execute the entire sampled sequence. This reduces reliance on correlations across the entire sampled sequence.

AuthorsOpenReview ↗

Official Comment by Authors

Q3. The role of spherical geometry.

Thank you for raising this point. As described in the paper, spherical geometry is essential to the key components of our method, including adaptive planning, closed-form waypoint inference, and waypoint selection for the low-level policy.

A3-1. First, the hyperspherical geometry supports high-level planning. Our planning requires repeated applications of RR, expressed as RKR^K. Since RKR^K preserves unit norms, we can compare different numbers of waypoints KK through alignment scores on the same hypersphere. Without this norm preservation, repeated matrix multiplications can amplify or shrink representations, making comparisons across KK sensitive to scale differences across tasks. Furthermore, with the von Mises–Fisher likelihood defined on the unit hypersphere, we derive a softmax distribution for selecting KK. Together, these properties make hyperspherical geometry essential to our formulation for selecting the number of waypoints, without training a separate high-level policy.

A3-2. Furthermore, we would like to emphasize that our hyperspherical representations allow us to reformulate our objective into a tractable form. Specifically, under the uniformity assumption on the unit hypersphere, we can reformulate the KL terms in Eq. 9 as entropy regularization, allowing us to model waypoint stochasticity. Moreover, since the learned orthogonal matrix RR preserves the hyperspherical geometry, we can derive our closed-form waypoint solution from the reformulated objective in Eq. 10 using only spherical linear interpolation.

A3-3. Hyperspherical geometry also bounds the reachability scores ρi=ϕ(s,ai)⊤zi\rho_i=\phi(s,a_i)^\top z_i in Eq. 15 within [−1,1][-1,1], since both ϕ(s,ai)\phi(s,a_i) and ziz_i lie on the unit hypersphere. Without constraining representations to the unit hypersphere, these scores would also depend on representation norms, which can vary across environments and require separate tuning of the reachability threshold ϵ\epsilon for each task. In contrast, our hyperspherical formulation provides a common scale, allowing us to use the same value, ϵ=0.4\epsilon=0.4, across all 12 benchmark environments, achieving the highest success rates on most tasks.

Q4. Explain under which assumptions on the dynamics RKR^K is expressive.

A4. As discussed in A1, RR models temporal relationships in the learned representation space of ψ\psi rather than directly in the original state space. We use ψ\psi to capture the relevant nonlinear structure of the dynamics. Our use of RKR^K assumes that the same orthogonal transformation RR can approximate temporal relationships between successive waypoints in this space. Please refer to A1-2 for empirical results.

Q5. A simulation experiment with a linear model in Rn\mathbb{R}^n.

A5. As discussed in A1-3, using an unconstrained linear model in Rn\mathbb{R}^n does not generally preserve hyperspherical geometry, which is essential to our formulation. Consequently, the same closed-form derivation no longer applies. Nevertheless, we constructed an alternative with an unconstrained linear model to match our setting as closely as possible, with the following changes:

  • Orthogonal matrix RR → unconstrained matrix AA.
  • Spherical linear interpolation → Euclidean linear interpolation.
  • Hyperspherical representations → representations in Rn\mathbb{R}^n.

The results below show that this alternative achieved low success rates (averaged over five seeds) across the five benchmark tasks.

Success rate3-tower (gripper)8-towerAnt big MazeAnt U4 MazeAnt U5 Maze
Rotation matrix RR (Ours)0.879±0.0210.369±0.0270.925±0.0080.954±0.0070.848±0.023
Unconstrained matrix AA with matched condition0.000±0.0000.000±0.0000.129±0.0220.068±0.0850.000±0.000

We observed that the norm of AA continued to grow during training despite regularization and that the norms of the learned representations varied across tasks. We believe that these differences in scale may have contributed to the poor performance of the Euclidean variant. Since its performance did not stabilize, we report its peak evaluation success rate during training for each task.

AuthorsOpenReview ↗

Official Comment by Authors

Q6. Assumptions underlying the closed-form solution.

A6. We appreciate your comment. As stated in the main text and appendix, we derive the waypoint objective under Assumptions 1 and 2 (Bayes optimality and uniformity) and the Markov assumption. We then apply a small-variance approximation and obtain closed-form solutions to the approximated objective. We will further clarify this distinction in the final version.

Q7. Explain assumption of uniformity of representation.

A7. Thank you for your question. The uniformity assumption (Assumption 2 in the paper) states that representations ψ(s)\psi(s) learned under the symmetric InfoNCE objective have a uniform marginal distribution on the unit hypersphere. Intuitively, contrastive learning pulls positive pairs together and pushes negative pairs apart, encouraging representations to spread uniformly over the hypersphere. We provide a proof in Appendix A.1 following prior work [1].

We use this assumption to rewrite the KL terms in Eq. 9 as entropy terms, yielding Eq. 10. The resulting entropy regularization allows us to control the stochasticity of the waypoint representations. The detailed derivation is provided in Appendix A.2.

[1] Wang and Isola. “Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere.”, 2020.


Thank you again for your valuable and insightful review.

We hope our responses have sufficiently addressed your questions, and we would be happy to provide any further clarification if needed.

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer EAmv,

Thank you again for your time and thoughtful feedback on our paper. In response to your comments, we have conducted additional experiments and evaluations, and added new figures to strengthen our paper.

As the discussion period is coming to an end, we wanted to kindly check whether our responses have addressed your concerns. If you have any remaining questions or would like further clarification, please let us know. We would be happy to address them before the deadline.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Author Response Summary

Dear Area Chair and Reviewers,

Thank you for your time and constructive feedback. As the author-reviewer discussion period comes to an end, we summarize the reviewers’ main concerns and how we addressed them.


Reviewer EAmv: We clarified that the orthogonal matrix RR models temporal relationships in the learned representation space and that preserving unit norms is essential to our closed-form derivation. We also presented additional experiments in mazes with two routes between the same start and goal, showing that the waypoints inferred under our factorization consistently followed a single valid route (Fig. 22). Finally, we explained why our formulation relies on hyperspherical geometry, discussed the assumptions underlying our derivation, and presented experimental evidence supporting our use of this geometry.

Reviewer eRsW: We discussed how hyperspherical geometry supports three key components of our method: choosing the number of waypoints, inferring waypoint representations in closed form, and selecting a waypoint for the low-level policy. In additional experiments across five tasks, we compared our method with a Euclidean variant and two alternative interpolation methods under matched conditions. Our method achieved higher success rates than all three alternatives on each task, providing empirical support for our use of hyperspherical geometry in planning. To address the reviewer’s concern about subgoal reachability, we compared the reachability scores with empirical success rates and showed that higher scores generally corresponded to higher success rates (Figs. 17–18).

Reviewer qR1w: We formally defined the distributions used in our derivation and noted that we treat the Markov factorization of the waypoint sequence as a modeling assumption. To examine this assumption, we conducted additional experiments that provided empirical support for it. We also added figures showing experimental results that support the uniformity and small-variance assumptions used in our derivation (Figs. 13–14).


Once again, we sincerely thank the Area Chair and Reviewers for their thoughtful evaluation and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.