- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes C-Slerp, an online, self-supervised, goal-conditioned RL method built on contrastive RL (CRL). It learns two scores: a state–state score on the unit hypersphere, where is a learned orthogonal matrix (via a Cayley transform) that makes the score asymmetric; the standard CRL critic . Waypoint inference is posed as a variational problem. Under a Bayes-optimality assumption and a uniformity assumption, the authors derive closed-form waypoint distributions: slerp from to , with each point mapped back by , plus a small isotropic noise term. The number of waypoints is sampled from a softmax over the alignment scores .
Critical strengths
-
Simple, appealing idea. The planner adds no network parameters and does no graph search. It reuses the learned geometry both to choose how many stages to plan and to place the waypoints. Choosing the number of stages per problem is a well-motivated gap in hierarchical GCRL.
-
Strong empirical results on hard sparse-reward tasks. Gripper-based stacking of 3–5 cubes with no curriculum, rewards or demonstrations, where every baseline gets zero success, is a notable result.
Critical weaknesses
-
in Eq. 5 is described only in words. Sec. 3 defines the discounted occupancy conditioned on , i.e., . Eq. 7 and App. A.2 use a state-only kernel that is never defined, i.e., which action marginalization and which policy or mixture it is taken over.
-
The Markov factorization ignores the commanded goal. A.2 writes But is goal-conditioned and hierarchical: its actions depend on the commanded goal, the active subgoal and the replanning phase. Treating these as hidden variables gives a mixture of Markov chains, which is not Markov in states. The derivation also conflates the commanded goal with the reached endpoint . Lemma 2 (A.4) relies on the same assumption. Eysenbach et al. (2024) make a similar factorization, but with offline data from a single policy that is not goal-conditioned, where it is easier to justify.
Questions for authors
-
Please give a formal definition of and , including the policy or mixture they are taken over. How do you justify the Markov factorization given that is goal-conditioned and hierarchical?
-
Can you report empirical diagnostics for Assumption 2 (e.g., a uniformity loss or an estimate of the embeddings' entropy on ) and for the condition under the learned ?
4 comments
Official Comment by Authors
Q3. Markov factorization of the waypoint sequence.
A3-1. Thank you for raising this point. As you pointed out, the waypoint sequence generated by our hierarchical goal-conditioned policy need not be Markovian even when the environment is an MDP. For this reason, we treated the Markov property of the waypoint sequence as a separate modeling assumption in the main text and appendix. We will state this point more clearly in the final version.
A3-2. To examine the Markov assumption empirically, we tested whether . We collected rollouts from the trained policies across a range of commanded goals, with 5,000 rollouts per task. For the main analysis, we sampled waypoint triples from trajectories using geometrically distributed time intervals.
Following the classifier-based approach in [1], we estimated the conditional mutual information (CMI) between and given :
We trained two binary classifiers with inputs and , respectively. We duplicate in the first classifier’s input so that both classifiers have the same input dimension and model size for a fair comparison. Each classifier estimated the probability that its input was a positive example. For positive examples, all waypoints came from the same trajectory. For negative examples, we replaced only with a waypoint sampled from a different trajectory. The prediction target was the agent’s position in the mazes. In the manipulation tasks, the target included the gripper pose and the block positions. The results are shown below:
| Mutual information (nats) | Estimated CMI [95% CI] | Null test [95% CI] | ||
|---|---|---|---|---|
| ant_big_maze | 2.476 ± 0.042 | 2.441 ± 0.033 | [−0.043, −0.013] | [−0.081, −0.035] |
| ant_u5_maze | 1.946 ± 0.021 | 1.919 ± 0.041 | [−0.024, −0.003] | [−0.025, −0.013] |
| 3_gripper | 2.082 ± 0.025 | 2.029 ± 0.042 | [−0.115, −0.005] | [−0.072, +0.039] |
| 4_gripper | 1.828 ± 0.028 | 1.741 ± 0.093 | [−0.186, +0.117] | [−0.069, +0.176] |
We report mutual information (MI) as (mean ± standard deviation) over five classifier seeds. Although true CMI is nonnegative, its estimate can be negative because we use two separate classifier networks to estimate the two MI terms and then take their difference. To account for this estimation error, we compared the observed estimates with those from a null test in which we replaced only with a waypoint sampled from a different trajectory. This gives us a baseline for interpreting the estimates when the true CMI is zero.
Across the four tasks, adding did not increase the estimated mutual information, and the 95% confidence intervals for the observed and null CMI estimates overlapped in each task. On these trajectories, we found little evidence that provided additional information about the next maze position or the next gripper pose and block positions in the manipulation tasks given .
[1] Mukherjee et al. “CCMI: Classifier based conditional mutual information estimation.” 2020.
Official Comment by Authors
Q4. Empirical diagnostics for the uniformity assumption.
A4. Following your insightful suggestion, we added an empirical analysis of the uniformity assumption to the revised manuscript. Figure 13 shows that as the uniformity term of the symmetric InfoNCE loss decreases during training, the empirical distribution approaches the ideal distribution for points sampled uniformly on the unit hypersphere. These results provide empirical support for our uniformity assumption across all evaluated tasks.
Q5. Empirical diagnostics for the small-variance assumption.
A5. We appreciate you raising this point. Please see Figure 14 in the revised manuscript, where we plot the quantity during training. Across all tasks, this quantity converges to a value much smaller than 1, providing empirical support for our small-variance assumption.
Thank you again for your valuable and insightful review. Also, please let us know if our responses have addressed your questions. If anything requires further clarification, please do not hesitate to let us know.
Official Comment by Authors
Dear Reviewer qR1w,
Thank you again for your time and thoughtful feedback on our paper. In response to your comments, we have conducted additional experiments and evaluations, and added new figures to strengthen our paper.
As the discussion period is coming to an end, we wanted to kindly check whether our responses have addressed your concerns. If you have any remaining questions or would like further clarification, please let us know. We would be happy to address them before the deadline.
Official Comment by Authors
Dear Reviewer qR1w,
We are grateful for your careful reading of our paper and your valuable suggestions.
Our responses are provided below.
Q1. Formal definition of pπ^γ(s+∣s).
A1. Thank you for raising this point. To define pπ^γ(s+∣s), we consider the following process. We independently sample an initial state s0∼p0 and a commanded goal gcmd∼pg. Starting from s0, we run the hierarchical goal-conditioned policy π^ described in Sec. 4.1 with gcmd fixed throughout the rollout. Every h steps, the planner takes the current state and gcmd as inputs and selects a waypoint representation z. The low-level policy πθ then takes actions conditioned on the current state and z for the next h steps. Let (s,a)=(st,at) denote the state and action at time t along the rollout. We independently draw T∼Geom(1−γ) on {1,2,…} and define s+=st+T, the state reached T steps after s.
This process induces the joint distribution pπ^γ(s,gcmd,a,s+). By marginalizing this joint distribution and conditioning on s or (s,a), we obtain
pπ^γ(s+∣s)=pπ^γ(s)∫G∫Apπ^γ(s,gcmd,a,s+),da,dgcmd,
pπ^γ(s+∣s,a)=pπ^γ(s,a)∫Gpπ^γ(s,gcmd,a,s+),dgcmd. Here, pπ^γ(s) and pπ^γ(s,a) are the corresponding marginals of the same joint distribution. Thank you again for pointing this out. We will make these definitions explicit in the final version.
Q2. Formal definition of pπ^γ,K(w1:K∣s,g).
A2. We appreciate your comment. As described in Sec. 4.1, we consider K intermediate waypoints w1:K=(w1,…,wK) along a trajectory generated by π^. As in A1, we sample the commanded goal gcmd∼pg and hold it fixed throughout the rollout. Starting at w0=s, we draw independent intervals Ti∼Geom(1−γ) for i=0,…,K. Each wi+1 is the state reached Ti steps after wi. We set g=wK+1, the endpoint on which we condition. This endpoint g need not equal the commanded goal gcmd.
This process defines the joint distribution pπ^γ,K(s,gcmd,w1:K,g). We integrate out gcmd and condition on (s,g) to obtain
pπ^γ,K(w1:K∣s,g)=pπ^γ,K(s,g)∫Gpπ^γ,K(s,gcmd,w1:K,g),dgcmd.
Here, pπ^γ,K(s,g) is the joint marginal of the starting state s=w0 and the reached endpoint g=wK+1.