acceptodds
Under review as a conference paper at ICLR 2027

Direction-Conditioned Policies for Online Goal-Conditioned Reinforcement Learning

Abstract

Contrastive Reinforcement Learning (CRL) learns representations that estimate goal reachability, yet its policy remains conditioned on raw goals and therefore does not directly exploit the geometry encoded by its critic. We introduce Direction-Conditioned Policies (DCP), a method built around a small modification to CRL: DCP selects previously visited states as waypoints during online training and conditions the policy on their direction and distance in representation space. At deployment, DCP applies the same interface directly to the final goal, requiring neither waypoint selection nor planning. Across nine navigation and manipulation tasks, DCP attains higher final success rates than CRL on seven tasks and spends more time near the goal on seven. Controlled maze experiments further show that DCP captures shortest-path geometry more accurately and that the supplied direction causally influences the actor's behavior. We identify waypoint coverage and ranking as limits to exploration, and show that learned candidate generation improves goal reaching in two controlled mazes.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
goal-conditioned reinforcement learning, contrastive reinforcement learning, self-supervised reinforcement learning, representation learning, online reinforcement learning, long-horizon control, robotic manipulation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_bKfEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DCP changes how a CRL actor receives its target. It uses the direction and distance between goal embeddings, selects visited waypoints during data collection, and acts directly toward the final goal at evaluation. Experiments on nine simulated tasks show higher final mean success rates than CRL on seven tasks and more time near the goal on seven tasks.

Critical strengths

+The change is simple and gives useful improvements on several tasks while keeping the CRL objective and network depth. +The analysis goes beyond learning curves. It checks the learned geometry, tests whether the actor uses the direction input, and reports cases where waypoint selection or candidate generation does not help.

Critical weaknesses

-The main method is a small extension of existing direction-conditioned policies. Latent directions have already been used in methods such as HILP and METRA. Bringing this idea into online CRL is useful, but the main change is the actor input and waypoint selection, while the baseline learning objectives remain largely unchanged. -The experiments do not fully explain where the gains come from. In Table 3, direct direction conditioning without waypoint selection has the highest mean performance in both controlled environments, but this version is missing from the main benchmark comparison. The actor also has 11–21% more parameters, without a parameter-matched control. These gaps make it difficult to separate the value of the learned representation from the other changes. -The broader lesson from the empirical study remains unclear. The direction interventions show that the actor uses its input, but do not establish why the learned geometry is a better choice than other goal representations. The study uses one CRL configuration, and the main mechanism analyses focus on controlled maze and pushing tasks. I would expect clearer evidence about when this conditioning should help before drawing a broader conclusion for CRL design.

Questions for authors
  1. What is the main new insight beyond adapting the direction-conditioning idea in HILP and METRA to online CRL? Which challenges specific to online learning does DCP address?
  2. Can you compare DCP with direct-goal direction conditioning on representative benchmark tasks? This simpler version has the highest mean performance in both Table 3 settings, so it remains unclear when waypoint selection helps.
  3. Can you include a parameter-matched raw-goal actor? This would help separate the benefit of the learned goal representation from the 11–21% increase in actor parameters.
  4. What do the current experiments tell us about when DCP helps or hurts? Which findings extend beyond the controlled maze and pushing studies, and what evidence supports them?
5 comments
AuthorsOpenReview ↗

Response to Reviewer bKfE (1/3)

We thank the reviewer for appreciating the simplicity of our change, its improvements on several tasks, and the depth of our analysis beyond learning curves. We address the critical weaknesses mentioned and questions asked by (1) running direct-goal direction conditioning (Direct-DCP) on all nine benchmark tasks with five seeds, (2) training a parameter-matched raw-goal CRL actor on the PusherEasy task to isolate whether more parameters help DCP unfairly, (3) using these results to show when DCP helps or hurts at benchmark scale, and (4) clarifying the new insight and the online-specific challenge relative to HILP and METRA. The new results show that the learned direction-and-distance input is the main source of the largest gains over raw goals, and that waypoint collection is needed on tasks where the final goal is rarely reached early in training. Full DCP is a trade-off: it gives up some peak success but, unlike Direct-DCP, never fails outright on a task, making it the safer choice. Our conclusion that DCP improves over CRL still holds.

W2, Q2, Q3: Direct-goal direction conditioning on the benchmark and a parameter-matched raw-goal actor

We now report Direct-DCP on all nine tasks with five seeds. Direct-DCP uses the DCP actor input but collects experience toward the final goal, without the pool or scorer. Together with CRL (raw final goal) and SSGC (raw waypoint coordinates), the benchmark now covers all four combinations of actor input (raw vs. learned direction) and "collection target" (final goal vs. waypoint).

Table: Final evaluation, five seeds per method (paper Table 2). Success rate (%) / time near goal (steps out of 1,000).

TaskCRLSSGCDCPDirect-DCP
PusherEasy4.4 / 2634.1 / 18155.4 / 29874.2 / 296
PusherHard28.8 / 12627.8 / 17147.7 / 28169.0 / 247
PusherHard Far12.5 / 489.0 / 4947.4 / 26154.1 / 176
AntMaze Big18.3 / 8923.3 / 10029.9 / 11528.4 / 117
AntMaze Hardest12.2 / 4814.2 / 6616.1 / 6318.4 / 59
Ant U-Maze40.8 / 26149.8 / 25355.0 / 23846.2 / 270
Humanoid U-Maze0.0 / 0.00.0 / 0.08.4 / 3.60.0 / 0.0
AntPush45.6 / 14351.4 / 19942.7 / 18219.5 / 65
AntSoccer22.6 / 7614.1 / 4812.9 / 4221.4 / 72

For the parameter-matched control, we widen only the first hidden layer of the raw-goal CRL actor (256 → 313 units) and keep all other settings unchanged.

Table: Parameter-matched raw-goal actor on PusherEasy (paper Section 4.2, Appendix H).

MethodActor parametersSeedsSuccess (%)Time near goal
CRL75,53454.426
CRL, first layer 31391,49454.731
DCP91,406555.4298

We thus safely conclude that the 11–21% increase in actor parameters does not explain DCP's gain.

W3, Q4: When does DCP help or hurt, and what evidence supports this beyond the controlled studies?

With the added experiments from Direct-DCP, we can better answer this question with the following points:

  1. The learned direction helps most on the Pusher tasks. In these tasks, direction-conditioned variants outperform raw-input variants in both success and time near goal. So, manipulation tasks like these, where raw goals in euclidean space offer little guidance, could be a setting where direction-conditioning in representation space helps a lot.
  2. Waypoint collection helps where the final goal is rarely reached early in training. On AntPush, no method reaches 1% success before 18M steps, and three of five Direct-DCP seeds never reach the goal, while all five DCP seeds learn. On Humanoid U-Maze, every DCP seed reaches 1% success within 13M steps, while Direct-DCP ends at 0% in every seed.
  3. Waypoints also keep the agent at the goal longer on pushing/manipulation tasks. On PusherHard and PusherHard Far, DCP stays for 66% and 61% of the steps after first reaching the goal, compared with 39% and 36% for Direct-DCP, with no seed overlap.
  4. Waypoints can hurt where direct collection already reaches the goal. This is the case on AntSoccer (12.9% vs. 21.4% for Direct-DCP, Welch p = 0.006) and in the controlled open room (Table 3).
AuthorsOpenReview ↗

Response to Reviewer bKfE (2/3)

So which variant should one use? Full DCP is a trade-off. Direct-DCP has higher success on five tasks, but it fails outright on two, and which tasks these are is not obvious before training. DCP gives up some peak success, but it never fails outright on a task: every DCP seed on every task reaches at least 1% success during training.

Table: Reliability across the nine tasks, five seeds each (paper Section 4.2). Relative values are a method's mean divided by the better of DCP and Direct-DCP on that task.

CRLDCPDirect-DCP
Seeds reaching 1% success (of 45)444538
Tasks beating CRL, success / time near goal–7 / 76 / 6
Worst task, relative success–0.60 (AntSoccer)0.00 (Humanoid U-Maze)
Worst task, relative time near goal–0.59 (AntSoccer)0.00 (Humanoid U-Maze)

When one does not want to gamble on whether the final goal will be reached early in training, DCP is the better bet.

Why the critic's own representation, rather than another goal representation? We argue for this particular goal representation, because it is the space in which the actor is already being trained on. In CRL, the actor is optimized to choose actions whose embedding ϕ(s,a)\phi(s,a) lies close to ψ(g)\psi(g) (Eq. 1), so the critic scores every action by where it moves the agent in ψ\psi-space. DCP's input, ψ(y)−ψ(q(s))\psi(y)-\psi(q(s)), is a displacement in this same space: the actor is told where the target lies in exactly the coordinates in which its own objective is measured. Other learned goal representations, such as those of HILP [a] and METRA [b], live in a different space from the critic that trains the actor and need their own objective. Online, that objective would also have to be trained on the data the actor collects, which adds a second moving representation to the same coupling discussed in our response to W1. DCP instead reuses the representation CRL already learns, at no extra cost.

Finally, we stress on the view that DCP is not a competitor to other goal representations but a way of using one. Since it changes only the actor input, any learned representation can take the place of ψ\psi in the direction and distance, and the same input can be directly combined with deeper or action-chunked CRL critics.

W1, Q1: The new insight beyond HILP and METRA, and the challenges specific to online learning

We invite the reviewer to view this work as an empirical study of whether the direction in CRL's own representation space can be used to condition the actor policy in online GCRL. HILP [a] and METRA [b] learn their representations with dedicated objectives (offline, or for unsupervised skill discovery) and then condition a policy on directions in that space. In our case, DCP adds no representation objective. And the actor reads the goal encoder of the critic while it is still being trained, on data that the same actor collects. This creates the following stability problem that does not arise offline: The direction toward a goal the agent has never reached is computed in a poorly trained region of the representation, while the data that would train that region depends on the policy following that direction. Our work addresses this challenge specifically and provides insights on the same. Since the other offline methods learn from a fixed dataset or with a separate objective, they address a different setting from ours.

We have thus shown one case of a simple use of direction in a learned representation space as a conditioning variable in online GCRL. We hope that this work encourages practitioners in the field to use learned representations as a conditioning variable for the policy, even while these representations are still being jointly trained.

We thank you again for your review, which has helped make our work much stronger. Please let us know if any more questions arise about this work, and we will do our best to address them!

AuthorsOpenReview ↗

Response to Reviewer bKfE (3/3)

Changes to the paper: We have revised the paper accordingly, with all changes marked in blue. Section 4.1, Table 2, and Figure 3 add Direct-DCP on all nine tasks. Section 4.2 adds what this comparison shows about each component, the goal-holding result, and the parameter-matched control. The abstract, introduction, and conclusion now summarize these findings. The introduction and Section 3.2 now state that DCP uses the critic's own goal encoder, the space in which the critic scores the actor's actions, and the conclusion lists other learned goal representations as future work. The new Appendix H gives the first-reach and goal-holding statistics and statistical tests. Despite these changes, we believe that our initial claims and methodology remain the same, with some added insights on when waypoint pooling works for DCP and when it might not.

Reference(s):

[a] Park, S., Kreiman, T., and Levine, S. Foundation Policies with Hilbert Representations. International Conference on Machine Learning (ICML), 2024.

[b] Park, S., Rybkin, O., and Levine, S. METRA: Scalable Unsupervised RL with Metric-Aware Abstraction. International Conference on Learning Representations (ICLR), 2024.

Reviewer_vtFbOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The authors point out that the critic in CRL has already learned representations related to goal reachability, which the actor could also exploit. This idea is particularly intuitive in navigation tasks with obstacles. DCP therefore aims to provide a more informative input representation, allowing the actor to better use the geometric relationships learned by the critic. Specifically, by constructing vv, rr, and dd, the method provides the actor with the physical state, as well as direction and distance information in the learned representation space.

Critical strengths
  1. The proposed method is simple, straightforward, and practical, with a reasonable and sound theoretical explanation.
  2. The experimental evaluation is extensive. The main experiments cover nine tasks, three methods, and five random seeds, along with additional ablation studies.
  3. The paper is rigorously written and logically organized, and the authors openly acknowledge the limitations of their algorithm.
Critical weaknesses
  1. This is a meaningful methodological improvement within an existing framework, with clear differences in its mechanism, although the degree of conceptual novelty is limited. Compared with existing methods such as HILP, DCP introduces relative direction and distance information into online CRL, reuses the goal encoder of the existing critic, preserves the main CRL optimization objective, and investigates how this input representation interacts with online landmark collection.
  2. The paper mainly changes two factors: whether the actor receives raw coordinates or direction and distance information in the learned representation space, and the landmarks selected during data collection. The main experimental design is reasonable. However, Direct Direction Conditioning achieves the highest mean performance in the two main environments. The authors report this result and explicitly acknowledge that landmarks may slow down learning. Nevertheless, the nine main tasks only compare CRL, SSGC, and the full DCP method, without including Direct Direction Conditioning. I suggest adding this baseline to all nine tasks for a more complete comparison.
  3. The theoretical explanation in the appendix is reasonable, but it is closer to an empirical justification than a rigorous theoretical analysis.
Questions for authors

As noted in the Critical Weaknesses above.

5 comments
AuthorsOpenReview ↗

Response to Reviewer vtFb (1/3)

We thank the reviewer for appreciating the simplicity and practicality of our method and the rigor of our writing. We address the critical weaknesses the reviewer has mentioned by (1) running five seeds of direct direction conditioning on all nine benchmark tasks, (2) training parameter-matched CRL on the PusherEasy task to isolate whether more parameters help DCP unfairly, (3) clarifying the online-specific problem that DCP addresses relative to HILP and other offline methods, and (4) clarifying the scope of our theoretical statement. The new results show that the learned direction-and-distance input is the main source of the largest gains, and that waypoint collection is needed on tasks where the final goal is rarely reached early in training. Full DCP is a trade-off: it gives up some peak success but, unlike Direct-DCP, never fails outright on a task, making it the safer choice. Our conclusion that DCP improves over CRL still holds.

W2: Direct direction conditioning on all nine tasks

We now report Direct-DCP on all nine tasks with five seeds. Direct-DCP uses the DCP actor input but collects experience toward the final goal, without the pool or scorer. Together with CRL and SSGC, the benchmark now covers all four combinations of actor input (raw vs. learned direction) and "collection target" (final goal vs. waypoint).

Table: Final evaluation, five seeds per method (paper Table 2). Success rate (%) / time near goal (steps out of 1,000).

TaskCRLSSGCDCPDirect-DCP
PusherEasy4.4 / 2634.1 / 18155.4 / 29874.2 / 296
PusherHard28.8 / 12627.8 / 17147.7 / 28169.0 / 247
PusherHard Far12.5 / 489.0 / 4947.4 / 26154.1 / 176
AntMaze Big18.3 / 8923.3 / 10029.9 / 11528.4 / 117
AntMaze Hardest12.2 / 4814.2 / 6616.1 / 6318.4 / 59
Ant U-Maze40.8 / 26149.8 / 25355.0 / 23846.2 / 270
Humanoid U-Maze0.0 / 0.00.0 / 0.08.4 / 3.60.0 / 0.0
AntPush45.6 / 14351.4 / 19942.7 / 18219.5 / 65
AntSoccer22.6 / 7614.1 / 4812.9 / 4221.4 / 72

These results help explain the role of each component:

  1. The direction input provides the largest gains. On all three Pusher tasks, both direction-conditioned variants outperform both raw-input variants in both metrics, and Direct-DCP has the highest success.
  2. Waypoint collection is needed where the final goal is rarely reached early in training. On AntPush, no method reaches 1% success before 18M steps, and three of five Direct-DCP seeds never reach the goal, while all five DCP seeds learn. On Humanoid U-Maze, every DCP seed reaches 1% success within 13M steps, while Direct-DCP ends at 0% in every seed.
  3. Waypoints keep the agent at the goal longer on pushing tasks. On PusherHard and PusherHard Far, DCP stays at the goal for 66% and 61% of the steps after first reaching it, compared with 39% and 36% for Direct-DCP, with no seed overlap.
  4. Waypoints can make learning worse when the agent already achieves goal during initial stages of training. This is the case on AntSoccer (21.4% for Direct-DCP vs. 12.9% for DCP, Welch p = 0.006), consistent with the open-room result in Table 3 that the reviewer noted.
AuthorsOpenReview ↗

Response to Reviewer vtFb (2/3)

So which variant should one use? Full DCP is a trade-off. Direct-DCP has higher success on five tasks, but it fails outright on two, and which tasks these are is not obvious before training. DCP gives up some peak success, but it never fails on a task: every DCP seed on every task reaches at least 1% success during training.

Table: Reliability across the nine tasks, five seeds each (paper Section 4.2). Relative values are a method's mean divided by the better of DCP and Direct-DCP on that task.

CRLDCPDirect-DCP
Seeds reaching 1% success (of 45)444538
Tasks beating CRL, success / time near goal–7 / 76 / 6
Worst task, relative success–0.60 (AntSoccer)0.00 (Humanoid U-Maze)
Worst task, relative time near goal–0.59 (AntSoccer)0.00 (Humanoid U-Maze)

When one does not want to gamble on whether the final goal will be reached early in training, DCP is the better bet.

We also widen only the first hidden layer of the raw-goal CRL actor (256 → 313 units) to match DCP's actor parameter count (91,494 vs. 91,406). On PusherEasy (five seeds), it reaches 4.7% success and 31 steps near the goal, compared with 4.4% and 26 for default CRL and 55.4% and 298 for DCP. We thus safely conclude that the 11–21% increase in actor parameters does not explain DCP's gain.

W1: Conceptual novelty relative to HILP

We invite the reviewer to view this work as an empirical study of whether the direction in CRL's own representation space can be used to condition the actor policy in online GCRL. HILP [a] learns a Hilbert representation offline with a dedicated objective and then trains directional policies on a fixed dataset. In our case, DCP adds no representation objective. And the actor reads the goal encoder of the critic while it is still being trained, on data that the same actor collects. This creates the following stability problem that does not arise offline: The direction toward a goal the agent has never reached is computed in a poorly trained region of the representation, while the data that would train that region depends on the policy following that direction. The new benchmark results show when this matters. Where the final goal is reached early (Pusher, AntSoccer, Ant U-Maze), conditioning directly on the final-goal direction is sufficient. Where it is not (AntPush, Humanoid U-Maze), visited waypoints, whose directions are learned from collected experience, are needed. As such, we demonstrate a trade-off between direct goal conditioning, which can reach higher success, and the more stable waypoint-pooled conditioning of full DCP, which learns on every task.

W3: The theoretical explanation is closer to an empirical justification

We agree, and we keep our contributions scoped to this empirical justification. As stated in our response to W1, we view this work as an empirical study, and the proposition in Appendix C is not claimed as a theoretical contribution. It serves as a geometric explanation: the disagreement between the waypoint and goal directions is determined by the latent triangle slack, so on-path waypoints should give nearly the same action as the final goal. Section 4.5 offers empirical evidence for this justification, i.e. action cosine is 0.789 for on-path waypoints vs. −0.072 for off-path waypoints. We leave a more theoretical investigation of CRL's representation space for future work.

We have thus shown one case of a simple use of direction in a learned representation space as a conditioning variable in online GCRL. We hope that this work encourages practitioners in the field to use learned representations as a conditioning variable for the policy, even while these representations are still being jointly trained.

We thank you again for your review, which has helped make our work much stronger. Please let us know if any more questions arise about this work, and we will do our best to address them!

AuthorsOpenReview ↗

Response to Reviewer vtFb (3/3)

Changes to the paper: We have revised the paper accordingly, with all changes marked in blue. Section 4.1, Table 2, and Figure 3 add Direct-DCP on all nine tasks. Section 4.2 adds what this comparison shows about each component, the goal-holding result, and the parameter-matched control. The abstract, introduction, and conclusion now summarize these findings. The introduction and Section 3.2 now state that DCP uses the critic's own goal encoder, the space in which the critic scores the actor's actions, and the conclusion lists other learned goal representations and a theoretical study of CRL's representation space as future work. The new Appendix H gives the first-reach and goal-holding statistics and statistical tests. Despite these changes, we believe that our initial claims and methodology remain the same, with some added insights on when waypoint pooling works for DCP and when it might not.

Reference(s):

[a] Park, S., Kreiman, T., and Levine, S. Foundation Policies with Hilbert Representations. International Conference on Machine Learning (ICML), 2024.

Reviewer_CPUbOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Direction-Conditioned Policies (DCP), a modification to the actor in contrastive RL (CRL) for online goal-conditioned learning. Instead of conditioning on raw goal coordinates, the actor receives the full state together with the normalized direction and distance from the current achieved goal to a target, both computed in the critic’s goal-encoder space. During data collection, the target is a waypoint chosen from a pool of recently visited states by an inner-product score against the final goal. During actor updates, the target is a hindsight future state. At deployment, the target is the final goal itself, so no pool, scorer, or planning is needed. Across nine JaxGCRL navigation and manipulation tasks with five seeds, DCP achieves higher final mean success than CRL on seven tasks, with the largest gains on the Pusher tasks. Controlled experiments in small mazes and a puck-pushing task examine four things: the learned geometry, whether the actor relies on the direction input, whether on-path waypoints preserve the final-goal action, and the limits of waypoint selection.

Critical strengths
  1. A simple, low-cost modification with clear gains on several tasks. DCP changes only the actor’s input layer. It keeps the CRL critic objective, network depth, and actor loss, and at deployment it needs only the actor and goal encoder. The overhead is small: throughput is about 95% of CRL, and the waypoint pool is discarded after training. Despite this simplicity, the improvements on several tasks are substantial and consistent across seeds, most notably on PusherEasy, PusherHard, PusherHard Far, and AntMaze Big, where the confidence intervals are clearly separated from CRL’s. The appendix analysis also shows that DCP tends to begin succeeding earlier and is less likely to fall back below a success threshold once it crosses it. Because the change is orthogonal to recent CRL improvements such as depth scaling and action-sequence conditioning, it could plausibly be combined with them.

  2. Careful, transparent diagnostic analysis, including negative results. The paper goes beyond benchmark numbers and tests each link in its motivation directly. It compares learned distances to exact graph shortest paths, and it runs causal interventions on the direction input (reversed, orthogonal, zero, wrong-goal), which show the actor relies on it. It checks whether on-path waypoints preserve the final-goal action, and it measures the gap between candidate availability and scorer selection. The authors report results that cut against their own method openly: waypoint scoring hurts in the open room, improved geometry does not translate into better control in the U-maze, the default scorer rarely picks on-path candidates, and AntSoccer remains a clear failure even with seeded successful states. Seed counts, uncertainty measures, and parameter differences are documented explicitly. This honesty makes the paper a useful reference for understanding what the critic’s representation can and cannot provide to the actor.

Critical weaknesses
  1. The source of the main gains is not identified. DCP changes two things at once: the actor input (direction and radius) and the collection target (scored waypoints). Yet the nine-task benchmark compares only CRL, SSGC, and full DCP. The controlled four-way ablation (Table 3) suggests the waypoint component does not help and can hurt. Direct direction conditioning is best in both layouts, and adding scoring lowers open-room success in every seed. Without a direct-goal DCP variant on the benchmark, one cannot tell whether the large Pusher improvements come from the input change, the waypoint-based collection, or their interaction. The DCP actor also has 11–21% more parameters than CRL, all in the first layer, and the authors acknowledge that a parameter-matched raw-goal baseline is missing. The headline claim is a performance gain over CRL, and the paper frames DCP around waypoint-guided collection, so both controls seem necessary.
Questions for authors

Questions

  1. Can the gains be attributed to the input change? Could the authors report direct-goal DCP (direction and radius input, final-goal collection, no pool or scorer) and a parameter-matched raw-goal CRL actor on the nine benchmark tasks? At minimum, the three Pusher tasks, where the gains are largest. Table 3 suggests direct conditioning may be sufficient. If it matches or beats full DCP on the benchmark, the contribution would be better framed as an actor-input change than as a waypoint method. Relatedly, why does scoring hurt in the open room in every seed? One possibility is that waypoint targets induce a state distribution different from what hindsight relabeling trains on. Another is that switching targets every 25 steps produces less coherent trajectories. Comparing trajectory statistics (path length, revisits, target-switch effects) with and without scoring would clarify whether the waypoint mechanism helps in any setting beyond the generator mazes.

  2. Is the CRL baseline on the Pusher tasks well tuned? The largest gains occur on the Pusher tasks, where CRL performs poorly. CRL reaches only 4.4% final success on PusherEasy, 2.0% on PusherHard at 50M steps, and 12.5 ± 24.0% on PusherHard Far. On PusherHard, CRL reaches 28.8% only at 100M steps, with a confidence interval of ±32.4. This suggests very large variance across seeds, possibly with some seeds failing to learn at all. Could the authors clarify whether the CRL baseline was tuned for these tasks, and compare against the Pusher results reported in the original JaxGCRL paper? If CRL is under-tuned here, the +19 to +51 pp improvements may be overstated. If the baseline is sound, could the authors explain why the direction input helps so much specifically on Pusher? For example, could the raw goal coordinates have a scale or representation issue that makes them hard for the CRL actor to use?

5 comments
AuthorsOpenReview ↗

Response to Reviewer CPUb (1/3)

We thank the reviewer for appreciating the simplicity and low cost of our method and the transparency of our analysis. We address the critical weaknesses mentioned and questions asked by (1) running five seeds of direct direction conditioning on all nine benchmark tasks, (2) training parameter-matched CRL on the PusherEasy task to isolate whether more parameters help DCP unfairly, and (3) clarifying CRL's settings and its per-seed behavior on the Pusher tasks. These results show that the learned direction-and-distance input is the main source of the largest gains, and that waypoint collection is still needed on tasks where the final goal is rarely reached early in training. We also show that more parameters do not help CRL close the gap. We surmise that full DCP is a trade-off, giving up some peak success but, unlike Direct-DCP, it never fails outright on a task, which makes it the safer choice. Our conclusion that DCP improves over CRL still holds.

W1, Q1: Can the gains be attributed to the input change?

We now report Direct-DCP on all nine tasks with five seeds. Direct-DCP uses the DCP actor input but collects experience toward the final goal, without the pool or scorer. Together with CRL and SSGC, the benchmark now covers all four combinations of actor input (raw vs. learned direction) and "collection target" (final goal vs. waypoint).

Table: Final evaluation, five seeds per method (paper Table 2). Success rate (%) / time near goal (steps out of 1,000).

TaskCRLSSGCDCPDirect-DCP
PusherEasy4.4 / 2634.1 / 18155.4 / 29874.2 / 296
PusherHard28.8 / 12627.8 / 17147.7 / 28169.0 / 247
PusherHard Far12.5 / 489.0 / 4947.4 / 26154.1 / 176
AntMaze Big18.3 / 8923.3 / 10029.9 / 11528.4 / 117
AntMaze Hardest12.2 / 4814.2 / 6616.1 / 6318.4 / 59
Ant U-Maze40.8 / 26149.8 / 25355.0 / 23846.2 / 270
Humanoid U-Maze0.0 / 0.00.0 / 0.08.4 / 3.60.0 / 0.0
AntPush45.6 / 14351.4 / 19942.7 / 18219.5 / 65
AntSoccer22.6 / 7614.1 / 4812.9 / 4221.4 / 72

On all three Pusher tasks, both direction-conditioned variants outperform both raw-input variants (CRL and SSGC) in both success and time near goal, regardless of the collection target. We conclude that the conditioning input is the main source of the largest gains.

However, some of these experiments also show that waypoint collection is essential on certain tasks. On AntPush, three of five Direct-DCP seeds never reach the goal at any evaluation checkpoint, while all five DCP seeds learn. On Humanoid U-Maze, Direct-DCP ends at 0% in every seed (as does CRL), while DCP reaches 8.4%. Both are tasks where the final goal is rarely reached early in training, and we observe that waypoints provide reachable targets in that sparse phase of training.

Table: First evaluation checkpoint (M steps) at which each seed reaches 1% success (paper Appendix H). – marks a seed that never does.

TaskCRLDCPDirect-DCP
AntPush29, 35, 41, 40, 4045, 34, 21, 18, 4827, 38, –, –, –
Humanoid U-Maze19, 33, 6, 10, 710, 13, 3, 4, 12–, –, –, 47, –
PusherEasy3, 2, 3, 2, 22, 2, 3, 2, 23, 2, 3, 2, 2

The two metrics we report in our experiments measure different things: "success" counts the episodes in which the puck reaches the goal at least once, while "time near goal" counts the number of steps it stays there. On PusherHard and PusherHard Far, Direct-DCP reaches the goal in more episodes, but once the puck reaches the goal, DCP keeps it there for most of the remaining episode, while Direct-DCP loses it again.

We explain why this happens with the following intuitive hypothesis: Since both variants train the actor on the same relabeled future goals from their own trajectories. DCP's collection target is a previously visited puck position, held fixed for 25 steps. Once the agent pushes the puck to that position, it keeps acting toward the same target for the rest of the window. When these steps are relabeled, they give the actor examples of arriving at a goal and then holding the puck in place, with the distance input close to zero. This ensures that DCP maintains the puck at that position. However, as Direct-DCP's target is always the final goal, the actor keeps pushing toward a single target and collects far fewer such arrive-and-hold examples.

AuthorsOpenReview ↗

Response to Reviewer CPUb (2/3)

Table: Percentage of post-reach steps spent in the goal region, final checkpoint (paper Appendix H). Mean [range across seeds]. The ranges do not overlap on PusherHard and PusherHard Far.

TaskDCPDirect-DCP
PusherEasy63 [48, 77]44 [29, 55]
PusherHard66 [52, 73]39 [31, 45]
PusherHard Far61 [52, 72]36 [28, 41]

So which variant should one use? Full DCP is a trade-off. Direct-DCP has higher success on five tasks, but it fails outright on two, and which tasks these are is not obvious before training. DCP gives up some peak success on the Pusher tasks and AntSoccer, but it never fails outright on a task: every DCP seed on every task reaches at least 1% success during training.

Table: Reliability across the nine tasks, five seeds each (paper Section 4.2). Relative values are a method's mean divided by the better of DCP and Direct-DCP on that task.

CRLDCPDirect-DCP
Seeds reaching 1% success (of 45)444538
Tasks beating CRL, success / time near goal–7 / 76 / 6
Worst task, relative success–0.60 (AntSoccer)0.00 (Humanoid U-Maze)
Worst task, relative time near goal–0.59 (AntSoccer)0.00 (Humanoid U-Maze)

When one does not want to gamble on whether the final goal will be reached early in training, DCP is the better bet.

We also widen only the first hidden layer of the raw-goal CRL actor (256 → 313 units), matching DCP's actor parameter count, and run it on PusherEasy with all other settings unchanged. So we can safely conclude that the added parameters do not explain DCP's gain.

Table: Parameter-matched raw-goal actor on PusherEasy (paper Section 4.2, Appendix H).

MethodActor parametersSeedsSuccess (%)Time near goal
CRL75,53454.426
CRL, first layer 31391,49454.731
DCP91,406555.4298

Q1 (contd.): Why does scoring hurt in the open room?

In the open room, every goal can be reached by moving straight toward it, so waypoints only redirect collection away from the evaluated goals without making them any easier to reach. Our main benchmark also shows the same pattern on AntSoccer, where collecting targets toward the final goal raises success from 12.9% to 21.4%.

Q2: Is the CRL baseline on the Pusher tasks well-tuned?

We note that all methods use the same JaxGCRL [a] CRL implementation and hyperparameters (backward InfoNCE, L2 energy, and identical replay, batch, and budget settings; Appendix A). We also note the following points:

  1. The same CRL configuration learns well on other tasks (45.6% on AntPush, 40.8% on Ant U-Maze).
  2. CRL can learn PusherHard in some seeds: its final success per seed is 0.4, 48.8, 1.2, 53.9, and 39.5%. We observe that it has high variance which produces the wide interval the reviewer noted. However, DCP and Direct-DCP succeed in every seed on all three Pusher tasks, with the lowest seed at 40.2%. We believe this is just due to the robustness that arises out of the coupling between the policy and the representation space, the conditioning offers.
  3. As we stated earlier, the parameter-matched actor stays at 4.7% on PusherEasy, so DCP doesnt work better than CRL just due to the increased number of parameters.

We believe this level of control is fair, especially because it is not the case that a baseline completely fails on every seed. It is also to be noted that we never tuned DCP as well.

AuthorsOpenReview ↗

Response to Reviewer CPUb (3/3)

Why does the direction help so much on Pusher? In the Pusher tasks we have an arm that has to push a puck toward a goal. A raw goal only gives the puck's target position, so a CRL actor has to figure out for itself how the arm, puck, and goal relate before it can push it the correct way. The critic's goal encoder already learns how the current state relates to the goal, and the direction toward the goal in that space tells the actor which way the puck must go. It is because of this, that we believe that DCP largely helps over raw-goal conditioning. Our controlled puck-pushing study (Section 4.4) confirms this. DCP makes contact from the correct side more often than CRL and succeeds more often. Reversing only the direction input drops DCP's success to 2.9%, even though contact remains frequent.

We thank you again for your review, which has helped make our work much stronger. Please let us know if any more questions arise about this work, and we will do our best to address them!

Changes to the paper: We have revised the paper accordingly, with all changes marked in blue. Section 4.1, Table 2, and Figure 3 add Direct-DCP on all nine tasks. Section 4.2 adds what this comparison shows about each component, the goal-holding result, and the parameter-matched control. The abstract, introduction, and conclusion now summarize these findings. The new Appendix H gives the first-reach and goal-holding statistics, statistical tests, and the per-seed parameter-matched and CRL Pusher results. Despite these changes, we believe that our initial claims and methodology remain the same, with some added insights on when waypoint pooling works for DCP and when it might not.

Reference(s):

[a] Bortkiewicz, M., Pałucki, W., Myers, V., Dziarmaga, T., Arczewski, T., Kuciński, Ł., and Eysenbach, B. Accelerating Goal-Conditioned Reinforcement Learning Algorithms and Research. International Conference on Learning Representations (ICLR), 2025.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions and Responses to Reviewers

We sincerely thank the reviewers and the area chair for their valuable time and effort spent in looking into our work. Below, we summarize the improvements we made to the paper, the additional evidence supporting our work, and our responses to the reviewers' concerns. This is for the ease of the program committee, including the area chair and the reviewers. We marked all changes in blue in the paper, and we name the relevant reviewers in parentheses wherever needed.

(1) What the reviewers valued. The reviewers found DCP simple and low-cost, while giving useful gains over CRL (CPUb, vtFb, bKfE). Reviewers also appreciated our thorough analyses through controlled studies and ablations of the method's components (CPUb, bKfE).

(2) Isolating where the gains come from and where the entire method helps (CPUb, vtFb, bKfE). Their main shared concern was that Direct-DCP and a parameter-matched raw-goal actor were missing from the benchmark, and we have added both.

  • We ran Direct-DCP (direction input, collecting toward the final goal without waypoints) on all nine tasks with five seeds. On all three Pusher tasks, both direction-conditioned variants outperform CRL and SSGC in both success and time near goal. This shows that the learned direction-and-distance input is the main source of the largest gains.
  • While the above is true, waypoints are still needed where the final goal is rarely reached early in training. Three of five Direct-DCP seeds never reach the goal on AntPush, and Direct-DCP ends at 0% in every seed on Humanoid U-Maze. Every DCP seed, on the other hand, learns on every task (45/45 seeds, against 38/45 for Direct-DCP).
  • We also show that using the waypoint selection strategy keeps the agent at the goal longer. On PusherHard and PusherHard Far, DCP stays at the goal for 66% and 61% of the steps after first reaching it, against 39% and 36% for Direct-DCP. Overall, DCP spends more time near the goal than Direct-DCP on 6 of 9 tasks.
  • Finally, to address concerns about DCP having slightly more parameters in its first layer, we trained a raw-goal CRL actor matched to DCP's parameter count. It reaches only 4.7% success on PusherEasy, far below DCP's 55.4%. The extra parameters therefore do not explain DCP's gain.

(3) The configuration of the CRL baseline (CPUb). We note that all methods use the same JaxGCRL CRL implementation and hyperparameters. CRL does learn PusherHard in some seeds (up to 53.9%) but not in others, while DCP and Direct-DCP succeed in every seed on all three Pusher tasks.

(4) Novelty of DCP in the online GCRL setting (vtFb, bKfE). Unlike HILP and METRA, DCP adds no new representation objective. The actor reads the critic's own goal encoder while it is still being trained, on data that the same actor collects. This coupling does not arise offline, and our results show when waypoints are needed to keep it stable.

To the best of our knowledge, DCP is one of the first methods to condition the actor on directions in the critic's own representation space during online GCRL, without any extra objective or planning at deployment. In our opinion, this makes DCP a simple recipe for practitioners: the direction input gives large gains over raw goals, and the waypoints make it reliable during online training.

Updates on the discussion.

  • Reviewer CPUb noted that our responses addressed their concerns and maintains their score in support of acceptance.
  • Reviewer vtFb noted that the Direct-DCP comparison addresses their main experimental concern and has raised their score to a 3.
  • Reviewer bKfE noted that their concern about the parameter-matched comparison is resolved. On the remaining concern about the waypoint selection strategy, we note that DCP with its waypoint selection strategy remains more reliable across tasks than Direct-DCP is and we would like to stress again that this work provides an intuitive empirically verified recipe for practitioners.

We thank everyone again for helping us make our work stronger!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.