- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
DCP changes how a CRL actor receives its target. It uses the direction and distance between goal embeddings, selects visited waypoints during data collection, and acts directly toward the final goal at evaluation. Experiments on nine simulated tasks show higher final mean success rates than CRL on seven tasks and more time near the goal on seven tasks.
Critical strengths
+The change is simple and gives useful improvements on several tasks while keeping the CRL objective and network depth. +The analysis goes beyond learning curves. It checks the learned geometry, tests whether the actor uses the direction input, and reports cases where waypoint selection or candidate generation does not help.
Critical weaknesses
-The main method is a small extension of existing direction-conditioned policies. Latent directions have already been used in methods such as HILP and METRA. Bringing this idea into online CRL is useful, but the main change is the actor input and waypoint selection, while the baseline learning objectives remain largely unchanged. -The experiments do not fully explain where the gains come from. In Table 3, direct direction conditioning without waypoint selection has the highest mean performance in both controlled environments, but this version is missing from the main benchmark comparison. The actor also has 11–21% more parameters, without a parameter-matched control. These gaps make it difficult to separate the value of the learned representation from the other changes. -The broader lesson from the empirical study remains unclear. The direction interventions show that the actor uses its input, but do not establish why the learned geometry is a better choice than other goal representations. The study uses one CRL configuration, and the main mechanism analyses focus on controlled maze and pushing tasks. I would expect clearer evidence about when this conditioning should help before drawing a broader conclusion for CRL design.
Questions for authors
- What is the main new insight beyond adapting the direction-conditioning idea in HILP and METRA to online CRL? Which challenges specific to online learning does DCP address?
- Can you compare DCP with direct-goal direction conditioning on representative benchmark tasks? This simpler version has the highest mean performance in both Table 3 settings, so it remains unclear when waypoint selection helps.
- Can you include a parameter-matched raw-goal actor? This would help separate the benefit of the learned goal representation from the 11–21% increase in actor parameters.
- What do the current experiments tell us about when DCP helps or hurts? Which findings extend beyond the controlled maze and pushing studies, and what evidence supports them?
5 comments
Response to Reviewer bKfE (2/3)
So which variant should one use? Full DCP is a trade-off. Direct-DCP has higher success on five tasks, but it fails outright on two, and which tasks these are is not obvious before training. DCP gives up some peak success, but it never fails outright on a task: every DCP seed on every task reaches at least 1% success during training.
Table: Reliability across the nine tasks, five seeds each (paper Section 4.2). Relative values are a method's mean divided by the better of DCP and Direct-DCP on that task.
| CRL | DCP | Direct-DCP | |
|---|---|---|---|
| Seeds reaching 1% success (of 45) | 44 | 45 | 38 |
| Tasks beating CRL, success / time near goal | – | 7 / 7 | 6 / 6 |
| Worst task, relative success | – | 0.60 (AntSoccer) | 0.00 (Humanoid U-Maze) |
| Worst task, relative time near goal | – | 0.59 (AntSoccer) | 0.00 (Humanoid U-Maze) |
When one does not want to gamble on whether the final goal will be reached early in training, DCP is the better bet.
Why the critic's own representation, rather than another goal representation? We argue for this particular goal representation, because it is the space in which the actor is already being trained on. In CRL, the actor is optimized to choose actions whose embedding lies close to (Eq. 1), so the critic scores every action by where it moves the agent in -space. DCP's input, , is a displacement in this same space: the actor is told where the target lies in exactly the coordinates in which its own objective is measured. Other learned goal representations, such as those of HILP [a] and METRA [b], live in a different space from the critic that trains the actor and need their own objective. Online, that objective would also have to be trained on the data the actor collects, which adds a second moving representation to the same coupling discussed in our response to W1. DCP instead reuses the representation CRL already learns, at no extra cost.
Finally, we stress on the view that DCP is not a competitor to other goal representations but a way of using one. Since it changes only the actor input, any learned representation can take the place of in the direction and distance, and the same input can be directly combined with deeper or action-chunked CRL critics.
W1, Q1: The new insight beyond HILP and METRA, and the challenges specific to online learning
We invite the reviewer to view this work as an empirical study of whether the direction in CRL's own representation space can be used to condition the actor policy in online GCRL. HILP [a] and METRA [b] learn their representations with dedicated objectives (offline, or for unsupervised skill discovery) and then condition a policy on directions in that space. In our case, DCP adds no representation objective. And the actor reads the goal encoder of the critic while it is still being trained, on data that the same actor collects. This creates the following stability problem that does not arise offline: The direction toward a goal the agent has never reached is computed in a poorly trained region of the representation, while the data that would train that region depends on the policy following that direction. Our work addresses this challenge specifically and provides insights on the same. Since the other offline methods learn from a fixed dataset or with a separate objective, they address a different setting from ours.
We have thus shown one case of a simple use of direction in a learned representation space as a conditioning variable in online GCRL. We hope that this work encourages practitioners in the field to use learned representations as a conditioning variable for the policy, even while these representations are still being jointly trained.
We thank you again for your review, which has helped make our work much stronger. Please let us know if any more questions arise about this work, and we will do our best to address them!
Response to Reviewer bKfE (3/3)
Changes to the paper: We have revised the paper accordingly, with all changes marked in blue. Section 4.1, Table 2, and Figure 3 add Direct-DCP on all nine tasks. Section 4.2 adds what this comparison shows about each component, the goal-holding result, and the parameter-matched control. The abstract, introduction, and conclusion now summarize these findings. The introduction and Section 3.2 now state that DCP uses the critic's own goal encoder, the space in which the critic scores the actor's actions, and the conclusion lists other learned goal representations as future work. The new Appendix H gives the first-reach and goal-holding statistics and statistical tests. Despite these changes, we believe that our initial claims and methodology remain the same, with some added insights on when waypoint pooling works for DCP and when it might not.
Reference(s):
[a] Park, S., Kreiman, T., and Levine, S. Foundation Policies with Hilbert Representations. International Conference on Machine Learning (ICML), 2024.
[b] Park, S., Rybkin, O., and Levine, S. METRA: Scalable Unsupervised RL with Metric-Aware Abstraction. International Conference on Learning Representations (ICLR), 2024.
Response to Reviewer bKfE (1/3)
We thank the reviewer for appreciating the simplicity of our change, its improvements on several tasks, and the depth of our analysis beyond learning curves. We address the critical weaknesses mentioned and questions asked by (1) running direct-goal direction conditioning (Direct-DCP) on all nine benchmark tasks with five seeds, (2) training a parameter-matched raw-goal CRL actor on the PusherEasy task to isolate whether more parameters help DCP unfairly, (3) using these results to show when DCP helps or hurts at benchmark scale, and (4) clarifying the new insight and the online-specific challenge relative to HILP and METRA. The new results show that the learned direction-and-distance input is the main source of the largest gains over raw goals, and that waypoint collection is needed on tasks where the final goal is rarely reached early in training. Full DCP is a trade-off: it gives up some peak success but, unlike Direct-DCP, never fails outright on a task, making it the safer choice. Our conclusion that DCP improves over CRL still holds.
W2, Q2, Q3: Direct-goal direction conditioning on the benchmark and a parameter-matched raw-goal actor
We now report Direct-DCP on all nine tasks with five seeds. Direct-DCP uses the DCP actor input but collects experience toward the final goal, without the pool or scorer. Together with CRL (raw final goal) and SSGC (raw waypoint coordinates), the benchmark now covers all four combinations of actor input (raw vs. learned direction) and "collection target" (final goal vs. waypoint).
Table: Final evaluation, five seeds per method (paper Table 2). Success rate (%) / time near goal (steps out of 1,000).
For the parameter-matched control, we widen only the first hidden layer of the raw-goal CRL actor (256 → 313 units) and keep all other settings unchanged.
Table: Parameter-matched raw-goal actor on PusherEasy (paper Section 4.2, Appendix H).
We thus safely conclude that the 11–21% increase in actor parameters does not explain DCP's gain.
W3, Q4: When does DCP help or hurt, and what evidence supports this beyond the controlled studies?
With the added experiments from Direct-DCP, we can better answer this question with the following points: