- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes CALP for online goal-conditioned reinforcement learning. It first creates nearby waypoint candidates, checks how well the controller can execute them, and then selects one using a learned temporal-distance model. A shared controller learns to follow these waypoints using local progress rewards. The critic does not carry returns across waypoint boundaries. Results on eight JaxGCRL tasks show good performance and shorter training time with shallow networks.
Critical strengths
+The paper studies a useful question: whether better planning structure can reduce the need for deep networks. +The experiments cover eight tasks with five seeds and include both depth-scaled CRL and online subgoal methods. The results show a promising performance–training-time trade-off.
Critical weaknesses
-The ablations establish that the components help, but give less evidence for the specific predictor targets. Section 2 distinguishes graded competence supervision from binary reachability, while Table 1 compares it with removing competence filtering. A binary-target version would help assess this particular design choice. Similarly, the quantile–MSE comparison changes both the estimation target and the loss form. Median regression with the same pinball loss would help isolate the benefit of choosing a lower quantile. -The computational comparison needs more explicit configuration details. Appendix B lists several settings and refers to existing defaults, but does not explicitly give the hidden widths and update frequencies for the actor, critics, competence model, and distance model. Since shallow networks and training efficiency are central to the paper, reporting these configurations and parameter counts would make the results easier to reproduce and assess.
Questions for authors
- Equation (9) seems to calculate the competence label over the executed command, with a timeout of 120 steps. However, Appendices A.2 and B call the 20-step H_TD the competence-label horizon. Please clarify which interval is actually used.
- What happens when no candidate passes the competence threshold? Equation (13) does not define this case, and it seems likely to occur during early learning.
- C predicts fractional displacement, while Equation (13) evaluates remaining distance from the full waypoint. How often are selected waypoints actually reached? Completion and timeout rates, together with progress from partially completed commands, would help explain how these two predictors work together under replanning. Partial execution is not necessarily harmful, but its role is currently unclear.
We thank the reviewer for the constructive suggestions.
1. Additional ablations.
We conducted the requested ablations on Ant Push (50M steps, three seeds):
Time at Goal (↑), mean ± standard deviation.
Graded supervision improves performance by 47% over binary supervision. Unlike binary labels, which indicate waypoint completion, graded labels preserve information about partially realized displacements, which can be useful for repeated local replanning.
Lower-quantile regression improves performance by 13% over median regression. Both variants use the same pinball loss, isolating the effect of optimistic temporal-distance estimation.
2. Implementation clarifications.
3. Architecture, reproducibility, and additional evaluation.
We have expanded the architecture documentation in our anonymous repository, including each module's depth, width, activation functions, and parameter counts.
All four CALP modules (actor, twin critics, competence, and temporal distance) use two hidden layers of width 256, with 290,574 total trainable parameters. In comparison, 64-layer Scaling CRL contains 12,779,658 parameters, approximately 44 times as many. The released source code provides the corresponding implementations and hyperparameters.
We also extend our evaluation to Arm Binpick Hard, a challenging manipulation task involving grasping and object transport. CALP achieves 145 ± 30, compared with 95 ± 24 for 64-layer Scaling CRL under the same 50M-step training budget, providing additional evidence of the effectiveness of our shallow planning framework.