acceptodds
Under review as a conference paper at ICLR 2027

Shallow Networks, Distant Goals: Competence-Aware Local Planning for Online GCRL

Abstract

Learning to reach distant goals is a central challenge in online goal-conditioned reinforcement learning (GCRL). Recent advances in contrastive reinforcement learning (CRL) demonstrate the benefits of scaling network depth, alongside increased computational demands. We investigate how far shallow networks can go with a suitable planning structure. To this end, we introduce Competence-Aware Local Planning (CALP), a simple approach that constructs long-horizon behavior through repeated local decisions. CALP searches locally in goal space, using learned execution competence to identify feasible subgoals and a lower-quantile temporal-distance estimate to guide selection toward the final goal. A shared controller executes these subgoals, with all components learned jointly from online interaction. Across eight navigation and manipulation tasks in JaxGCRL, CALP improves goal-reaching performance over other online GCRL baselines using two-hidden-layer networks and a single local selection step per planning decision, with the largest gains on obstacle-interaction tasks. These results demonstrate the effectiveness of local goal-space planning for long-horizon goal reaching with shallow networks.

Primary area:
reinforcement learning
Keywords:
goal-conditioned reinforcement learning, subgoal planning, temporal distance learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Z6DzOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CALP for online goal-conditioned reinforcement learning. It first creates nearby waypoint candidates, checks how well the controller can execute them, and then selects one using a learned temporal-distance model. A shared controller learns to follow these waypoints using local progress rewards. The critic does not carry returns across waypoint boundaries. Results on eight JaxGCRL tasks show good performance and shorter training time with shallow networks.

Critical strengths

+The paper studies a useful question: whether better planning structure can reduce the need for deep networks. +The experiments cover eight tasks with five seeds and include both depth-scaled CRL and online subgoal methods. The results show a promising performance–training-time trade-off.

Critical weaknesses

-The ablations establish that the components help, but give less evidence for the specific predictor targets. Section 2 distinguishes graded competence supervision from binary reachability, while Table 1 compares it with removing competence filtering. A binary-target version would help assess this particular design choice. Similarly, the quantile–MSE comparison changes both the estimation target and the loss form. Median regression with the same pinball loss would help isolate the benefit of choosing a lower quantile. -The computational comparison needs more explicit configuration details. Appendix B lists several settings and refers to existing defaults, but does not explicitly give the hidden widths and update frequencies for the actor, critics, competence model, and distance model. Since shallow networks and training efficiency are central to the paper, reporting these configurations and parameter counts would make the results easier to reproduce and assess.

Questions for authors
  1. Equation (9) seems to calculate the competence label over the executed command, with a timeout of 120 steps. However, Appendices A.2 and B call the 20-step H_TD the competence-label horizon. Please clarify which interval is actually used.
  2. What happens when no candidate passes the competence threshold? Equation (13) does not define this case, and it seems likely to occur during early learning.
  3. C predicts fractional displacement, while Equation (13) evaluates remaining distance from the full waypoint. How often are selected waypoints actually reached? Completion and timeout rates, together with progress from partially completed commands, would help explain how these two predictors work together under replanning. Partial execution is not necessarily harmful, but its role is currently unclear.
1 comment

We thank the reviewer for the constructive suggestions.

1. Additional ablations.

We conducted the requested ablations on Ant Push (50M steps, three seeds):

Graded competenceBinary competenceLower quantile (τ=0.2)Median (τ=0.5)
587 ± 31398 ± 39587 ± 31519 ± 35

Time at Goal (↑), mean ± standard deviation.

Graded supervision improves performance by 47% over binary supervision. Unlike binary labels, which indicate waypoint completion, graded labels preserve information about partially realized displacements, which can be useful for repeated local replanning.

Lower-quantile regression improves performance by 13% over median regression. Both variants use the same pinball loss, isolating the effect of optimistic temporal-distance estimation.

2. Implementation clarifications.

  • Competence-label interval: Equation (9) uses the actual executed command duration, capped at H = 120 steps. Commands terminate upon waypoint completion, timeout, stalled progress, or episode termination.
  • TD horizon: H_TD = 20 applies only to the truncated critic backup. We will correct its inconsistent description in Appendices A.2 and B.
  • Empty feasible set: When competence masking removes all candidates, the planner falls back to the active direction set.

3. Architecture, reproducibility, and additional evaluation.

We have expanded the architecture documentation in our anonymous repository, including each module's depth, width, activation functions, and parameter counts.

All four CALP modules (actor, twin critics, competence, and temporal distance) use two hidden layers of width 256, with 290,574 total trainable parameters. In comparison, 64-layer Scaling CRL contains 12,779,658 parameters, approximately 44 times as many. The released source code provides the corresponding implementations and hyperparameters.

We also extend our evaluation to Arm Binpick Hard, a challenging manipulation task involving grasping and object transport. CALP achieves 145 ± 30, compared with 95 ± 24 for 64-layer Scaling CRL under the same 50M-step training budget, providing additional evidence of the effectiveness of our shallow planning framework.

Reviewer_FGswOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Competence-Aware Local Planning (CALP), which decomposes long-horizon problems into a sequence of local goal-reaching problems, unlike Scaling CRL, which increases network depth to achieve distant goals in online GCRL. CALP generates local candidate subgoals around the currently achieved goal and uses an execution competence model to retain only candidates that the actual controller can reach. It then selects the waypoint with the smallest estimated temporal distance to the final goal using a lower-quantile temporal distance model. The selected waypoint is executed by a single shared local TD3 controller. All components are jointly learned through online interaction. The authors evaluate CALP on various navigation, locomotion, and manipulation tasks in JaxGCRL. Through these experiments, they test whether using a local planning structure instead of increasing network depth is effective for long-horizon GCRL and whether it can improve the trade-off between performance and computational cost.

Critical strengths
  1. Intuitive Problem Formulation: The authors identify a key limitation of existing methods, in which a goal-conditioned value function must simultaneously handle local feasibility and global goal progress. CALP addresses this issue by separating these two potentially conflicting functions into complementary components. The competence model determines whether the current controller can actually reach a given waypoint, while the temporal-distance model determines which feasible waypoint best leads toward the final goal.
  2. High Compute Efficiency and Strong Empirical Performance: Unlike Scaling CRL, which uses deep networks, CALP achieves the best performance in 6 out of 8 domains while using only two-hidden-layer networks, and shows performance comparable to deep Scaling CRL variants in the remaining 2 domains. CALP also requires substantially lower wall-clock training costs, demonstrating strong results not only in final performance but also in terms of the performance–compute trade-off.
  3. Ablation Studies Demonstrating the Importance of Key Components: The experiments show that the impact of each component varies depending on the characteristics of the environment. In environments where local motion is largely unobstructed, removing the competence filter has little effect on performance. In contrast, in environments where obstacles can lead to infeasible subgoals, removing the filter significantly degrades performance. This supports the effectiveness of the competence model in filtering out candidates that the controller cannot reliably reach. Additionally, replacing the learned temporal distance with random or Euclidean selection leads to substantial performance degradation in obstacle-rich tasks, while lower-quantile regression consistently outperforms MSE. These ablation results demonstrate that the competence filter and learned temporal distance play important roles in CALP's performance, particularly in obstacle-rich tasks.
Critical weaknesses
  1. Limited Novelty: The approach of decomposing long-horizon goal reaching into a sequence of local subgoal-reaching problems has been explored in prior work. For example, [1] restricts subgoals to a region that the low-level policy can reach within k steps, within which the high-level policy maximizes task rewards. This shares a similar principle with CALP, which uses a learned competence model to filter local candidates and then selects the feasible waypoint with the smallest estimated temporal distance to the final goal. Similarly, [2] decomposes distant goal reaching into a sequence of easier subgoal-reaching problems, optimizes them using GCRL, and constructs a graph for planning. This shares similarities with CALP's approach of solving distant goal-reaching problems through subgoal decomposition. Therefore, CALP's main contribution may be better framed not as solving long-horizon tasks through repeated local decisions, but as demonstrating that well-designed, task-aligned local planning can reduce the need for explicit long-horizon prediction.
  2. CALP's Applicability Relies on Structured Low-Dimensional Environments: The current evaluation does not clearly separate the effect of network depth from that of CALP's local planning strategy. The evaluated tasks may be suitable for decomposition into relatively simple local decisions. In such tasks, CALP can avoid much of the long-horizon reasoning required by end-to-end CRL by repeatedly solving local planning problems. Thus, the strong performance may be attributed to a planning structure well suited to the target tasks, rather than to the effectiveness of shallow networks themselves. Consequently, I am concerned that the current experimental results may provide insufficient evidence to support the broader claim that shallow networks can generally match the performance of deep CRL.
  3. Insufficient Clarity on CALP's Overall Network Architecture and Definition of Depth: The authors present their main results by comparing CALP's two-hidden-layer networks with 16/64-layer Scaling CRL. However, CALP includes not only an actor and twin critics but also separate execution-competence and temporal-distance models. While the paper explicitly states that the actor and critics use two-hidden-layer networks, it does not provide sufficient details about the overall architecture, including the number of layers, width, and activation functions of the competence and temporal-distance models. Furthermore, the network-depth ablation states that the depth of the "networks used by CALP" was varied, but it is unclear which components were modified together. Therefore, it is difficult to determine exactly which architectural quantities are being compared between CALP with depth=2 and Scaling CRL with depth=16/64. Although the reproducibility statement indicates that model architectures are provided in Appendix B, the relevant details are difficult to identify in the appendix. Providing the complete architecture and parameter count for each module would help readers more accurately interpret the paper's central shallow-vs-deep comparison.

[1] Zhang, Tianren, et al. "Generating adjacency-constrained subgoals in hierarchical reinforcement learning." Advances in neural information processing systems 33 (2020): 21579-21590.

[2] Eysenbach, Ben, Russ R. Salakhutdinov, and Sergey Levine. "Search on the replay buffer: Bridging planning and reinforcement learning." Advances in neural information processing systems 32 (2019).

Questions for authors
  1. Is Graded Competence Estimation Really Necessary?: CALP presents learning execution competence from a graded hindsight target, rather than using binary reachability targets, as one of its key distinctions from prior work. However, during inference, a threshold is applied to binarize candidates as feasible or infeasible before filtering. Therefore, it is unclear what advantages graded competence supervision provides over simpler binary reachability supervision. Since the current ablation only examines the removal of the competence estimator entirely, it is difficult to isolate the contribution of the proposed graded competence target. A direct comparison with a competence estimator trained using binary reachability targets would help clarify the necessity and contribution of the proposed graded formulation.
  2. What Are the Network Architectures and Depth Configurations of Each CALP Module?: CALP includes competence and temporal-distance models in addition to the actor and critics, but their architectures are not sufficiently described. It is also unclear which modules have their depths modified in the network-depth ablation. Clarifying the layer configuration and architecture of each module, along with their parameter counts, would help readers more accurately interpret the comparison with the 16/64-layer Scaling CRL results.
4 comments

We thank the reviewer for the detailed feedback and for raising important questions about the contribution and architectural comparison.

1. Central contribution and novelty.

We agree that subgoal decomposition has been explored in prior work. CALP builds on this principle through a lightweight, fully online framework combining graded execution competence, lower-quantile temporal guidance, and single-step local waypoint selection, differing from HRAC's hierarchical policy and SoRB's replay-graph search.

We appreciate the reviewer's observation that CALP's task-aligned planning structure plays an important role in its performance. Indeed, this is central to our research motivation: we investigate whether exploiting goal-space structure through explicit local planning can reduce reliance on deep networks for long-horizon goal reaching.

Our results support this hypothesis. CALP with two-hidden-layer networks outperforms deep CRL on six of eight tasks and matches it on the remaining two, while substantially reducing training time. The depth ablation (Figure 11) further shows relatively modest improvements from increasing CALP's network depth.

2. Performance on challenging tasks.

Ant Push is particularly challenging because successful navigation requires physical interactions that can alter the environment's reachability, as highlighted in the original JaxGCRL paper. CALP substantially outperforms the evaluated baselines. Our additional ablations further demonstrate the value of its design: graded competence improves performance from 398 ± 39 to 587 ± 31, while lower-quantile distance improves it from 519 ± 35 to 587 ± 31.

We additionally evaluate Arm Binpick Hard, involving grasping, lifting, and object transport. CALP achieves 145 ± 30, compared with 95 ± 24 for 64-layer Scaling CRL under the same 50M-step training budget.

These results further support the effectiveness of exploiting goal-space structure to achieve strong performance on challenging tasks with shallow networks.

3. Architecture clarification.

We have documented the architectures of all four CALP modules, including their widths, depths, activation functions, and parameter counts, in our anonymous repository.

CALP uses 290,574 trainable parameters, compared with 12,779,658 for 64-layer Scaling CRL, approximately 44× fewer parameters.

We hope these clarifications and additional experiments help better convey the contribution and practical value of CALP.

I appreciate the authors' clarifications, but my main concern remains unresolved. The current results may mainly show that a well-designed explicit local planner can solve these benchmarks with shallow networks. This does not necessarily support the broader claim that shallow models can replace deep long-horizon reasoning in general. To strengthen this claim, the method should be evaluated on tasks where local decomposition itself is difficult, or compared against hand-designed/classical local planners to clarify whether the gain comes from learning or from the explicit planning structure.

I also remain unclear about the necessity of graded competence supervision. CALP trains the competence estimator using continuous graded targets, but at inference time its prediction is thresholded and used as a binary feasibility filter. Does continuous supervision provide a meaningful advantage over directly learning binary reachability? In this context, could the authors also clarify what exactly is being compared in the reported additional ablation result (398 ±\pm 39 to 587 ±\pm 31)?

We thank the reviewer for the follow-up and appreciate the opportunity to clarify our contributions.

1. Local planning and network depth.

We agree that CALP's strong performance benefits from its explicit planning structure. Indeed, our central research question is whether exploiting goal-space structure through local planning can reduce the network depth required for long-horizon goal reaching, rather than whether shallow networks can universally replace deep reasoning.

Importantly, CALP goes beyond hand-designed local planning. Our ablations retain the same candidate generation and competence filtering, but replace learned temporal-distance guidance with Euclidean or uniform selection. On Ant Push, performance drops from 590.5 to 224.4 and 189.0, respectively (Table 1). This demonstrates that the learned planning components contribute substantially beyond the explicit planning structure alone.

2. Graded versus binary competence.

Binary competence predicts whether the commanded waypoint was reached, whereas graded competence predicts the fraction of commanded displacement actually realized (Eq. 9).

Although both are thresholded at inference, graded supervision preserves information about partially successful commands during learning. On Ant Push, this improves Time at Goal from 398 ± 39 to 587 ± 31 over three seeds, under the same 50M-step training budget.

Detailed results and configurations are provided in our additional experiments.

We hope these clarifications help distinguish the contribution of learned local planning from the broader question of network depth.

Thank you for the additional explanation. My remaining concern is not simply task difficulty, but the extent to which CALP's explicit planning structure is tailored to the structure of the current benchmarks. I understand the authors' point that exploiting goal-space structure can substantially reduce the required network depth, and I agree that this is an interesting advantage of CALP. However, most of the evaluated tasks still appear to admit relatively clear local coordinates, geometric detours, or locally defined progress. Although Ant Push and Arm Binpick Hard are challenging tasks, they do not particularly test cases where locally reasonable decisions can lead to poor long-term outcomes, or where ordered or irreversible decisions are critical. Therefore, I remain uncertain about how well CALP would extend to more abstract long-horizon problems where an effective local decomposition is itself difficult to define. This is the main reason my original concern about generalization remains.

For graded competence, the additional result shows a clear empirical gain over binary supervision. However, I still find the explanation insufficient to establish why the graded target is beneficial, especially since both predictions are ultimately thresholded at inference. In my view, the current result mainly shows that the graded variant performs better, but does not explain well why this gain occurs. Therefore, I still consider the motivation and empirical justification for graded supervision to be somewhat limited. Given that these concerns remain unresolved, I will maintain my original score.

Reviewer_FZwtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In contrastive RL, a single function has to learn the degree of reachability in the state-action-goal interaction space, which is usually highly complex so that an expressive network with large depth is preferred, making the training slow. In this paper, the authors proposed that this hard task can be decomposed into two parts: 1) an estimator to predict the distance between any subgoal and the specified final goal, and 2) a separate estimator to decide whether a subgoal is plausible. With these two components together, the authors found that across several tasks, their model learns faster and better than the classical and scaling CRL method. However, the success of this method relies on the nature of the task representation space. Strong priors like the low-dimensional, hand-specified, locally Euclidean goal representation make the candidate generation useful, while in CRL family no such prior is exploited.

Critical strengths
  1. The realization that in some GCRL tasks, knowledge can be reflected in network structure design, e.g., using two simpler networks to learn the ultimate reachability problem separately.
  2. Clear ablation experiments that identify the relationship between components and environments. For example, the competence filter does not hurt open tasks much.
Critical weaknesses
  1. The proposed method relies heavily on the flat nature of the task.
  2. The method also relies on a good, abstract, and meaningful representation of the task (this is also related to point 1). For example, in some of the tasks used, the goal is the coordinates, which makes the distance function (without considering reachability) nearly trivial to solve, and the candidate generation algorithm makes sense naturally. On the other hand, if a single estimator has to learn this from scratch, it may be slowed down by poor exploration efficiency.
Questions for authors
  1. From fig6 we can see that the ant u maze and the ant hardest maze are two tasks in which the distance estimate is not that "flat", and from fig1 we can tell in these two tasks the performance differences of CALP and deep CRL are small. I wonder if the authors has systematically evaluates the "sweet region" of the CALP method (for example, turn the maze into lattice based connectivity graph and then measure the laplacian or degree distribution). If the "width" is not too narrow, I'd consider this method as more promising than it appears currently.
  2. I wonder if the authors have checked tasks with raw sensory inputs like images, at least for the goal space? For example, use the exactly the same task but replace the goal with a 3rd person view of the robot at the goal position.
3 comments

We thank the reviewer for the thoughtful feedback.

1. Goal-space structure and task complexity.

We agree that CALP exploits structured goal representations. However, low-dimensional goals can still involve highly complex dynamics and reachability. Ant Push is a particularly important example. As discussed in the original JaxGCRL paper (Bortkiewicz et al., 2025, Secs. 4.2 and 5.2), the Ant must physically move a box to open a path toward its goal. Pushing it in the wrong direction can make the task unsolvable, requiring exploration and an understanding of how physical interactions change the environment's connectivity. The benchmark authors also identify Ant Push as a challenging task where existing methods achieve limited performance.

CALP performs particularly well in this setting. Removing competence filtering reduces performance from 590.5 to 365.9, while replacing learned temporal distance with Euclidean selection reduces it to 224.4 (Table 1). These results show that coordinate-based candidate generation alone is insufficient: learning local executability and temporal connectivity is crucial.

2. Performance across goal difficulties.

Following the reviewer's suggestion, we introduce a 13 × 13 Ant Maze with longer detours, preserving JaxGCRL's original dynamics and maze scaling. We measure difficulty as the shortest collision-free path length divided by the straight-line distance. Our additional experiments include detailed MuJoCo renderings and a color-coded goal layout.

Detour Ratio1–22–33–44+
CALP72%56%39%23%
Scaling CRL45%48%21%6%

Success rate (%), three seeds, with multiple goals and 20 evaluation episodes per difficulty range per seed.

CALP maintains higher success rates across all four detour ranges, including goals requiring substantial geometric detours.

We also add Arm Binpick Hard, which requires grasping, lifting, and transporting an object. CALP achieves 145 ± 30, compared with 95 ± 24 for 64-layer Scaling CRL under the same 50M-step training budget.

3. Image-based goals.

We appreciate the suggestion to consider image-based goals and agree that this is an interesting direction for extending CALP. Such an extension would involve learning a suitable visual representation and locally meaningful goal-space geometry, introducing additional encoder design and training complexity. We believe this deserves dedicated investigation. In the present work, we focus on understanding the benefits of local planning in structured goal spaces, and we will clarify this scope in the revision.

Thank you for the rebuttal and for the Euclidean-selection ablation is the most convincing part of the response and goes straight. to my second concern.

I keep my score because my first concern was about the breadth of CALP's regime, not about whether low-dimensional tasks can be dynamically hard, the 13×13 maze varies path length inside the same goal space rather than probing sensitivity to the representation prior. The image-goal scope argument is reasonable and should appear as a stated limitation

We thank the reviewer for the follow-up and appreciate the recognition of our Euclidean-selection ablation. We understand the concern regarding the breadth of CALP's applicability and would like to clarify two points.

First, our evaluation follows the benchmark environments used in the Scaling CRL paper, ensuring a direct and consistent comparison with prior work. These environments span navigation, locomotion, and manipulation, with substantially different dynamics and control challenges. We further extend the evaluation to Arm Binpick Hard and a more challenging Ant Maze. Across these diverse tasks, CALP matches or outperforms deep Scaling CRL while requiring substantially less training time.

Second, exploiting structured, low-dimensional goal spaces is an intentional inductive bias and a central motivation of our work. Such representations are meaningful in many robotic control problems, where task objectives can be expressed through positions, configurations, or other compact goal variables. We believe that understanding how to leverage this structure to improve learning efficiency is a valuable research question in its own right.

We agree that extending CALP to learned or image-based goal representations would further broaden its applicability, and we will explicitly acknowledge this limitation in the revised paper. We hope the reviewer will also consider that our central contribution is demonstrating how goal-space structure can reduce the need for network depth while achieving strong performance across diverse and challenging online GCRL tasks, rather than pursuing representation-agnostic generality.

We appreciate the reviewer's continued consideration.

Comments on the paper

1 comment

We thank all reviewers for their constructive feedback. We have revised the manuscript to incorporate the feedback, added new experiments, and clarified several methodological and implementation details. The main changes in the updated PDF are summarized below:

  • Competence supervision: We added ablations comparing graded execution-progress supervision with binary waypoint-completion labels. Graded supervision achieves substantially better performance on Ant Push and Pusher Hard, demonstrating the importance of learning from partial execution progress (Table 1).
  • Temporal-distance learning: We added median-quantile ablations and reorganized the component ablation table to clarify the contributions of competence estimation, learned temporal-distance guidance, and distance representation. The results further support the effectiveness of lower-quantile temporal-distance estimation (Sec. 5).
  • Long-detour maze experiment: We introduced a new Ant Maze experiment evaluating success rates across different detour ratios, using matched evaluation goals and the original JaxGCRL Ant dynamics. CALP consistently outperforms 64-layer Scaling CRL across all evaluated detour ranges, including the most challenging goals (Appendix A).
  • Additional manipulation evaluation: We added results on Arm Binpick Hard, extending the evaluation to grasping, lifting, and object transport. CALP achieves higher time-at-goal performance than 64-layer Scaling CRL under the same training budget (Appendix A).
  • Architecture and implementation details: We clarified the network architectures, parameter counts, competence-label construction, waypoint timeout, and fallback behavior to improve reproducibility (Appendix A).
  • Discussion and limitations: We refined the central contribution of the paper, emphasizing how exploiting structured goal spaces through competence-aware local planning can reduce the network depth required for long-horizon goal reaching. We also clarified the assumptions on goal representations and the challenges of extending CALP to image-based goals (Sec. 6).

Together, these revisions strengthen the empirical evidence and clarify the scope of our contributions. Our evaluation covers a broad and diverse set of JaxGCRL tasks, including maze navigation, object manipulation, and contact-rich control, spanning different dynamics and exploration challenges. Across these settings, our results demonstrate that competence-aware local planning, combined with lightweight learned temporal guidance, can substantially reduce the network depth and computational cost required for effective long-horizon goal reaching.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.