- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
This paper proposes BRIDGE, a bidirectional subgoal alignment framework for offline goal-conditioned hierarchical reinforcement learning (HRL). The method introduces two regularization terms: (1) reachability regularization, which encourages the high-level policy to generate subgoals that induce valuable actions under the low-level policy, and (2) mutual information regularization, which encourages low-level actions to remain dependent on the assigned subgoals. The authors evaluate BRIDGE on a wide range of tasks, demonstrating improved performance over several flat RL and HRL baselines.
Critical strengths
-
Extensive empirical evaluation. The paper evaluates the proposed method across diverse environments and provides ablation studies, robustness evaluations, and scalability analysis.
-
Intuitive and complementary regularization mechanisms. The two regularizers are well motivated. Reachability regularization encourages the high-level policy to generate executable and task-relevant subgoals, while mutual information regularization encourages the low-level policy to remain responsive to the assigned subgoals.
-
Interesting investigation of HRL effectiveness. The paper attempts to investigate when and why hierarchical methods succeed or fail, particularly when the task structure is not naturally aligned with hierarchical decomposition. This is an interesting question.
Critical weaknesses
-
Insufficient comparison with existing subgoal alignment methods. The main contribution lies in the proposed two regularizers to improve subgoal alignment. However, the paper does not sufficiently investigate how or why these mechanisms are more effective than those in prior work. The main experimental section focuses largely on comparisons with methods based on different learning paradigms, while more directly related subgoal alignment methods are only compared in Appendix C. Furthermore, Appendix C reports only aggregate success rates, without detailed task-level comparisons or mechanism-level analysis. This makes it difficult to assess the methodological advantages of BRIDGE over existing alignment approaches.
-
Limited analysis supporting the first claimed contribution. The paper claims to systematically investigate when and why hierarchical methods succeed or fail. However, the analysis in Section 5.3 mainly compares performance across selected environments, with relatively limited discussion of the factors underlying these differences. The failure-mode analysis also provides limited insight into how subgoal misalignment contributes to the observed failures. A more detailed discussion would better support this claimed contribution.
Questions for authors
-
Could the authors provide per-environment or task-category results for the methods compared in Appendix C?
-
Appendix D reports subgoal-action cosine similarity to evaluate reachability and dependency. Could the authors also report these metrics for existing subgoal alignment methods, rather than only comparing BRIDGE with its ablated variants? This would provide more direct evidence of the effectiveness of the proposed alignment mechanisms.
-
Could the authors explicitly identify the key factors that affect the performance of HRL and provide further discussion on how they influence HRL performance?
7 comments
R1. Methodological distinctions from prior subgoal-alignment methods and task-level comparisons
We thank the reviewer for highlighting the need for more direct comparisons with existing subgoal-alignment methods. We have expanded the discussion of CHER [1], BrHPO [2], and PIPER [3], moved the methodological comparison into the main text, and added detailed task-level results. Below, we clarify how BRIDGE differs from these methods, why its two regularizers are complementary, and how these distinctions translate into empirical performance.
(1) Methodological distinctions from CHER, BrHPO, and PIPER.
The key distinction is how BRIDGE evaluates high-level subgoals and coordinates them with low-level execution. Rather than relying primarily on intrinsic subgoal-reaching returns or distance-based reachability, BRIDGE directly evaluates candidate subgoals through the final-task value of the actions induced by the current low-level policy.
Specifically, for state , candidate subgoal , and final goal , BRIDGE defines
where estimates the expected return toward the final task goal . This provides an executor-aware, task-oriented coordination signal: a subgoal is evaluated according to the final-task value of the behavior it induces under the current low-level policy.
Complementarily, BRIDGE introduces a variational mutual-information regularizer for , explicitly encouraging low-level actions to remain dependent on their assigned subgoals.
The following table summarizes the methodological differences.
| Method | Main coordination signal | High-level mechanism | Low-level mechanism |
|---|---|---|---|
| CHER [1] | Worker intrinsic subgoal-reaching return | Incorporates worker feedback into the manager objective | Optimizes intrinsic subgoal-reaching behavior |
| BrHPO [2] | Bidirectional reachability/distance feedback | Discourages poorly reachable subgoals | Makes the two levels mutually responsive through reachability-based feedback |
| PIPER [3] | Primitive-informed preferences, regularized by a primitive value estimate | Learns a preference reward and relabels high-level replay | Provides primitive feasibility feedback for preference construction |
| BRIDGE | and conditional subgoal–action dependence | Scores candidates by the final-task value of executor-induced actions | Encourages explicit subgoal responsiveness through the inverse-model/MI surrogate |
Unlike CHER's intrinsic-return coordination and BrHPO's distance-based reachability shaping, BRIDGE directly assesses whether subgoal-induced execution contributes to the final task. Unlike PIPER's preference-based reward construction, BRIDGE uses final-goal-conditioned critic feedback to evaluate the actions produced by the current executor. Its additional MI regularizer explicitly targets low-level subgoal responsiveness, establishing a complementary coordination mechanism rather than relying on high-level feasibility feedback alone.
(2) Task-level comparisons with existing coordination methods.
To directly address the lack of detailed comparisons in Appendix C, we provide task-level success rates (%) against CHER, BrHPO, and PIPER. The best result for each task is highlighted in bold.
| Method | PointMaze-M | PointMaze-L | AntMaze-M | AntMaze-L | Humanoid-M | Humanoid-L | Cube-Single | Cube-Double | Puzzle-4×4 | Puzzle-4×5 | Scene | Overall |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| CHER [1] | 72 ± 3 | 38 ± 3 | 76 ± 2 | 48 ± 4 | 29 ± 2 | 9 ± 2 | 58 ± 4 | 3 ± 1 | 40 ± 4 | 18 ± 3 | 26 ± 4 | 38 ± 2 |
| BrHPO [2] | 74 ± 3 | 37 ± 3 | 60 ± 4 | 45 ± 3 | 34 ± 2 | 13 ± 2 | 43 ± 4 | 5 ± 2 | 41 ± 3 | 22 ± 2 | 28 ± 4 | 37 ± 3 |
| PIPER [3] | 71 ± 3 | 38 ± 3 | 79 ± 2 | 52 ± 4 | 31 ± 2 | 11 ± 1 | 61 ± 4 | 4 ± 1 | 37 ± 4 | 17 ± 3 | 20 ± 3 | 38 ± 3 |
| BRIDGE (Ours) | 55 ± 4 | 63 ± 2 | 91 ± 2 | 62 ± 3 | 37 ± 3 | 16 ± 2 | 72 ± 3 | 6 ± 1 | 85 ± 4 | 20 ± 3 | 54 ± 4 | 51 ± 2 |
BRIDGE achieves the highest success rate on 9 of 11 tasks, with an overall success rate of 51%, outperforming CHER and PIPER (38%) and BrHPO (37%). Notably, BRIDGE improves over the strongest competing method by 44 percentage points on Puzzle-4×4 (85% vs. 41%) and 26 percentage points on Scene (54% vs. 28%). These gains are particularly pronounced in tasks requiring coordinated, task-relevant execution across multiple stages.
Together, the methodological analysis and direct task-level comparisons clarify BRIDGE's contribution: it couples final-task-aware subgoal evaluation with explicitly subgoal-responsive execution, providing a complementary coordination mechanism that achieves stronger overall performance than existing subgoal-alignment approaches.
R2. Understanding when and why hierarchical methods succeed or fail
We thank the reviewer for this valuable suggestion. We strengthen our analysis by identifying three key factors governing HRL performance: task–hierarchy compatibility, high-level subgoal executability, and low-level subgoal responsiveness. Beyond comparing success rates, we provide task-independent structural diagnostics, direct subgoal execution measurements, and controlled interventions to explain how these factors contribute to hierarchical learning failures.
1. What determines the effectiveness of hierarchical decomposition?
To characterize when hierarchical decomposition is beneficial, we introduce three task- and data-based diagnostics that are independent of learned policy performance: compositional demand, semantic-operation duration, and temporal consistency. These metrics quantify whether a task provides meaningful and temporally stable intermediate operations that can support hierarchical planning.
We define:
Here, denotes the set of valid sequences of task-defined semantic operations connecting initial state to goal . A semantic operation is a goal-relevant intermediate task unit with a clearly defined objective, initiation condition, and completion criterion (e.g., reaching a waypoint or completing an object interaction). It may involve multiple low-level actions and environment steps. A sequence is valid if its operations follow a feasible order and collectively achieve the goal. is the number of operations in sequence ; is the duration of an individual operation measured in environment steps; and is the empirical frequency of operation type . The diagnostics are computed using task-defined semantic events and offline trajectory statistics, without relying on learned policies or their success rates.
Specifically, compositional demand () measures the expected minimum number of semantic operations required to complete a task, with larger values indicating a greater need for multi-stage planning. Operation duration () captures the median execution length of individual semantic operations in environment steps, with longer durations offering greater potential for temporal abstraction. Temporal variability () measures the frequency-weighted variation in execution duration within each operation type, where lower values indicate more predictable operation durations and more stable temporal boundaries for hierarchical coordination.
| Task | Best flat baseline | Best prior HRL baseline | BRIDGE | |||
|---|---|---|---|---|---|---|
| PointMaze-Large | 2.40 | 11 | 0.763 | 60% | 38% | 63% |
| Scene-Play | 3.00 | 2 | 0.507 | 54% | 28% | 54% |
| Puzzle-4×4 | 7.80 | 33 | 0.128 | 22% | 85% | 85% |
The results reveal distinct structural characteristics. Puzzle-4×4 exhibits high compositional demand (), long semantic operations ( steps), and low temporal variability (), providing clear and consistent intermediate abstractions. Accordingly, hierarchical learning substantially outperforms flat learning (85% vs. 22%). In contrast, PointMaze-Large exhibits substantially higher temporal variability (), making fixed-horizon subgoal execution less consistent, while Scene-Play has shorter operations ( steps) and lower compositional demand (), offering limited opportunities for temporal abstraction. On these tasks, conventional HRL underperforms the strongest flat methods.
These findings indicate that the benefit of hierarchy depends not merely on task complexity or horizon, but on whether intermediate operations are sufficiently meaningful, temporally extended, and consistent to support effective hierarchical coordination.
We sincerely thank the reviewer for the insightful and constructive feedback. In response, we have strengthened our methodological comparisons, introduced task-level diagnostics, and conducted additional experiments to better understand hierarchical decomposition, subgoal execution, and cross-level alignment.