acceptodds
Under review as a conference paper at ICLR 2027

Bridging Hierarchical Reinforcement Learning via Subgoal Alignment

Abstract

Hierarchical reinforcement learning (HRL) decomposes complex tasks into high-level subgoal planning and low-level execution, enabling improved performance in long-horizon settings. However, HRL does not consistently outperform flat policies because imperfect subgoal design can cause misalignment between planning and control, leading to information loss and suboptimal behavior, particularly when the task structure is not well aligned with hierarchical decomposition. To address this issue, we propose BRIDGE, a bidirectional consistency regularization framework to achieve subgoal alignment in HRL, enhancing coordination between hierarchical levels. Our method introduces Reachability Regularization to constrain high-level subgoals based on low-level reachability, and Mutual Information Regularization to enforce dependence between subgoals and state–action trajectories, strengthening both top-down feasibility and bottom-up consistency. Extensive experiments across offline goal-conditioned manipulation and locomotion tasks show that our approach improves HRL performance over existing baselines, particularly in scenarios where conventional hierarchical methods suffer from task–hierarchy mismatch.

Primary area:
reinforcement learning
Keywords:
hierarchical reinforcement learning, subgoal alignment

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_vFZ7OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper proposes BRIDGE, a bidirectional subgoal alignment framework for offline goal-conditioned hierarchical reinforcement learning (HRL). The method introduces two regularization terms: (1) reachability regularization, which encourages the high-level policy to generate subgoals that induce valuable actions under the low-level policy, and (2) mutual information regularization, which encourages low-level actions to remain dependent on the assigned subgoals. The authors evaluate BRIDGE on a wide range of tasks, demonstrating improved performance over several flat RL and HRL baselines.

Critical strengths
  1. Extensive empirical evaluation. The paper evaluates the proposed method across diverse environments and provides ablation studies, robustness evaluations, and scalability analysis.

  2. Intuitive and complementary regularization mechanisms. The two regularizers are well motivated. Reachability regularization encourages the high-level policy to generate executable and task-relevant subgoals, while mutual information regularization encourages the low-level policy to remain responsive to the assigned subgoals.

  3. Interesting investigation of HRL effectiveness. The paper attempts to investigate when and why hierarchical methods succeed or fail, particularly when the task structure is not naturally aligned with hierarchical decomposition. This is an interesting question.

Critical weaknesses
  1. Insufficient comparison with existing subgoal alignment methods. The main contribution lies in the proposed two regularizers to improve subgoal alignment. However, the paper does not sufficiently investigate how or why these mechanisms are more effective than those in prior work. The main experimental section focuses largely on comparisons with methods based on different learning paradigms, while more directly related subgoal alignment methods are only compared in Appendix C. Furthermore, Appendix C reports only aggregate success rates, without detailed task-level comparisons or mechanism-level analysis. This makes it difficult to assess the methodological advantages of BRIDGE over existing alignment approaches.

  2. Limited analysis supporting the first claimed contribution. The paper claims to systematically investigate when and why hierarchical methods succeed or fail. However, the analysis in Section 5.3 mainly compares performance across selected environments, with relatively limited discussion of the factors underlying these differences. The failure-mode analysis also provides limited insight into how subgoal misalignment contributes to the observed failures. A more detailed discussion would better support this claimed contribution.

Questions for authors
  1. Could the authors provide per-environment or task-category results for the methods compared in Appendix C?

  2. Appendix D reports subgoal-action cosine similarity to evaluate reachability and dependency. Could the authors also report these metrics for existing subgoal alignment methods, rather than only comparing BRIDGE with its ablated variants? This would provide more direct evidence of the effectiveness of the proposed alignment mechanisms.

  3. Could the authors explicitly identify the key factors that affect the performance of HRL and provide further discussion on how they influence HRL performance?

7 comments

We sincerely thank the reviewer for the insightful and constructive feedback. In response, we have strengthened our methodological comparisons, introduced task-level diagnostics, and conducted additional experiments to better understand hierarchical decomposition, subgoal execution, and cross-level alignment.

R1. Methodological distinctions from prior subgoal-alignment methods and task-level comparisons

We thank the reviewer for highlighting the need for more direct comparisons with existing subgoal-alignment methods. We have expanded the discussion of CHER [1], BrHPO [2], and PIPER [3], moved the methodological comparison into the main text, and added detailed task-level results. Below, we clarify how BRIDGE differs from these methods, why its two regularizers are complementary, and how these distinctions translate into empirical performance.

(1) Methodological distinctions from CHER, BrHPO, and PIPER.

The key distinction is how BRIDGE evaluates high-level subgoals and coordinates them with low-level execution. Rather than relying primarily on intrinsic subgoal-reaching returns or distance-based reachability, BRIDGE directly evaluates candidate subgoals through the final-task value of the actions induced by the current low-level policy.

Specifically, for state ss, candidate subgoal zz, and final goal gg, BRIDGE defines Ug(s,z)=Ea∼πL(⋅∣s,z)[Qg(s,a)],U_g(s,z) = \mathbb{E}_{a\sim\pi_L(\cdot\mid s,z)} \left[Q_g(s,a)\right],

where Qg(s,a)Q_g(s,a) estimates the expected return toward the final task goal gg. This provides an executor-aware, task-oriented coordination signal: a subgoal is evaluated according to the final-task value of the behavior it induces under the current low-level policy.

Complementarily, BRIDGE introduces a variational mutual-information regularizer for I(z;a∣s)I(z;a\mid s), explicitly encouraging low-level actions to remain dependent on their assigned subgoals.

The following table summarizes the methodological differences.

MethodMain coordination signalHigh-level mechanismLow-level mechanism
CHER [1]Worker intrinsic subgoal-reaching returnIncorporates worker feedback into the manager objectiveOptimizes intrinsic subgoal-reaching behavior
BrHPO [2]Bidirectional reachability/distance feedbackDiscourages poorly reachable subgoalsMakes the two levels mutually responsive through reachability-based feedback
PIPER [3]Primitive-informed preferences, regularized by a primitive value estimateLearns a preference reward and relabels high-level replayProvides primitive feasibility feedback for preference construction
BRIDGEUg(s,z)U_g(s,z) and conditional subgoal–action dependenceScores candidates by the final-task value of executor-induced actionsEncourages explicit subgoal responsiveness through the inverse-model/MI surrogate

Unlike CHER's intrinsic-return coordination and BrHPO's distance-based reachability shaping, BRIDGE directly assesses whether subgoal-induced execution contributes to the final task. Unlike PIPER's preference-based reward construction, BRIDGE uses final-goal-conditioned critic feedback to evaluate the actions produced by the current executor. Its additional MI regularizer explicitly targets low-level subgoal responsiveness, establishing a complementary coordination mechanism rather than relying on high-level feasibility feedback alone.

(2) Task-level comparisons with existing coordination methods.

To directly address the lack of detailed comparisons in Appendix C, we provide task-level success rates (%) against CHER, BrHPO, and PIPER. The best result for each task is highlighted in bold.

MethodPointMaze-MPointMaze-LAntMaze-MAntMaze-LHumanoid-MHumanoid-LCube-SingleCube-DoublePuzzle-4×4Puzzle-4×5SceneOverall
CHER [1]72 ± 338 ± 376 ± 248 ± 429 ± 29 ± 258 ± 43 ± 140 ± 418 ± 326 ± 438 ± 2
BrHPO [2]74 ± 337 ± 360 ± 445 ± 334 ± 213 ± 243 ± 45 ± 241 ± 322 ± 228 ± 437 ± 3
PIPER [3]71 ± 338 ± 379 ± 252 ± 431 ± 211 ± 161 ± 44 ± 137 ± 417 ± 320 ± 338 ± 3
BRIDGE (Ours)55 ± 463 ± 291 ± 262 ± 337 ± 316 ± 272 ± 36 ± 185 ± 420 ± 354 ± 451 ± 2

BRIDGE achieves the highest success rate on 9 of 11 tasks, with an overall success rate of 51%, outperforming CHER and PIPER (38%) and BrHPO (37%). Notably, BRIDGE improves over the strongest competing method by 44 percentage points on Puzzle-4×4 (85% vs. 41%) and 26 percentage points on Scene (54% vs. 28%). These gains are particularly pronounced in tasks requiring coordinated, task-relevant execution across multiple stages.

Together, the methodological analysis and direct task-level comparisons clarify BRIDGE's contribution: it couples final-task-aware subgoal evaluation with explicitly subgoal-responsive execution, providing a complementary coordination mechanism that achieves stronger overall performance than existing subgoal-alignment approaches.

R2. Understanding when and why hierarchical methods succeed or fail

We thank the reviewer for this valuable suggestion. We strengthen our analysis by identifying three key factors governing HRL performance: task–hierarchy compatibility, high-level subgoal executability, and low-level subgoal responsiveness. Beyond comparing success rates, we provide task-independent structural diagnostics, direct subgoal execution measurements, and controlled interventions to explain how these factors contribute to hierarchical learning failures.

1. What determines the effectiveness of hierarchical decomposition?

To characterize when hierarchical decomposition is beneficial, we introduce three task- and data-based diagnostics that are independent of learned policy performance: compositional demand, semantic-operation duration, and temporal consistency. These metrics quantify whether a task provides meaningful and temporally stable intermediate operations that can support hierarchical planning.

We define:

Lsem=E(s,g)[minω∈Ω(s,g)∣ω∣]L_{\mathrm{sem}}=\mathbb{E}_{(s,g)}\left[\underset{\omega\in\Omega(s,g)}{\mathrm{min}}|\omega|\right]

ℓ~=median⁡(ℓ),\widetilde{\ell} = \operatorname{median}(\ell),

CVtype=∑up(u)Std⁡(ℓ∣u)Mean⁡(ℓ∣u).\mathrm{CV}_{\mathrm{type}} = \sum_u p(u) \frac{\operatorname{Std}(\ell\mid u)} {\operatorname{Mean}(\ell\mid u)}.

Here, Ω(s,g)\Omega(s,g) denotes the set of valid sequences of task-defined semantic operations connecting initial state ss to goal gg. A semantic operation is a goal-relevant intermediate task unit with a clearly defined objective, initiation condition, and completion criterion (e.g., reaching a waypoint or completing an object interaction). It may involve multiple low-level actions and environment steps. A sequence is valid if its operations follow a feasible order and collectively achieve the goal. ∣ω∣|\omega| is the number of operations in sequence ω\omega; ℓ\ell is the duration of an individual operation measured in environment steps; and p(u)p(u) is the empirical frequency of operation type uu. The diagnostics are computed using task-defined semantic events and offline trajectory statistics, without relying on learned policies or their success rates.

Specifically, compositional demand (LsemL_{\mathrm{sem}}) measures the expected minimum number of semantic operations required to complete a task, with larger values indicating a greater need for multi-stage planning. Operation duration (ℓ~\widetilde{\ell}) captures the median execution length of individual semantic operations in environment steps, with longer durations offering greater potential for temporal abstraction. Temporal variability (CVtype\mathrm{CV}_{\mathrm{type}}) measures the frequency-weighted variation in execution duration within each operation type, where lower values indicate more predictable operation durations and more stable temporal boundaries for hierarchical coordination.

TaskLsemL_{\mathrm{sem}}ℓ~\widetilde\ellCVtype\mathrm{CV}_{\mathrm{type}}Best flat baselineBest prior HRL baselineBRIDGE
PointMaze-Large2.40110.76360%38%63%
Scene-Play3.0020.50754%28%54%
Puzzle-4×47.80330.12822%85%85%

The results reveal distinct structural characteristics. Puzzle-4×4 exhibits high compositional demand (7.807.80), long semantic operations (3333 steps), and low temporal variability (0.1280.128), providing clear and consistent intermediate abstractions. Accordingly, hierarchical learning substantially outperforms flat learning (85% vs. 22%). In contrast, PointMaze-Large exhibits substantially higher temporal variability (0.7630.763), making fixed-horizon subgoal execution less consistent, while Scene-Play has shorter operations (22 steps) and lower compositional demand (3.003.00), offering limited opportunities for temporal abstraction. On these tasks, conventional HRL underperforms the strongest flat methods.

These findings indicate that the benefit of hierarchy depends not merely on task complexity or horizon, but on whether intermediate operations are sufficiently meaningful, temporally extended, and consistent to support effective hierarchical coordination.

Reviewer_vvLfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary
  • The paper targets the misalignment between high-level subgoals and low-level execution by adding regularizers to an offline goal-conditioned hierarchical backbone. It is evaluated on 55 OGBench tasks and four D4RL Adroit tasks, against 4 flat and 3 hierarchical baselines.
Critical strengths
  • The motivation for having two regularizers is clear: one is for when the subgoal cannot be executed. the other is for when the subgoal is ignored.

  • The two regularizers adds almost no inference cost.

  • The experiments are on a diverse range of environments.

  • The experiment is covers many different aspects.

Critical weaknesses
  • It is unclear how much of the gain is from the regularizers rather than the flow-matching backbone or the choice of critic.

  • Limited novelty. Although the related work section includes methods that target subgoal reachability but the discussion and comparison is limited. They only stated that BRIDGE’s reachability term uses task-conditioned critic feedback compared to BrHPO but did not provide per-task results, and the other three are not compared.

  • Although the experiments includes various ablation and evaluation on scalability and robustness, they are only for a small subset of environments/methods. E.g. scalability is only on puzzle-4x6, and robustness is only on BRIDGE along. Therefore it is difficult to infer that BRIDGE is scalability on other tasks as well, and whether it is more robustness than the baselines.

Questions for authors
  • How are goals or success defined for the Adroit expert datasets?
  • SHARSA scores 0% on all five PointMaze-Large tasks (and HGCBC 1%). Have you tried different setups?
  • What is the setup for where the inference setup is measured?
  • There are two RD-HRL citations (line 652 - 658).
4 comments

We sincerely thank the reviewer for the thoughtful and constructive feedback. In response, we have conducted additional ablation studies, expanded comparisons with related methods, and extended evaluations of scalability and robustness. We have also clarified experimental protocols and implementation details to further strengthen the paper.

R1. Disentangling the contributions of regularization and the flow-matching backbone

We thank the reviewer for raising this important question. To disentangle the contributions of the proposed regularizers and the flow-matching backbone, we conduct a controlled ablation across all 55 tasks in OGBench, applying the same reachability and MI regularizers to both standard HRL and flow-based HRL. To ensure a fair comparison, both approaches use the same critic architecture and an identical critic training protocol, including the training objective, offline data, and optimization configuration. Importantly, the Qg(s,a)Q_g(s,a) used in the reachability regularizer is exactly the task-conditioned critic learned by the underlying HRL algorithm; BRIDGE does not introduce a separate critic or modify the critic architecture or training procedure.

MethodBaseline+Reachability+MI+Both (BRIDGE)
Standard HRL33%36%35%42%
Flow-based HRL40%42%43%51%

The results clearly distinguish the contributions of the backbone and our regularizers. First, the flow-matching backbone alone improves success from 33% to 40% (+7%). Second, our regularizers provide substantial additional gains under both backbones: +9% for standard HRL (33% → 42%) and +11% for flow-based HRL (40% → 51%). This demonstrates that the improvements are not merely attributable to the flow-matching backbone. Both regularizers also yield consistent individual improvements, while their combination achieves the strongest performance.

R2. Methodological novelty and detailed comparisons with prior subgoal coordination methods

We thank the reviewer for highlighting the need to better distinguish BRIDGE from existing subgoal coordination methods. We have expanded the related-work discussion, explicitly compared BRIDGE with CHER [1], BrHPO [2], and PIPER [3], and added task-level experimental comparisons to the main text. These additions clarify both the methodological novelty of BRIDGE's coupled objectives and its empirical advantages over closely related approaches.

(1) Methodological distinctions and novelty.

Existing methods coordinate hierarchical policies through different mechanisms:

  • CHER [1] incorporates the worker's intrinsic subgoal-reaching return into the manager's objective and propagates cooperative gradients.
  • BrHPO [2] uses distance-based reachability feedback to regularize high-level subgoal selection and shape low-level intrinsic rewards.
  • PIPER [3] combines task-oriented feedback with primitive subgoal-achievability values to construct preference-based rewards for high-level learning.

In contrast, BRIDGE introduces a coupled coordination mechanism that jointly addresses the final-task usefulness of high-level subgoal selection and the low-level executor's responsiveness to those subgoals. Specifically, its high-level regularizer evaluates candidate subgoals through the final-goal-conditioned value of the actions induced by the current low-level policy:

Ug(s,z)=Ea∼πL(⋅∣s,z)[Qg(s,a)].U_g(s,z)=\mathbb{E}_{a\sim\pi_L(\cdot\mid s,z)}[Q_g(s,a)].

Unlike intrinsic subgoal-reaching values or geometric reachability measures, this objective directly evaluates the final-task value of subgoal-induced execution. Complementarily, the low-level regularizer maximizes a variational surrogate of I(z;a∣s)I(z;a\mid s), encouraging the executor to produce actions that depend on the assigned subgoal.

The novelty lies in the functional coupling of these two objectives, rather than either regularizer in isolation. Without subgoal-responsive execution, πL(⋅∣s,z)=πL(⋅∣s)\pi_L(\cdot\mid s,z)=\pi_L(\cdot\mid s) makes Ug(s,z)U_g(s,z) independent of zz, eliminating the high-level coordination signal's ability to distinguish candidate subgoals. Conversely, subgoal-dependent actions alone do not guarantee progress toward the final task. BRIDGE addresses these complementary limitations by using mutual-information regularization to maintain subgoal-dependent execution and final-goal-conditioned critic feedback to favor task-useful behavior.

(2) Detailed comparisons with closely related methods.

To substantiate these distinctions empirically, we extend the evaluation with task-level comparisons against CHER, BrHPO, and PIPER. The following table reports success rates (%), with the best result in each column highlighted in bold.

MethodPointMaze-MPointMaze-LAntMaze-MAntMaze-LHumanoid-MHumanoid-LCube-SingleCube-DoublePuzzle-4×4Puzzle-4×5SceneOverall
CHER [1]72 ± 338 ± 376 ± 248 ± 429 ± 29 ± 258 ± 43 ± 140 ± 418 ± 326 ± 438 ± 2
BrHPO [2]74 ± 337 ± 360 ± 445 ± 334 ± 213 ± 243 ± 45 ± 241 ± 322 ± 228 ± 437 ± 3
PIPER [3]71 ± 338 ± 379 ± 252 ± 431 ± 211 ± 161 ± 44 ± 137 ± 417 ± 320 ± 338 ± 3
BRIDGE (Ours)55 ± 463 ± 291 ± 262 ± 337 ± 316 ± 272 ± 36 ± 185 ± 420 ± 354 ± 451 ± 2

BRIDGE achieves the highest success rate in 9 of 11 task groups, with an overall success rate of 51%, outperforming CHER (38%), BrHPO (37%), and PIPER (38%). Particularly substantial improvements are observed on Puzzle-4×4 (+44 %) and Scene (+26 %) relative to the strongest competing method in each task group. These results provide direct task-level evidence that BRIDGE is competitive with and often outperforms existing coordination approaches across diverse environments.

In summary, BRIDGE's methodological contribution is a coupled, executor-mediated coordination framework that connects final-task value feedback at the high level with explicit subgoal responsiveness at the low level. The expanded methodological analysis and direct comparisons clarify how this formulation differs from prior reachability-based coordination methods and provide stronger empirical support for its effectiveness.

R3. Scalability across environments and robustness against baselines

We thank the reviewer for suggesting a broader evaluation of scalability and robustness. To provide further evidence, we extend both evaluations to three environments and include comparisons with HIQL and SHARSA, examining BRIDGE's performance across different data scales and perturbation settings.

Scalability across environments. We extend the data scalability evaluation to PointMaze-Large, Scene-play, and Puzzle 4×4, comparing BRIDGE, HIQL and SHARSA at different training data fractions (10%, 50% and 100%).

EnvironmentMethod10%50%100%
PointMaze-LargeHIQL204252
SHARSA103
BRIDGE244963
Scene-playHIQL244046
SHARSA264452
BRIDGE254554
Puzzle-4×4HIQL285978
SHARSA306582
BRIDGE256485

Success rates (%).

At full data, BRIDGE achieves the highest success rate across all three environments. These results demonstrate that BRIDGE's effectiveness extends across different task structures and data scales, with stronger advantages as more training data becomes available.

Robustness against baselines. We further extend the robustness evaluation to Scene-play, Puzzle-4×4, and PointMaze-Large, comparing BRIDGE against HIQL and SHARSA under observation noise, action noise, and gravity variations.

EnvironmentMethodNominalObs. noiseAct. noise0.75G1.5G
Scene-playHIQL4633404342
SHARSA5241404947
BRIDGE5444535049
Puzzle-4×4HIQL7864637374
SHARSA8273708081
BRIDGE8577768286
PointMaze-LargeHIQL5249505252
SHARSA10111
BRIDGE6365646466

Success rates (%). Observation and action noise are gaussian noise, σ = 0.1.

BRIDGE achieves the highest success rate in most perturbed settings. Notably, it outperforms both baselines under every perturbation on PointMaze-Large and maintains strong performance across the other two environments. These results demonstrate that BRIDGE's performance advantages largely persist under observation, action, and physical dynamics perturbations.

R4. Adroit task goals and success criteria

We follow the original task objectives and success criteria defined in the D4RL Adroit environments. Specifically, each environment computes a per-step goal_achieved indicator based on the following task-specific conditions:

TaskGoal and achievement criterion
DoorOpen the door until its hinge angle reaches at least 1.35 radians.
HammerDrive the nail into the target until its position error is below 0.01 m.
PenAlign the pen with the target, requiring a position error below 0.075 m and an orientation-vector dot product above 0.95.
RelocateMove the object to the target position with a position error below 0.1 m.

These criteria are directly inherited from the benchmark environments, ensuring consistency with the original Adroit task definitions.

R5. Additional hyperparameter configurations tuning for SHARSA and HGCBC

We conducted additional experiments with alternative configurations for both SHARSA and HGCBC to examine whether their low success rates result from suboptimal hyperparameter settings. Specifically, we varied the number of flow integration steps, exponential moving average (EMA) settings, and learning rates, where applicable to each method. With additional hyperparameter tuning, the best SHARSA configuration improves its average success rate across the five PointMaze-Large tasks from 0% to 3%, while HGCBC's average success rate remains at 1%. Despite this improvement, both methods still substantially underperform BRIDGE, indicating that the performance gap cannot be attributed solely to the originally reported baseline configurations.

R6. Inference latency measurement protocol

We measure inference latency using a shared benchmarking script based on the JAX implementation. Each measurement times a complete sample_actions call for a single observation (batch size = 1), with a goal provided when required.

The benchmark uses the observation dimensions of antmaze-large-navigate-v0, with preallocated zero-valued observation and goal vectors. We perform 5 warm-up calls to complete JIT compilation, followed by 50 timed calls. Each call is timed using time.perf_counter(), with block_until_ready() applied to the returned action to ensure device synchronization before recording the elapsed time. Both actor levels use Euler ODE integration with 10 steps at inference in the implementation described in Appendix I.

R7. Duplicate RD-HRL reference

We thank the reviewer for identifying the duplicate RD-HRL references. We remove the duplicate entry and update the corresponding in-text citations in the revised manuscript.

References

[1] Kreidieh et al. Inter-Level Cooperation in Hierarchical Reinforcement Learning.

[2] Luo et al. Bidirectional-Reachable Hierarchical Reinforcement Learning with Mutually Responsive Policies. RLC, 2024.

[3] Singh et al. PIPER: Primitive-Informed Preference-based Hierarchical Reinforcement Learning via Hindsight Relabeling. ICML, 2024.

Reviewer_Ri6sOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies offline hierarchical goal-conditioned reinforcement learning, where the low-level policy is expected to execute the subgoals proposed by the high-level policy. It mainly aims to address the potential mismatch between the low-level and high-level policies, one of the inherent challenges in this field. Specifically, it uses a Q-function to encourage the high-level policy to generate subgoals that account for the low-level policy’s reachability, while using mutual information regularization to encourage the low-level policy to produce actions aligned with the given subgoals. Experiments are conducted on OGBench, an offline RL benchmark, demonstrating the usefulness of the proposed method. The paper further examines its effectiveness through ablation studies and visualizations.

Critical strengths
  1. The motivation that simply introducing a hierarchy is insufficient in goal-conditioned RL is valid, and the argument that misalignment between high-level and low-level policies can degrade HRL performance is reasonable. To address this misalignment, the paper proposes a bidirectional regularization approach which is intuitive at a conceptual level.

  2. The method demonstrates substantial performance improvements across a diverse range of benchmark tasks, with visualizations and ablation studies providing strong empirical support for its effectiveness.

Critical weaknesses
  1. The abstract and introduction state “particularly when the task structure is not well aligned with hierarchical decomposition.” and “HRL’s structural complexity can be unnecessary or even harmful when tasks lack a natural hierarchical structure”. However, the experimental analysis in Section 5.3 does not sufficiently justify categorizing pointmaze-large-navigate and scene-play as having less explicit hierarchical structure, while treating puzzle-4×4 as more suitable for hierarchy. Depending on how the hierarchy is defined, these tasks can also admit natural subgoal decompositions. Therefore, the results do not sufficiently support the interpretation that the proposed method improves performance specifically by addressing task–hierarchy mismatch.

  2. In Section 4.2, the mutual information (MI) regularizer minimizes the expected squared L2 distance between the subgoal zz and its prediction from the current state and the generated action. However, it is unclear whether minimizing this prediction error encourages actions that help achieve zz, and further justification or empirical evidence is needed to support this connection.

  3. The main text is not sufficiently self-contained, and the overall training procedure is difficult to follow. Section 4 presents the base learning objective and two regularizers without clearly explaining how they fit into the full algorithm before moving on to the experiments. In particular, the encoder fθzf_{\theta_z} appears in Eq. (7) without sufficient introduction, and how it is trained is unclear.

Questions for authors
  1. In Section 4.2, the MI regularizer encourages the subgoal zz to be recoverable from the current state and the generated action. Could the authors provide evidence that this also improves the low-level policy’s ability to achieve zz, beyond making its actions distinguishable across subgoals?

  2. Could the authors clarify the criteria for determining whether a task is suitable for hierarchical decomposition? What evidence supports attributing the observed improvements to addressing task–hierarchy mismatch?

6 comments

We thank the reviewer for the thoughtful questions. We clarify the criteria for assessing a hierarchical interface, distinguish subgoal responsiveness from subgoal achievement, and explain how the encoder and the two regularizers enter the training procedure.

R1. What makes a task suitable for hierarchical decomposition?

We thank the reviewer for highlighting the need to better justify our task categorization. Our distinction is that PointMaze-Large and Scene-Play exhibit less explicit hierarchical structure, whereas Puzzle-4×4 is more naturally compatible with hierarchical decomposition. This distinction is grounded in the extent to which task structure provides meaningful, temporally coherent intermediate operations.

To substantiate this characterization, we introduce task-based diagnostics independent of learned policy performance, examining the compositional demand of each task in terms of the number of meaningful intermediate operations required for completion, as well as its temporal organization, including the duration and consistency of these operations. Specifically, we quantify these characteristics as follows:

Lsem=E(s,g)[minω∈Ω(s,g)∣ω∣]L_{\mathrm{sem}}=\mathbb{E}_{(s,g)}\left[\underset{\omega\in\Omega(s,g)}{\mathrm{min}}|\omega|\right]

ℓ~=median⁡(ℓ),\widetilde{\ell} = \operatorname{median}(\ell),

CVtype=∑up(u)Std⁡(ℓ∣u)Mean⁡(ℓ∣u).\mathrm{CV}_{\mathrm{type}} = \sum_u p(u) \frac{\operatorname{Std}(\ell\mid u)} {\operatorname{Mean}(\ell\mid u)}.

Here, Ω(s,g)\Omega(s,g) denotes the set of valid sequences of task-defined semantic operations connecting the initial state ss to goal gg, where semantic operation refers to a task-defined, goal-relevant unit of progress (e.g., reaching a waypoint or completing an object interaction). ℓ\ell denotes the execution duration (in environment steps) of an individual semantic operation, uu denotes its operation type, and p(u)p(u) is the empirical frequency of operation type uu.

Accordingly, LsemL_{\mathrm{sem}} measures compositional depth, with larger values indicating a greater need for intermediate operations. ℓ~\widetilde{\ell} measures the typical execution duration of an operation, and CVtype\mathrm{CV}_{\mathrm{type}} measures within-type duration variability. Longer, temporally consistent operations provide stronger support for stable high-level abstractions, whereas short or highly variable operations offer less explicit temporal structure for hierarchical coordination. These diagnostics rely on predefined semantic events and offline trajectories rather than learned value functions or policy success rates.

TaskStructural characterizationLsemL_{\mathrm{sem}}ℓ~\widetilde{\ell} (steps)CVtype\mathrm{CV}_{\mathrm{type}}
PointMaze-LargeLess explicit hierarchy2.4011.000.763
Scene-PlayLess explicit hierarchy3.002.000.507
Puzzle-4×4Hierarchy-compatible7.8033.000.128

The results provide task-level evidence supporting our characterization:

  • Puzzle-4×4 (hierarchy-compatible) exhibits the highest compositional demand (7.807.80), longest operation duration (3333 steps), and lowest duration variability (0.1280.128). Its sequential, temporally consistent operations provide clear and stable intermediate abstractions, making it naturally compatible with hierarchical decomposition.
  • PointMaze-Large (less explicit hierarchy) exhibits substantially higher duration variability (0.7630.763), indicating that intermediate transitions lack consistent temporal boundaries. Consequently, its task structure provides weaker support for stable hierarchical decomposition.
  • Scene-Play (less explicit hierarchy) exhibits lower compositional demand (3.003.00) and much shorter operation durations (22 steps), suggesting limited opportunities for meaningful temporal abstraction and less pronounced hierarchical structure.

Overall, these diagnostics substantiate our distinction: Puzzle-4×4 exhibits explicit compositional and temporal structure conducive to hierarchical learning, whereas PointMaze-Large and Scene-Play provide less explicit hierarchical organization. This distinction helps explain why conventional hierarchical methods may benefit differently across tasks and motivates BRIDGE's improved high–low coordination, particularly when the underlying task structure offers weaker support for hierarchical decomposition.

R2. Does MI Improve Subgoal Execution Beyond Action Distinguishability?

We thank the reviewer for raising this question. To directly examine whether MI regularization improves subgoal execution rather than merely making actions more informative about the assigned subgoal, we evaluate actual low-level policy rollouts on PointMaze-Large.

Evaluation protocol and metrics. We hold each assigned latent subgoal ztz_t fixed for H=25H=25 low-level steps and evaluate 50 episodes per configuration. Let dk=∥ϕ(st,st+k)−zt∥2d_k=\|\phi(s_t,s_{t+k})-z_t\|_2 denote the distance between the assigned subgoal and the achieved state transition in the latent subgoal space, where ϕ\phi is the subgoal representation function. We measure:

  • Subgoal Achievement Error (SAE ↓): E[dH]\mathbb{E}[d_H], the residual latent subgoal error after execution.
  • Subgoal Success Rate (SSR ↑): E[1(dH<τ)]\mathbb{E}[\mathbf{1}(d_H<\tau)], the fraction of rollouts satisfying a fixed latent achievement threshold τ\tau at step HH. Here we make τ=1\tau=1
  • Subgoal Progress Ratio (SPR ↑): E[(d0−dH)/(d0+ϵ)]\mathbb{E}[(d_0-d_H)/(d_0+\epsilon)], the normalized reduction in latent subgoal distance.
  • Native Final-Goal Progress (↑): E[D(st,g)−D(st+H,g)]\mathbb{E}[D(s_t,g)-D(s_{t+H},g)], where DD measures environment-native distance to the final task goal gg. This directly measures physical progress toward the task goal, independently of latent-space metrics.
VariantSAE ↓SSR ↑SPR ↑Native progress ↑
Base0.75478%0.7871.954
Reachability only0.69886%0.8242.409
MI only0.66692%0.8562.306
BRIDGE (Full)0.64194%0.8932.931

Compared with the Base policy, MI regularization alone reduces subgoal execution error by 11.7%, increases subgoal success from 78% to 92% (+14 percentage points), and improves native final-goal progress by 18.0%. These improvements are measured from executed state transitions rather than subgoal prediction accuracy, providing direct empirical evidence that MI regularization improves subgoal-directed behavior beyond action distinguishability. Combining MI with reachability regularization yields the best results across all four metrics, demonstrating their complementary benefits for executable subgoal alignment.

R3. Clarification of the Training Procedure and Goal Encoder

We thank the reviewer for this suggestion. We clarify the complete training procedure, particularly how the goal encoder is learned and how the two regularizers interact with the base hierarchical learner. We will incorporate these details into Section 4 to make the method self-contained.

1. Unified training procedure. BRIDGE integrates advantage-weighted hierarchical flow matching with reachability and mutual-information (MI) regularization:

\underbrace{\mathcal{L}_V+\mathcal{L}_Q+\mathcal{L}_l+\mathcal{L}_h}_{\text{Base hierarchical learning}} +\lambda_r\mathcal{L}_{\mathrm{reach}} +\lambda_{\mathrm{MI}}\mathcal{L}_{\mathrm{MI}}.$$ Each training iteration consists of three steps: 1. **Critic learning:** Sample transitions and goals from the offline dataset; update the value function using expectile regression and the Q-function using TD learning. 2. **Hierarchical policy learning:** Train the low- and high-level actors using advantage-weighted flow matching, jointly learning the goal encoder through the low-level objective. 3. **Coordination regularization:** Apply reachability regularization to the high-level actor and MI regularization to the low-level actor. The combined objective summarizes the training procedure; **each loss updates only its designated parameters**, as detailed below. **2. Goal encoder and its learning signal.** The goal encoder $f_{\theta_z}$ (denoted $f_\phi$ in Algorithm 1) maps a state–goal pair to a 10-dimensional, unit-normalized latent representation: $$z_l=f_{\theta_z}(s,g_l),\qquad z_h^\star=f_{\theta_z}(s,s_h),$$ where $g_l$ is a sampled low-level goal and $s_h$ is an intermediate state sampled from the offline dataset. The low-level actor conditions on $z_l$, while the high-level actor learns to predict the encoded intermediate-state target $z_h^\star$. **Crucially, the encoder is learned end-to-end through the base low-level flow-matching loss:** $$\mathcal L_l = \mathbb E\left[ w_l\left\| v_{\theta_l}(s,f_{\theta_z}(s,g_l),x_t,t) -(a_{\mathcal D}-x_0) \right\|_2^2 \right],$$ $$x_t=(1-t)x_0+ta_{\mathcal D}.$$ where $a_{\mathcal D}$ is the dataset action, $x_0$ is Gaussian noise, and $w_l$ is the advantage weight. Gradients propagate through the goal-conditioning representation to $f_{\theta_z}$, enabling the encoder to learn action-relevant subgoal representations directly from offline data. **Thus, the encoder does not depend on MI regularization for its learning signal.** **3. Regularization and parameter-update scopes.** The two regularizers coordinate different levels of the hierarchy: - **Reachability regularization** evaluates high-level subgoals using the low-level actor and target critic. It updates **only the high-level actor**, while treating the low-level actor and critic as fixed, differentiable evaluators. - **MI regularization** encourages low-level actions to retain information about their assigned subgoals by reconstructing the subgoal from the state–action pair: $$\mathcal L_{\mathrm{MI}} =\mathbb E\left[ \left\|q_\psi(s,a_\pi)-z\right\|_2^2 \right],$$ $$z=\operatorname{sg}(f_{\theta_z}(s,g_l)),$$ where $a_\pi\sim\pi_l(\cdot\mid s,z)$, $q_\psi$ is the inverse predictor, and $\operatorname{sg}$ denotes stop-gradient. This loss updates **the low-level actor and inverse predictor**, but not the goal encoder. The stop-gradient applies only to the MI branch; the encoder remains trainable through the base flow-matching objective. We will define $f_{\theta_z}$ before Eq. (7), unify its notation with Algorithm 1, and include a compact training algorithm in Section 4 that explicitly specifies the update sequence, gradient flow, and parameter-update scopes.
Reviewer_PeXnOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors propose BRIDGE, an offline goal-conditioned hierarchical reinforcement learning method that tries to improve the alignment between high-level subgoal prediction and low-level execution to achieve the predicted subgoal. The method discusses two regularization terms: reachability regularization that uses a goal-conditioned value function to guide high-level subgoal prediction, and mutual information regularization, which encourages low-level actions to depend on (and thus not ignore) the assigned subgoals. The experiments on OGBench and D4RL benchmarks show improvements over several flat and hierarchical baselines.

Critical strengths
  1. The paper is well-motivated and addresses an important problem in HRL: the alignment between high-level subgoal prediction and low-level execution.

  2. The experimental evaluation is strong, covering several benchmarks and baselines. The ablations also discuss important aspects of the approach.

Critical weaknesses

Limited methodological novelty. Several prior works have investigated improving coordination between high-level subgoal prediction and low-level execution. For instance, CHER [1] incorporates low-level feedback into high-level policy optimization, while BrHPO [2] explicitly investigates bidirectional reachability and coordination between hierarchical policies. Furthermore, PIPER [3] incorporates low-level value-based regularization into high-level subgoal selection. Although BRIDGE introduces a different Q-based objective and mutual information regularization, the paper does not sufficiently distinguish its methodological contributions from these existing approaches or justify the novelty of their combination. Further, these approaches are not sufficiently cited, which raises concerns.

Insufficient evidence for the claimed subgoal alignment. The proposed reachability regularizer maximizes the final-goal Q-value of actions induced by predicted subgoals, but does not directly optimize whether the low-level policy can achieve those subgoals within its execution horizon. A more detailed evaluation would strengthen this claim. For instance, PEAR [4] measures the distance between predicted and achieved subgoals throughout training, which the authors should consider adding to the paper. This is true for the mutual information regularizer as well: it does not directly optimize either subgoal achievement or final-goal reachability.

[1] Kreidieh et al., 2019. Inter-Level Cooperation in Hierarchical Reinforcement Learning.

[2] Luo et al., 2024. Bidirectional-Reachable Hierarchical Reinforcement Learning with Mutually Responsive Policies.

[3] Singh et al., 2024. PIPER: Primitive-Informed Preference-based Hierarchical Reinforcement Learning via Hindsight Relabeling.

[4] Singh et al., 2025. PEAR: Primitive Enabled Adaptive Relabeling for Boosting Hierarchical Reinforcement Learning.

Questions for authors
  1. How does BRIDGE differ from previous HRL approaches, particularly CHER [1], BrHPO [2], and PIPER [3]? What is the specific methodological contribution beyond combining existing regularization ideas in an offline RL setting?

  2. Can the authors compare the subgoal reachability metric evaluation (PEAR [4]) to report a more direct subgoal-reachability estimate?

  3. Can the authors also provide quantitative evidence that the proposed regularization specifically improves performance under hierarchy mismatch?

7 comments

Thank you for the detailed feedback and for highlighting closely related work. We strengthen the related-work discussion and clarify the distinction between our training objectives, measured subgoal execution, and robustness to hierarchy mismatch.

R1. Methodological novelty relative to CHER, BrHPO, and PIPER

We acknowledge that improving coordination between hierarchical policies is an established research direction, including CHER [1], BrHPO [2], and PIPER [3]. Our methodological contribution is more specific: BRIDGE evaluates high-level subgoals using the final-goal value of actions generated by the current low-level executor, while explicitly regularizing the executor's dependence on those subgoals. These objectives are coupled through the same subgoal-to-action mapping, rather than simply regularizing two independent policies.

(1) How BRIDGE differs from CHER, BrHPO, and PIPER?

For a state ss, candidate subgoal zz, and final goal gg, BRIDGE evaluates Ug(s,z)=Ea∼πL(⋅∣s,z)[Qg(s,a)],U_g(s,z) = \mathbb{E}_{a\sim\pi_L(\cdot\mid s,z)} \left[Q_g(s,a)\right], where Qg(s,a)Q_g(s,a) estimates the action’s return for the final task goal gg. Thus, the high-level regularizer evaluates a candidate subgoal through the task value of the actions it induces under the current executor. The low-level regularizer complements this feedback by optimizing a variational surrogate for I(z;a∣s)I(z;a\mid s), encouraging the executor’s actions to retain information about the assigned subgoal.

The methodological distinction lies in what the high-level coordination signal evaluates and how the low-level policy is regularized. The table compares these mechanisms using common notation, within each method’s underlying learning objective.

MethodMain coordination signalHigh-level mechanismLow-level mechanism
CHER [1]Worker intrinsic subgoal-reaching returnIncorporates worker feedback into the manager objectiveOptimizes intrinsic subgoal-reaching behavior
BrHPO [2]Bidirectional reachability/distance feedbackDiscourages poorly reachable subgoalsMakes the two levels mutually responsive through reachability-based feedback
PIPER [3]Primitive-informed preferences, regularized by a primitive value estimateLearns a preference reward and relabels high-level replayProvides primitive feasibility feedback for preference construction
BRIDGEUg(s,z)U_g(s,z) and conditional subgoal–action dependenceScores candidates by the final-task value of executor-induced actionsEncourages explicit subgoal responsiveness through the inverse-model/MI surrogate

(2) What the coupling contributes beyond the individual regularizers?

BRIDGE's methodological contribution is not merely the addition of two regularizers, but their functional coupling through the same executor-conditioned action distribution. Specifically, the high-level objective evaluates subgoals based on the final-task value of their induced actions, while the low-level MI regularizer encourages these actions to remain responsive to the assigned subgoals. This coupling addresses two complementary degeneracies that neither regularizer can resolve alone.

First, the MI regularizer preserves the subgoal-discriminative capacity required by the high-level Q-based objective. Recall that BRIDGE evaluates a candidate subgoal (z) through

Ug(s,z)=Ea∼πL(⋅∣s,z)[Qg(s,a)].U_g(s,z)=\mathbb{E}_{a\sim\pi_L(\cdot\mid s,z)}[Q_g(s,a)].

For two candidate subgoals (z_1) and (z_2), their difference in executor-mediated task value is ∫Qg(s,a)[πL(a∣s,z1)−πL(a∣s,z2)]da.\int Q_g(s,a) \left[ \pi_L(a\mid s,z_1)-\pi_L(a\mid s,z_2) \right]da.

The two regularizers are functionally coupled, addressing complementary limitations within the same subgoal-to-action-to-task-value pathway.

First, the MI regularizer supports the effectiveness of the high-level Q-based objective. The high-level regularizer distinguishes subgoals through the task values of their induced actions. If the low-level policy ignores its subgoal input, i.e., πL(a∣s,z)=πL(a∣s)\pi_L(a\mid s,z)=\pi_L(a\mid s), then Ug(s,z)U_g(s,z) becomes constant across subgoals, making the value signal uninformative for subgoal selection. The MI regularizer mitigates this degeneracy by encouraging subgoal-dependent actions.

Second, the Q-based regularizer provides the task relevance that MI alone cannot ensure. While MI encourages distinguishable subgoal-conditioned behaviors, these behaviors may not contribute to the final task. By evaluating their induced actions using Qg(s,a)Q_g(s,a), the high-level regularizer favors subgoals that lead to task-useful behaviors.

Thus, the contribution goes beyond simply combining two regularizers: MI encourages the subgoal responsiveness required for meaningful executor-conditioned value evaluation, while the Q-based objective guides this responsiveness toward task-relevant behavior. Unlike CHER, BrHPO, and PIPER, which primarily rely on intrinsic rewards, reachability feedback, or primitive-informed preferences, BRIDGE couples final-task value evaluation with explicit subgoal–action dependence through the current low-level executor. The novelty lies in jointly addressing these interdependent failure modes within a unified offline HRL framework, rather than in either regularizer individually.

(3) BRIDGE's performance advantage.

We have added direct task-level comparisons with CHER, BrHPO, and PIPER to demonstrate BRIDGE’s overall performance advantage over these closely related coordination methods. The table reports task-level success rates (%); bold indicates the highest reported score in each column.

MethodPointMaze-MPointMaze-LAntMaze-MAntMaze-LHumanoid-MHumanoid-LCube-SingleCube-DoublePuzzle-4×4Puzzle-4×5SceneOverall
CHER72 ± 338 ± 376 ± 248 ± 429 ± 29 ± 258 ± 43 ± 140 ± 418 ± 326 ± 438 ± 2
BrHPO74 ± 337 ± 360 ± 445 ± 334 ± 213 ± 243 ± 45 ± 241 ± 322 ± 228 ± 437 ± 3
PIPER71 ± 338 ± 379 ± 252 ± 431 ± 211 ± 161 ± 44 ± 137 ± 417 ± 320 ± 338 ± 3
BRIDGE (Ours)55 ± 463 ± 291 ± 262 ± 337 ± 316 ± 272 ± 36 ± 185 ± 420 ± 354 ± 451 ± 2

BRIDGE leads in 9 of 11 task groups, achieving 51% overall success versus 38% for CHER and PIPER, 37% for BrHPO. Its largest gains over the strongest baseline in each task group occur on Puzzle-4×4 (+44%) and Scene (+26%).

R2. Direct evidence of subgoal alignment

We thank the reviewer for suggesting a direct evaluation of subgoal execution. We agree that it is necessary to assess how closely the assigned subgoals are realized within the low-level execution horizon. To address this concern, we evaluate the distance between assigned and achieved subgoals throughout training, following the predicted-versus-achieved evaluation principle used in PEAR [4],

\|g_t^{\mathrm{eval}}-\psi(s_{t+H})\|_2,$$ $$z_t=f_{\theta_z}(s_t,g_t^{\mathrm{eval}}),$$ where $g_t^{\mathrm{eval}}$ denotes the subgoal predicted by the high-level policy, $z_t$ is its encoded low-level condition, and $\psi$ extracts the matching native state. We report final-task success separately to distinguish subgoal execution from completion of the overall task. The evaluation covers five PointMaze-Large tasks, 50 episodes per task, $H=25$ low-level steps and three training seeds. We report all five evaluated checkpoints, from initialization to 100k training steps. | Variant | Metric | 0 | 25k | 50k | 75k | 100k | | ----------------- | ---------------------- | -----------------: | ----------------: | ----------------: | ----------------: | ----------------: | | PEAR | Subgoal distance ↓ | 21.111 ± 4.332 | 1.601 ± 0.311 | 1.169 ± 0.176 | 0.996 ± 0.138 | 0.981 ± 0.302 | | | Task success (%) ↑ | 0.0 | 45.9 ± 3.4 | 49.1 ± 1.2 | 50.5 ± 4.6 | 51.9 ± 3.4 | | Base HRL | Subgoal distance ↓ | 21.111 ± 4.332 | 2.104 ± 0.391 | 1.462 ± 0.214 | 1.327 ± 0.198 | 1.214 ± 0.176 | | | Task success (%) ↑ | 0.0 | 40.8 ± 4.6 | 45.8 ± 2.5 | 47.4 ± 2.7 | 49.7 ± 2.4 | | Reachability-only | Subgoal distance ↓ | 21.111 ± 4.332 | 1.823 ± 0.344 | 1.128 ± 0.143 | 0.984 ± 0.142 | 0.894 ± 0.136 | | | Task success (%) ↑ | 0.0 | 43.8 ± 4.2 | 48.3 ± 2.0 | 48.4 ± 2.2 | 50.2 ± 2.4 | | MI-only | Subgoal distance ↓ | 21.111 ± 4.332 | 1.891 ± 0.361 | 1.194 ± 0.156 | 1.036 ± 0.159 | 0.927 ± 0.149 | | | Task success (%) ↑ | 0.0 | 43.1 ± 4.5 | 47.8 ± 2.1 | 52.0 ± 2.3 | 51.8 ± 2.2 | | **Full (BRIDGE)** | **Subgoal distance ↓** | **21.111 ± 4.332** | **1.614 ± 0.415** | **0.958 ± 0.077** | **0.835 ± 0.193** | **0.758 ± 0.271** | | | **Task success (%) ↑** | **0.0** | **45.7 ± 5.1** | **57.2 ± 1.6** | **56.8 ± 1.2** | **62.9 ± 1.6** | BRIDGE achieves lower mean distances at 50k, 75k, and 100k. At 100k, BRIDGE reduces the mean distance by 22.7% (0.758 vs. 0.981), alongside 11.0 percentage points higher task success (62.9% vs. 51.9%). These results provide direct behavioral evidence that BRIDGE’s assigned subgoals are realized more closely within the fixed execution horizon. The intermediate checkpoints further show that this advantage is observed across later stages of training, extending the evidence beyond a final-checkpoint comparison.

Comments on the paper

1 comment

Dear Reviewers and Area Chair,

We sincerely thank the reviewers and the Area Chair for their time, consideration, and constructive feedback. Their comments helped us strengthen the methodological comparisons, provide direct evidence of subgoal alignment, and better understand when and why hierarchical decomposition succeeds or fails.

Core contributions

  • We identify cross-level misalignment as a key limitation of offline hierarchical reinforcement learning. High-level subgoals must be both executable by the low-level policy and influential on its behavior. BRIDGE addresses these two requirements through complementary reachability and mutual information (MI) regularization.
  • We introduce a coupled optimization mechanism in which the high-level policy evaluates subgoals through the final-task value of executor-induced actions, while MI regularization encourages explicit subgoal-dependent action generation.
  • We investigate three factors governing HRL effectiveness: task–hierarchy compatibility, high-level subgoal executability, and low-level subgoal responsiveness. Our analysis shows that task structure determines the potential benefit of hierarchy, while cross-level alignment determines whether this benefit can be realized.

Additional evidence and revisions

  • Methodological comparisons (PeXn, vvLf, vFZ7). We expand comparisons with CHER, BrHPO, and PIPER, clarifying their differences in coordination mechanisms. Across 11 task groups, BRIDGE achieves 51% overall success, outperforming CHER (38%), BrHPO (37%), and PIPER (38%), and achieves the highest performance in 9 groups. Its subgoal–action cosine similarity reaches 0.972, compared with 0.924 for the strongest baseline.

  • Direct subgoal alignment (PeXn, Ri6s, vFZ7). Following PEAR's evaluation principle, we measure assigned-versus-achieved subgoal distances throughout training. At 100k steps, BRIDGE reduces subgoal distance by 22.7% compared with PEAR (0.758 vs. 0.981), while improving task success by 11.0 percentage points (62.9% vs. 51.9%). We also provide PEAR's data usage, imitation coefficients, relabeling threshold, and dataset refresh schedule.

  • Actual subgoal execution with MI (Ri6s, vFZ7). Additional low-level rollout experiments show that MI alone improves subgoal success rate from 78% to 92%, reduces execution error by 11.7%, and improves native final-goal progress by 18.0%. Full BRIDGE further achieves 94% subgoal success, demonstrating improvements beyond action distinguishability.

  • Robustness to hierarchy mismatch (PeXn, vFZ7). Under perturbed subgoal guidance, BRIDGE improves success from 50.9% to 74.8% and reduces performance degradation from 20.4 to 6.9 percentage points at H=20. At H=25, success increases from 51.0% to 76.7%, while the performance drop decreases from 21.0 to 12.6 percentage points.

  • Task suitability and executor capability (Ri6s, vFZ7). We introduce task-level diagnostics based on compositional demand, semantic-event intervals, and temporal consistency, with precise definitions and deterministic event-extraction rules. We further compare weak and strong executors on identical state–subgoal pairs. Although stronger executors improve subgoal success, they do not necessarily yield a hierarchical advantage over flat policies. In particular, Puzzle-4×4 achieves 68% HRL success versus 22% for flat RL, whereas PointMaze-Large and Scene-Play show little or no hierarchical advantage.

  • Backbone ablations, scalability, and robustness (vvLf). Controlled ablations across all 55 OGBench tasks show that the regularizers improve success from 33% to 42% under standard HRL and from 40% to 51% under flow-based HRL. We also extend scalability and robustness evaluations to three environments, comparing BRIDGE with HIQL and SHARSA under different data fractions, observation/action noise, and gravity variations. BRIDGE achieves the highest success at full data across all three environments and maintains strong performance under most perturbations.

  • Implementation and reproducibility (Ri6s, vvLf). We clarify the training procedure, goal encoder learning, and parameter-update scopes. We additionally provide D4RL Adroit success criteria, further SHARSA and HGCBC hyperparameter tuning, inference-latency measurement details, and corrected references.

The additional experiments and clarifications extend our evidence beyond aggregate task performance to direct subgoal execution, controlled hierarchy-mismatch interventions, cross-method comparisons, and systematic task-structure analysis. Full results and implementation details are provided in our individual responses.

We sincerely appreciate the reviewers' suggestions. Their feedback has helped us improve our analysis, clarify our methodological contributions, and strengthen the paper. We hope the Area Chair and reviewers will find this additional evidence helpful in reassessing our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.