Bridging Hierarchical Reinforcement Learning via Subgoal Alignment

Hierarchical reinforcement learning (HRL) decomposes complex tasks into high-level subgoal planning and low-level execution, enabling improved performance in long-horizon settings. However, HRL does not consistently outperform flat policies because imperfect subgoal design can cause misalignment between planning and control, leading to information loss and suboptimal behavior, particularly when the task structure is not well aligned with hierarchical decomposition. To address this issue, we propose BRIDGE, a bidirectional consistency regularization framework to achieve subgoal alignment in HRL, enhancing coordination between hierarchical levels. Our method introduces Reachability Regularization to constrain high-level subgoals based on low-level reachability, and Mutual Information Regularization to enforce dependence between subgoals and state–action trajectories, strengthening both top-down feasibility and bottom-up consistency. Extensive experiments across offline goal-conditioned manipulation and locomotion tasks show that our approach improves HRL performance over existing baselines, particularly in scenarios where conventional hierarchical methods suffer from task–hierarchy mismatch.