acceptodds
Under review as a conference paper at ICLR 2027

Hierarchical Time-Aware Bootstrapping for Off-Policy Subgoal Value Learning

Abstract

Off-policy hierarchical reinforcement learning must estimate the values of high-level decisions while the low-level policy changes. HIRO adapts replay data through subgoal relabeling, but after a label change, the value update targets the relabeled subgoal instead of the subgoal the high-level policy originally needed to update. We propose Hierarchical Time-aware Bootstrapping (HTB), which evaluates specified subgoals under the current low-level policy while retaining accumulated task rewards. Remaining execution time distinguishes subgoal continuation from a new high-level decision. Together with primitive-action conditioning, it enables off-policy Bellman updates based on the stationary environment transition law. HTB combines these one-step updates with multi-step suffix returns and truncated relabeling, reducing dependence on intermediate value estimates. A shared value component supports learning across actions, while nonnegative residuals constrain upward corrections relative to that component. At a fixed mixture weight of .95, HTB achieves 32.8% AntFall success versus 9.6% for matched local HIRO over five paired seeds at 10M environment steps. Ablations identify contributions from recursive continuation and mixed supervision; fixed-policy tests show more accurate predictions for actions whose returns were excluded from fitting.

Primary area:
learning theory
Keywords:
Hierarchical reinforcement learning, Policy non-stationarity, Off-policy learning, Subgoal value learning, Goal relabeling

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_AbCwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

A new algorithm, Hierarchical Time-aware Bootstrapping (HTB), is proposed in this paper. It is an off-policy hierarchical RL algorithm designed to address the nonstationarity caused by low-level policies which are changing over time during the training process of RL agent. In contrast to existing methods (such as HIRO), HTB enables direct value learning for specified subgoals without relying exclusively on hindsight relabeling. Specifically, HTB introduces a new Bellman updates which are condiitoned on remaining-ime. This new Bellman update distinguishes continued subgoal execution from new high-level decisions. In addition, the proposed algorithm also employs a shared subgoal-value and action-dependent residual decomposition to estimate values under the current low-level policy, by using past primitive transitions. Further, it combines one-step Bellman updates with multi-step suffix returns. In this case, with truncated subgoal relabeling, the dependence on intermediate value estimates is reduced and accumulated task rewards are preserved.

Experiments on AntPush, AntFall, Reacher, and Maze environments demonstrate improvements over HIRO in several settings, including an increase in AntFall success rate from 9.6% to 32.8%. Ablation studies demonstrate the contributions of recursive residual correction and mixed supervision. In addition, more experiments demonstrate improved value estimation for actions without direct training labels. However, performance improvements vary across environments, particularly in sparse-reward tasks.

Critical strengths
  1. An important limitation of existing off-policy HRL methods is addressed. This paper identifies a meaningful limitation of HIRO: hindsight subgoal relabeling changes which high-level action receives value supervision, and the originally selected subgoal can be potentially neglected. HTB addresses this issue by enabling direct evaluation of specified subgoals under the current low-level policy.

  2. The proposed time-aware Bellman formulation is technically sound. By introducing remaining-time-conditioned value functions, the proposed method explicitly distinguishes continued low-level execution from new high-level decisions. Theoretical propositions establishing the corresponding Bellman formulation are provided.

  3. This paper has an effective integration of complementary learning mechanisms. HTB combines one-step Bellman updates, multi-step suffix returns, and truncated subgoal relabeling. Since these components address different aspects of value estimation, their integration represents a coherent design of algorithm.

Critical weaknesses
  1. The first weakness is its limited novelty. The main components of HTB, including remaining-time-conditioned Bellman updates, primitive-step off-policy learning, value–advantage decomposition, and multi-step bootstrapping, are closely related with previous techniques, such as intra-option learning, time-aware value functions, and dueling networks. Although the integration of these technique is meaningful in some sense, authors do not provide sufficient evidence why previous methods cannot achieve similar functions and improvement with direct combinations.

  2. Theoretical guarantees for the proposed algorithm are not sufficiently established. The provided theoretical analysis only establishes Bellman formulation, and the convergence or performance guarantees are not provided. In particular, the convergence of the coupled actor–critic updates is not provided. Besides, there is no theoretical guarantee that the proposed residual-based learning consistently improves estimation accuracy.

  3. Althought HTB achieves significant improvements on AntFall, its benefits are less convincing on other tasks. For example, on Reacher-v1 sparse without relabeling, HTB achieves 96.1% success compared with 95.4% for the multi-step-only baseline. On sparse Maze tasks, even the simpler baseline can outperform HTB. These raise concerns whether the components in HTB offer consistent benefits across different control problems.

  4. This paper does not provide sufficient comparisons with other related previous methods. Although HTB is compared against HIRO and several internal variants, it lacks a direct comparison with the method of intra-option learning [1] which incorporates remaining-time conditioning and the method of conventional action-conditioned critic. Conducting these comparisons is important to determine whether HTB's residual decomposition provides benefits beyond existing off-policy temporal abstraction techniques. Moreover, comparisons with more recent hierarchical RL methods are also limited.

  5. In this paper, off-policy evaluation is based on restrictive assumptions. The theoretical formulation has strong assumptions, such as Markovian state representations, the known mapping from state to goal space, and sufficient action coverage. These assumptions may not hold in practical environments with partial observations.

[1] Intra-Option Learning about Temporally Abstract Actions. R. Sutton, et al.

[2] Time Limits in Reinforcement Learning. F. Pardo, et al.

Questions for authors
  1. Could the authors clarify the fundamental methodological distinction between HTB and the combinations of previously established techniques, such as intra-option off-policy [1] and remaining-time conditioning [2]? In particular, which capability of specified-subgoal evaluation cannot be achieved through such a combination?

  2. Proposition 1 is dependent on assumptions, such as the Markovian state representations and sufficient primitive-action coverage. However, the paper acknowledges that the Push/Fall observations omit object states, which may violate thses assumptions. Could the authors clarify how these violations affect the validity of the proposed off-policy updates?

  3. The experiments demonstrate that the shared V-A architecture improves value-prediction accuracy for actions excluded from training. However, lower prediction error does not necessarily imply better decision-making, such as high-level policy improvement. Can authors provide more evidence that improved value estimation leads to better decision-making?

  4. Why is the V-A residual decomposition needed instead of a conventional action-conditioned critic?

  5. On Reacher-v1 sparse, removing relabeling increases the success rate of HTB from 32.38% to 96.13% at lambda=0.95lambda=0.95. Why can removing subgoal relabeling produce such dramatic performance improvements?

5 comments
AuthorsOpenReview ↗

Response to Reviewer AbCw

Response to Reviewer AbCw

Thank you for identifying the importance of specified-subgoal evaluation and the validity of the time-aware formulation. The method follows a problem-driven design chain: construct the required off-policy update, reduce repeated bootstrapping, share value information across changing primitive actions, and control upward target amplification. We clarify this chain below and add the requested direct-Q and end-to-end evidence.

W1. A problem-driven construction rather than independent ingredients

The first weakness is its limited novelty. The main components of HTB, including remaining-time-conditioned Bellman updates, primitive-step off-policy learning, value–advantage decomposition, and multi-step bootstrapping, are closely related with previous techniques, such as intra-option learning, time-aware value functions, and dueling networks. Although the integration of these technique is meaningful in some sense, authors do not provide sufficient evidence why previous methods cannot achieve similar functions and improvement with direct combinations.

Response. HTB was constructed to solve one concrete problem: likelihood relabeling can remove direct supervision from the subgoal the manager needs to evaluate. Its design follows the obstacles encountered in making that update practical.

First, the remaining time is the time left in an already scheduled H-step subgoal invocation. At its boundary the task continues under a new manager decision; this differs from an episode time limit [2] and from choosing a timed subgoal as a high-level action [5]. Combined with primitive-action conditioning and goal transport, it gives an intra-option-style evaluator for continuously parameterized, changing workers [1].

Second, using that evaluator alone repeatedly propagates learned intermediate estimates. Multi-step suffix returns and their matching truncated relabeling replace intermediate estimates with observed rewards; nonnegative residuals filter upward corrections relative to a learned reference. Third, the changing worker shifts the action distribution. The dueling-inspired V−A structure [3] supplies a shared value offset rather than learning each newly queried action entirely through direct-Q extrapolation.

These choices have direct controls: one-step-only HTB (λ=0) reaches 6.50% on Reacher-v1 sparse and 0% on dense Maze-v1, versus 96.13% and 73.75% at λ=.95 without correction. On corrected AntFall, the matched time-aware direct-Q control reaches 0% versus 32.80% for full HTB. Together, these controls tie the targeted off-policy learning procedure to the original supervision problem and show why its practical estimation design matters.

W2. What the one-step theory establishes

Theoretical guarantees for the proposed algorithm are not sufficiently established. The provided theoretical analysis only establishes Bellman formulation, and the convergence or performance guarantees are not provided. In particular, the convergence of the coupled actor–critic updates is not provided. Besides, there is no theoretical guarantee that the proposed residual-based learning consistently improves estimation accuracy.

Response. The theoretical guarantee concerns the off-policy one-step construction, not just the presence of a time input. Proposition 1 identifies the conditional Bellman expectation for a recorded primitive transition, independently of the collecting worker, when the specified subgoal is chosen consistently with the sampling conditions. Remaining time then provides the correct continuation or manager-boundary case. The appendix additionally establishes the signed-residual reconstruction for fixed policies and boundary values. This is the intra-option-style policy-evaluation foundation extended to the task, subgoal and primitive-action value granularities.

The multi-step branch uses HIRO-style approximate behavior correction, and the practical algorithm couples learned actors, target networks and nonlinear estimators. The paper does not identify their global convergence with the fixed-policy Bellman results. The residual architecture's role is supported both by its shared-offset identity and by the matched end-to-end controls: on AntFall, HTB versus direct Q is 32.80% versus 0% with correction and 24.40% versus 6.80% without. We will distinguish these operator-level guarantees and empirical learning results explicitly.

W3. Robustness relative to HIRO and the internal mixture control

Althought HTB achieves significant improvements on AntFall, its benefits are less convincing on other tasks. For example, on Reacher-v1 sparse without relabeling, HTB achieves 96.1% success compared with 95.4% for the multi-step-only baseline. On sparse Maze tasks, even the simpler baseline can outperform HTB. These raise concerns whether the components in HTB offer consistent benefits across different control problems.

Response. The main comparison is HTB's ability to learn the required subgoal value under a changing worker, evaluated against HIRO across tasks with different correction demands. On the complex Push/Fall tasks, the matched five-seed success rates improve from 0% to 14.00% and from 9.60% to 32.80%. The benefit also appears on Reacher: without correction, Reacher-v1 sparse improves from independently implemented HIRO's 76.00% to HTB's 96.13% over eight paired seeds.

The 95.38% multi-step-only result is a different comparison: it is the λ=1 endpoint of the remaining-time architecture, not the separate HIRO implementation. A task on which that endpoint is already strong does not require the short branch to create another large gain. For transparency, the four sparse fixed-configuration comparisons are:

Sparse taskHIRO, no correctionMulti-step only, λ=1Full HTB, λ=.95
AntReacher-v032.13 ± 32.1454.25 ± 37.7848.63 ± 41.81
AntReacher-v176.00 ± 30.2795.38 ± 7.9396.13 ± 3.44
AntMaze-v146.88 ± 33.0535.00 ± 39.9119.38 ± 31.78
AntMazeW-v214.00 ± 11.9014.75 ± 15.1110.38 ± 11.41

All entries use no correction, eight seeds and the same 1M tail-window metric. The sparse Maze rows delimit the benefit of this fixed mixture. The clearer component evidence is on corrected Fall: multi-step only gives 13.60%, adding the full mixed update gives 32.80%, and removing next-step residual recursion gives 0%. We will keep the complete task results while organizing the contribution around the supervision capability and the settings where the corresponding practical components matter.

W4. Direct-Q controls and transfer to a reachability-constrained hierarchy

This paper does not provide sufficient comparisons with other related previous methods. Although HTB is compared against HIRO and several internal variants, it lacks a direct comparison with the method of intra-option learning [1] which incorporates remaining-time conditioning and the method of conventional action-conditioned critic. Conducting these comparisons is important to determine whether HTB's residual decomposition provides benefits beyond existing off-policy temporal abstraction techniques. Moreover, comparisons with more recent hierarchical RL methods are also limited.

Response. We have added a time-aware, parameter-matched direct-Q estimator with the same continuous subgoal interface, one-step/multi-step targets and correction setting. It is the direct control for the conventional action-conditioned evaluator in this task, rather than a transplant of a tabular discrete-option implementation. On AntFall at 10M, five-seed success is 0% versus HTB's 32.80% with correction, and 6.80% versus 24.40% without.

For external coverage, we completed Maze/MazeW HIRO controls, task-adapted official HRAC and flat TD3. HAC's path-independent hindsight-action rewards [4] and HiTS's timed-goal interface [5] address different settings. HRAC and HIGL improve reachability and exploration [6,7], whereas HTB changes how replay updates the required subgoal. These mechanisms can be combined. We therefore also inserted the HTB estimator/update into official-code HRAC, keeping its worker and reachability learner unchanged. The complete 1M transfer matrix uses three fresh paired training seeds and λ=.95:

TaskCorrectionHRACHRAC+HTB
AntReacher-v1on95.33 ± 2.5297.33 ± 2.89
AntReacher-v1off99.67 ± 0.5898.67 ± 1.53
AntMaze-v1on73.33 ± 7.64100.00 ± 0.00
AntMaze-v1off90.00 ± 5.0081.67 ± 10.41

Values are mean±SD success (%) over the last twenty native evaluations. Corrected dense AntMaze improves from 73.33% to 100%, demonstrating a concrete benefit within a reachability-constrained hierarchy. The complete matrix keeps the task-dependent effects visible. This transfer test is more informative about the update mechanism than attributing every difference between two complete exploration architectures to non-stationarity correction.

W5. State sufficiency, goal transport and action coverage

In this paper, off-policy evaluation is based on restrictive assumptions. The theoretical formulation has strong assumptions, such as Markovian state representations, the known mapping from state to goal space, and sufficient action coverage. These assumptions may not hold in practical environments with partial observations.

Response. These conditions identify what an off-policy update can learn from the available data. HTB uses the stationary primitive environment law, a consistent goal-transport map and coverage of the actions required by the evaluated worker. The state-sufficiency and coverage requirements are the usual requirements for identifying an off-policy Bellman expectation; explicitly choosing the supervised subgoal does not remove them. The goal map is part of the goal-conditioned interface, as in HIRO-style relative subgoals.

The construction therefore gives direct supervision for a specified command on the identified action support. Remaining time removes the ambiguity about continuing versus invoking the next high-level decision. It does not assert that an arbitrary partial observation is Markov or that an unsupported action value can be inferred without data. We will place these conditions alongside the proposition so that the theoretical guarantee and the benchmark observation interface are clearly distinguished.

Q1. Why the one-step construction is not the complete practical algorithm

Could the authors clarify the fundamental methodological distinction between HTB and the combinations of previously established techniques, such as intra-option off-policy [1] and remaining-time conditioning [2]? In particular, which capability of specified-subgoal evaluation cannot be achieved through such a combination?

Response. Adding an execution clock to an intra-option-style primitive evaluator gives the one-step starting point of HTB, not the complete practical learning procedure. It does not by itself solve repeated propagation of intermediate estimation errors or poor coverage of newly selected primitive actions under a changing worker.

We test whether this one-step supervision is sufficient by setting λ=0 while retaining the HTB architecture. The controlled results are:

TaskRewardCorrectionOne-step only, λ=0Full HTB, λ=.95
AntReacher-v1sparseoff6.50 ± 3.0296.13 ± 3.44
AntMaze-v1denseoff0.00 ± 0.0073.75 ± 9.54
AntFallPaper-v0denseon0.00 ± 0.0032.80 ± 21.10

Navigation uses eight seeds at 1M; AntFall uses five seeds at 10M. Values are success (%), mean±SD. These results explain the subsequent design choices: multi-step suffix returns replace learned intermediate continuations with observed rewards, nonnegative residuals filter upward corrections, and V−A shares information across actions whose sampling distribution changes. A suitably extended option framework can express the conditional evaluator; expressing it is different from obtaining a reliable continuous-control learner. HTB contributes the specific goal-supervision construction and the practical procedure supported by these controls.

Q2. Partial observations and the reason for mixed backups

Proposition 1 is dependent on assumptions, such as the Markovian state representations and sufficient primitive-action coverage. However, the paper acknowledges that the Push/Fall observations omit object states, which may violate thses assumptions. Could the authors clarify how these violations affect the validity of the proposed off-policy updates?

Response. Yes, omitting object state can violate the sufficient-state assumption. In that case, conditioning only on the critic's observation can mix hidden physical states in a policy-dependent way; Proposition 1 should not be applied literally to that observation as if it were a complete Markov state.

This is also a practical reason not to rely entirely on repeated one-step bootstrapping. Under function approximation and off-policy replay, an aliased or poorly generalized value can be propagated repeatedly through learned continuations. HTB trades this dependence against multi-step suffix returns: intermediate rewards are observed, and the learned bootstrap is deferred to the high-level boundary. The single-step path retains supervision for the specified subgoal, while the long path reduces the number of intermediate estimates. This mitigates an estimation problem; it does not turn partial observations into a fully observed process or supply a convergence theorem for the neural learner. We will make the distinction explicit.

Reviewer_HExtOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes HTB for off-policy hierarchical reinforcement learning when the low-level policy changes during training. HTB estimates subgoal values using low-level actions and the time remaining until the next high-level decision. It combines one-step TD targets for the original subgoals with multi-step targets for relabeled subgoals. Experiments evaluate the method and its components on several continuous control tasks.

Critical strengths

The paper distinguishes between correcting historical behavior and determining which subgoal receives supervision. Conditioning on the remaining time and action provides a clear formal framework for analyzing value updates for a specified subgoal.

Critical weaknesses

W1. Baseline comparisons are insufficient, and the benefits of the new components remain unclear.

  • The only external algorithm baseline is a local implementation of HIRO. Other representative HRL methods and flat baselines are missing. Maze/MazeW also lack HIRO comparisons. The local HIRO implementation performs substantially worse on Push and Fall than the original paper reports. The authors should clarify the protocol differences and provide further baseline validation.
  • HTB has a lower mean success rate than the multi-step TD baseline on three of the four tasks with sparse rewards (Table 8). The authors chose the highlighted settings after examining the results. Please use the same hyperparameter tuning budget for HTB and the baselines, then evaluate the chosen settings on new random seeds.

W2. The writing and figure organization make the paper difficult to read and its conclusions difficult to verify.

  • The relationships among subgoal transport, query selection, one-step and multi-step updates, and the different variants lack a centralized, intuitive explanation. Readers need to consult the appendix frequently. The definitions and complete settings of the different Reacher/Maze environment versions are also unclear.
  • Figs. 4, 7, 9, and 10 show differences between HTB variants, but not each variant’s success rate or return. Please put these results and the HIRO results in one table. In Fig. 3, HIRO has zero success on Push, so its curve is hard to distinguish from the horizontal axis. The main text should also state which tasks improve and which do not.
Questions for authors

Q1. Can HTB correct an underestimated value?

  • In Eq. 11, the one-step TD target cannot exceed (\bar V_t). If this estimate is too low, how does HTB correct it when the multi-step update trains on a different subgoal?
  • Table 24 reports higher MAE when (A) is constrained to be nonnegative. Could the authors test Ant Fall without this constraint on (A) and its target? Keep the recursion in Eq. 8 unchanged and report success rates and errors against rollout returns.

Q2. What does HTB add to intra-option TD learning?

  • How do Eqs. 2–3 differ from intra-option TD updates when options terminate after a fixed number of steps? Please move the discussion in Appendix G into the main text and identify what is new in the update and training procedure.

Q3. Does the critic need the low-level action as an input?

  • Could Table 2 include a critic that predicts the policy return without using the low-level action? Also, how much does changing the first action change the rollout return when the state, subgoal, and remaining time are fixed?

Q4. What is the training cost compared with HIRO?

  • With (H=10), HTB trains ten sets of high-level networks and uses every step in each segment for training. How many parameters do HTB and HIRO have? How long does one update take, and how long does a full training run take on the same hardware?
6 comments
AuthorsOpenReview ↗

Response to Reviewer HExt

Response to Reviewer HExt

Thank you for the detailed questions. HTB addresses a specific limitation of relabeling-based non-stationarity correction: the subgoal receiving the regression target can differ from the subgoal the manager needs to evaluate. Our response separates this update mechanism from subgoal exploration and reachability, explains the role of each design choice, and provides the requested controlled experiments and absolute results.

W1a. Distinguish non-stationarity correction from exploration and reachability

W1. Baseline comparisons are insufficient, and the benefits of the new components remain unclear.

The only external algorithm baseline is a local implementation of HIRO. Other representative HRL methods and flat baselines are missing. Maze/MazeW also lack HIRO comparisons. The local HIRO implementation performs substantially worse on Push and Fall than the original paper reports. The authors should clarify the protocol differences and provide further baseline validation.

Response. HIRO is the direct comparator for the replay-correction mechanism studied here, rather than simply the most recent HRL system. HAC's hindsight-action construction requires rewards independent of the lower-level path, using −1/0 subgoal-reaching costs [3]; two trajectories reaching the same endpoint with accumulated rewards 1 and 10 cannot generally be interchanged. HiTS instead uses timed subgoals to handle dynamic environments [4]. HRAC and HIGL address reachability and exploration within HIRO-style goal-conditioned hierarchies [5,6]. Comparing complete systems is useful, but does not isolate their non-stationarity update mechanisms.

We have nevertheless added the requested coverage: 64 HIRO runs on Maze/MazeW (both correction modes, eight seeds), official-code HRAC adaptations on four task–reward combinations, and five-seed 10M flat TD3 controls on Push/Fall. More directly, we replaced HRAC's high-level estimator/update with HTB while retaining its worker and reachability-learning procedure. The complete fresh-seed transfer comparison is below: three paired seeds per cell, 1M, final twenty native evaluations, mean±SD; HTB uses λ=.95 and independent time branches.

TaskCorrectionHRACHRAC+HTB
AntReacher-v1on95.33 ± 2.5297.33 ± 2.89
AntReacher-v1off99.67 ± 0.5898.67 ± 1.53
AntMaze-v1on73.33 ± 7.64100.00 ± 0.00
AntMaze-v1off90.00 ± 5.0081.67 ± 10.41

On dense AntMaze with correction enabled, success increases from 73.33% to 100%. The other cells show why we distinguish a reusable correction mechanism from a claim that replacing the estimator improves every complete HRL system.

Our HTB–HIRO comparisons use fixed seed sets, their means rather than a selected best seed, matched interaction budgets, and matched common hyperparameters. We also audited the original-code HIRO reproduction separately. It uses the authors' 2020 compatibility revision with TF 1.15.5, Gym 0.12.5 and MuJoCo 200, rather than claiming an identical 2018 software stack. Its ten-seed final means are 11.80±19.85% on Push and 28.60±21.85% on Fall; Only individual checkpoints reach 88% and 72%. Notably, this result remains significantly inferior to our proposed HTB algorithm. Furthermore, to ensure a strictly fair comparison, both HTB and HIRO were evaluated within the exact same codebase, sharing identical random seeds and common hyperparameter settings. Importantly, HIRO's Table 1 reports each seed's best policy within 10M and then averages ten seeds, whereas Figure 4 averages seeds at each training time [2].

W1b. The λ=1 control, fixed hyperparameters, and fresh seeds

HTB has a lower mean success rate than the multi-step TD baseline on three of the four tasks with sparse rewards (Table 8). The authors chose the highlighted settings after examining the results. Please use the same hyperparameter tuning budget for HTB and the baselines, then evaluate the chosen settings on new random seeds.

Response. Table 8 separates two comparisons. “Multi-step only” is the λ=1 remaining-time control: it removes one-step value supervision while retaining the time-indexed architecture. HIRO is the separately implemented external comparator. The former tests the contribution of the short branch; the latter tests the complete update mechanism. On Reacher-v1 sparse without correction, the corresponding success rates are HTB 96.13%, multi-step only 95.38%, and HIRO 76.00%. On Reacher-v0 sparse they are 48.63%, 54.25%, and 32.13%. Thus a small or negative difference from the internal λ=1 endpoint is not the same comparison as the difference from HIRO. The sparse Maze results remain in the complete table.

The paired comparisons hold the common HIRO/TD3 hyperparameters and interaction budget fixed. We do not select a seed to represent either method. We have additionally evaluated the frozen configurations on three previously unused seeds, with zero additional tuning for every method. The complete eight-task navigation validation is:

TaskFixed HTB, λ=.95Multi-step only, λ=1HIRO
Reacher-v0 dense93.00±4.3696.00±5.2995.00±2.65
Reacher-v0 sparse33.67±54.8549.33±42.6759.33±46.76
Reacher-v1 dense99.00±1.7397.67±0.5896.33±1.53
Reacher-v1 sparse98.00±1.0094.00±3.0096.67±2.31
Maze-v1 dense71.67±16.0780.00±13.2360.00±5.00
Maze-v1 sparse18.33±31.7558.33±51.0730.00±51.96
MazeW-v2 dense13.33±14.0516.33±6.4327.33±9.07
MazeW-v2 sparse10.33±15.3123.67±10.213.67±2.31

Entries are success (%), mean±SD across seeds 202611160–162 at 1M; HTB, multi-step only and HIRO use no correction. These new-seed results are not pooled with the original eight-seed results. They answer the request for evaluation without new-seed tuning; the original grid and fixed-configuration results will be labeled separately. It is worth noting that λ=1\lambda=1 is a variant of HTB rather than a standalone baseline. Moreover, these environments do not exhibit strong non-stationarity during training; Push and Fall are the challenging tasks characterized by strong non-stationarity. This highlights the necessity of trading off between multi-step and one-step bootstrapping, as they each offer distinct advantages in sparse-reward and dense-reward environments, respectively.

W2a. A centralized explanation of the design chain and task definitions

W2. The writing and figure organization make the paper difficult to read and its conclusions difficult to verify.

The relationships among subgoal transport, query selection, one-step and multi-step updates, and the different variants lack a centralized, intuitive explanation. Readers need to consult the appendix frequently. The definitions and complete settings of the different Reacher/Maze environment versions are also unclear.

Response. The presentation will follow the construction of the algorithm, not a list of disconnected components. The starting point is that HIRO-style relabeling may change the subgoal receiving supervision. To update the required subgoal instead, we first introduce remaining-time and primitive-action conditioning, obtaining the off-policy Bellman update in Proposition 1. A practical one-step-only estimator then faces repeated propagation of estimation error and sparse data for actions produced by a changing worker. Multi-step suffix returns reduce the number of learned intermediate continuations; truncated relabeling makes the likelihood window agree with that suffix; V−A shares a value offset across primitive actions; and nonnegative residuals filter upward corrections relative to that offset. The ablations test these specific roles.

We will put this chain, one shared execution example, and the two supervised subgoals at the start of the method. A single environment table will specify each Reacher/Maze version's geometry, observations, goal coordinates, reward, success criterion, termination and evaluation rule, rather than leaving the version names to carry these definitions.

W2b. Absolute baseline and ablation results

Figs. 4, 7, 9, and 10 show differences between HTB variants, but not each variant’s success rate or return. Please put these results and the HIRO results in one table. In Fig. 3, HIRO has zero success on Push, so its curve is hard to distinguish from the horizontal axis. The main text should also state which tasks improve and which do not.

Response. Below are absolute results rather than differences alone. The navigation table uses the fixed λ=.95 for all four HTB variants, λ=1 for the multi-step-only control, no correction throughout, eight paired seeds, 1M, and the mean of each seed's last twenty native evaluations. Values are success (%), mean±SD across training seeds.

TaskRewardHIRO, no correctionMulti-step only, λ=1Full HTB, λ=.95No next residualSigned, nonrecursive
AntReacher-v0dense94.25 ± 2.1992.63 ± 3.9695.38 ± 3.2094.50 ± 2.3991.63 ± 6.25
AntReacher-v0sparse32.13 ± 32.1454.25 ± 37.7848.63 ± 41.8152.50 ± 29.4345.88 ± 27.77
AntReacher-v1dense96.50 ± 2.2097.13 ± 1.7397.25 ± 2.2597.88 ± 0.9997.88 ± 1.73
AntReacher-v1sparse76.00 ± 30.2795.38 ± 7.9396.13 ± 3.4492.38 ± 9.4985.13 ± 21.11
AntMaze-v1dense66.25 ± 17.0660.62 ± 18.7973.75 ± 9.5469.38 ± 19.9055.62 ± 30.64
AntMaze-v1sparse46.88 ± 33.0535.00 ± 39.9119.38 ± 31.7825.62 ± 36.5920.62 ± 35.90
AntMazeW-v2dense26.25 ± 6.8626.88 ± 6.3329.25 ± 6.5428.00 ± 4.8425.63 ± 6.05
AntMazeW-v2sparse14.00 ± 11.9014.75 ± 15.1110.38 ± 11.4113.38 ± 19.3920.38 ± 13.90

“No next residual” removes next-step residual recursion; “Signed, nonrecursive” also uses signed outputs/targets; “Policy subgoal, nonrecursive” changes the queried command on that nonrecursive base. These are not three names for the same single-factor intervention.

For Push/Fall, the following tables use five paired seeds, 10M and the final fifty evaluation episodes. All HTB variants use λ=.95 except the explicitly labeled λ=1 control. Dashes mean that a matched result is not available in this table, not zero.

Success (%):

TaskCorrectionHIROMulti-step only, λ=1Full HTB, λ=.95No next residualSigned, nonrecursive
AntPushPaper-v0on0.00 ± 0.0013.20 ± 28.4114.00 ± 20.6916.80 ± 24.077.60 ± 16.99
AntPushPaper-v0off—0.00 ± 0.000.80 ± 1.790.00 ± 0.004.80 ± 10.73
AntFallPaper-v0on9.60 ± 13.2213.60 ± 18.6832.80 ± 21.100.00 ± 0.0010.00 ± 14.56
AntFallPaper-v0off—4.80 ± 10.7324.40 ± 25.8214.40 ± 19.770.00 ± 0.00

Unscaled external episode return:

TaskCorrectionHIROMulti-step only, λ=1Full HTB, λ=.95No next residualSigned, nonrecursive
AntPushPaper-v0on-5553.60 ± 365.68-5862.91 ± 1087.66-5045.34 ± 443.84-5194.51 ± 465.17-5845.45 ± 816.20
AntPushPaper-v0off—-5507.36 ± 253.68-5478.09 ± 192.77-6349.86 ± 691.68-5444.14 ± 201.22
AntFallPaper-v0on-8137.82 ± 339.70-7612.90 ± 394.72-7262.39 ± 666.79-10431.76 ± 2737.52-8540.46 ± 1054.13
AntFallPaper-v0off—-7922.14 ± 380.57-7577.43 ± 887.17-7580.89 ± 490.40-7857.18 ± 266.12

We will use visible open markers and a 0% label for the zero-success HIRO curve, without offsetting the data. The main text will emphasize the Push/Fall gains and the Reacher comparisons to HIRO, while explicitly retaining the sparse Maze and mixture-control outcomes. The full λ grid is kept separate from these fixed-configuration tables.

Reviewer_fJUWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This submission provide HTB learning framewrok that time-aware primitive-step value updates with relabeled multi-step returns for off-policy hierarchical RL. Experiments on AntFall shows the effectiveness to an extent.

The formulation (hierarchical replay under a changing low-level policy) is sensible and the experiments are carefully designed and documented.

Critical strengths
  • The problem is well motivated: relabeling changes which subgoal receives direct value supervision, while the controller may need to evaluate its original or newly proposed subgoals.

  • The Bellman formulation clearly distinguishes continuing a subgoal from making a new high-level decision.

Critical weaknesses
  • Representation needs improvement, e.g. Figure1 being referred too after confusing reader with complex flows. Sec. 3 introduces concepts with overwhelming math lacking intuitive introductions / explanations, which makes it hard to follow.
  • The one-step branch helps substantially on Fall, but adds little on several Reacher tasks and hurts some sparse tasks (Table 10). The frozen-policy diagnostic shows lower prediction error for a shared representation, but does not demonstrate better subgoal ranking or evaluate the actual trained HTB critic, which makes the contribution gain seems to be task dependent.
Questions for authors
  • Can the authors compare full HTB with recursive signed residuals and a matched direct-(Q) critic?

  • Do the trained HTB critics improve value accuracy and ranking for original and current-policy subgoals?

2 comments
AuthorsOpenReview ↗

Response to Reviewer fJUW

Response to Reviewer fJUW

Thank you for recognizing the supervision problem and the distinction between continuing a subgoal and making a new high-level decision. We have completed the requested recursive-signed and matched direct-Q controls, together with rollout-based evaluation of the actual trained critics. Below we explain how these results support the method and how we will make the presentation more direct.

W1. Introduce the execution example before the equations

Representation needs improvement, e.g. Figure1 being referred too after confusing reader with complex flows. Sec. 3 introduces concepts with overwhelming math lacking intuitive introductions / explanations, which makes it hard to follow.

Response. We will introduce the method through one complete high-level execution segment before the equations. The example first shows why relabeling can redirect supervision away from the required subgoal, then follows the remaining-time countdown and the two backup paths. For example, after four steps of a ten-step invocation, the continuation has six steps remaining rather than starting a new ten-step invocation. The same diagram will mark the specified subgoal supervised by the one-step branch and the suffix-relabeled subgoal supervised by the multi-step branch. We will move the reference to Figure 1 forward and use this example consistently throughout Section 3.

W2. Performance across different non-stationarity regimes

The one-step branch helps substantially on Fall, but adds little on several Reacher tasks and hurts some sparse tasks (Table 10). The frozen-policy diagnostic shows lower prediction error for a shared representation, but does not demonstrate better subgoal ranking or evaluate the actual trained HTB critic, which makes the contribution gain seems to be task dependent.

Response. The relevant comparison spans different demands on non-stationarity correction. HIRO's original analysis identifies a particular benefit of correction on the harder Push/Fall tasks [1]. In our matched experiments, HTB improves AntFall success from 9.60±13.22% for HIRO to 32.80±21.10%, and AntPush from 0.00±0.00% to 14.00±20.69% (five paired seeds, 10M).

The benefit is not confined to these tasks. With correction disabled on Reacher-v1 sparse, HTB achieves 96.13±3.44% versus 76.00±30.27% for the independently implemented HIRO, using eight paired seeds. The multi-step-only remaining-time control already reaches 95.38±7.93% there: this identifies a setting where adding the short branch changes little, rather than showing that HIRO and that control are the same implementation. The sparse Maze component results delimit the benefit of this fixed mixture. We have also evaluated the actual trained critics, not just refitted representations; the accuracy and ranking results are given in our answer to your second question.

Q1. Recursive signed residuals and parameter-matched direct Q

Can the authors compare full HTB with recursive signed residuals and a matched direct-(Q) critic?

Response. We have completed both requested controls on AntFall, with and without correction, using all five paired seeds at 10M. Recursive signed retains the next-residual recursion and removes both output nonnegativity and target projection. Matched direct Q retains remaining time, the specified-subgoal update, mixed backups and the corresponding correction setting; critic parameter counts differ by approximately 0.24%.

EstimatorCorrectionSuccess (%)Rollout policy-value MAE
Full HTBon32.80±21.10496.57±196.10
Recursive signedon13.20±18.20568.81±199.36
Matched direct Qon0.00±0.00985.19±164.65
Full HTBoff24.40±25.82693.07±245.93
Recursive signedoff5.20±11.63915.51±140.14
Matched direct Qoff6.80±15.21857.22±243.90

Entries are mean±training-seed SD. Success uses the final 50 evaluation episodes; MAE uses complete simulator rollouts of each model's own frozen hierarchy on common states and commands, with no learned-value tail. MC values use reward scale .1 and discount .99 at manager boundaries. These results support the practical roles of the nonnegative recursive estimator and the V−A decomposition. The existing nonrecursive signed ablation is kept separate from this newly requested recursive control.

Q2. Actual critic accuracy and continuous-subgoal decisions

Do the trained HTB critics improve value accuracy and ranking for original and current-policy subgoals?

Response. Return accuracy does improve in the actual trained critics. For the comparison against HIRO, we use a separate paired-critic probe: five paired HTB/HIRO models on 120 common full-physics states and 2,880 complete rollout branches. Mean MAE is 497.02 for HTB versus 527.57 for HIRO; at the early probe time it is 251.85 versus 353.77, and for each model's current-manager command it is 484.92 versus 532.20. Each reference return follows the corresponding frozen hierarchy.

In a continuous subgoal space, a global improvement in ordering is a stronger claim than either lower prediction error or improved control. Our finite-candidate ranking probe does not establish that claim (pairwise accuracy: HTB 50.97%, HIRO 53.61%). We therefore support the decision-making argument with actual control results: AntFall success is 32.80% versus HIRO's 9.60%, and the matched direct-Q control achieves 0% with correction and 6.80% without, versus HTB's 32.80% and 24.40%. The probes cover fresh-rollout and counterfactual commands; they do not retrospectively reconstruct historical replay labels.

Closing

In summary, HTB addresses the loss of direct supervision for the subgoal of interest under relabeling. The requested recursive-signed and matched direct-Q controls strengthen the evidence for its practical estimator, while the trained-critic rollouts provide a direct assessment of value accuracy. We will also bring the execution example and the two backup paths forward to make the construction easier to follow. Thank you again for recognizing the motivation and Bellman formulation, and for the constructive suggestions that have helped us sharpen both the presentation and the empirical evidence.

References

[1] Nachum et al. Data-Efficient Hierarchical Reinforcement Learning. NeurIPS 2018. Paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.