- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
A new algorithm, Hierarchical Time-aware Bootstrapping (HTB), is proposed in this paper. It is an off-policy hierarchical RL algorithm designed to address the nonstationarity caused by low-level policies which are changing over time during the training process of RL agent. In contrast to existing methods (such as HIRO), HTB enables direct value learning for specified subgoals without relying exclusively on hindsight relabeling. Specifically, HTB introduces a new Bellman updates which are condiitoned on remaining-ime. This new Bellman update distinguishes continued subgoal execution from new high-level decisions. In addition, the proposed algorithm also employs a shared subgoal-value and action-dependent residual decomposition to estimate values under the current low-level policy, by using past primitive transitions. Further, it combines one-step Bellman updates with multi-step suffix returns. In this case, with truncated subgoal relabeling, the dependence on intermediate value estimates is reduced and accumulated task rewards are preserved.
Experiments on AntPush, AntFall, Reacher, and Maze environments demonstrate improvements over HIRO in several settings, including an increase in AntFall success rate from 9.6% to 32.8%. Ablation studies demonstrate the contributions of recursive residual correction and mixed supervision. In addition, more experiments demonstrate improved value estimation for actions without direct training labels. However, performance improvements vary across environments, particularly in sparse-reward tasks.
Critical strengths
-
An important limitation of existing off-policy HRL methods is addressed. This paper identifies a meaningful limitation of HIRO: hindsight subgoal relabeling changes which high-level action receives value supervision, and the originally selected subgoal can be potentially neglected. HTB addresses this issue by enabling direct evaluation of specified subgoals under the current low-level policy.
-
The proposed time-aware Bellman formulation is technically sound. By introducing remaining-time-conditioned value functions, the proposed method explicitly distinguishes continued low-level execution from new high-level decisions. Theoretical propositions establishing the corresponding Bellman formulation are provided.
-
This paper has an effective integration of complementary learning mechanisms. HTB combines one-step Bellman updates, multi-step suffix returns, and truncated subgoal relabeling. Since these components address different aspects of value estimation, their integration represents a coherent design of algorithm.
Critical weaknesses
-
The first weakness is its limited novelty. The main components of HTB, including remaining-time-conditioned Bellman updates, primitive-step off-policy learning, value–advantage decomposition, and multi-step bootstrapping, are closely related with previous techniques, such as intra-option learning, time-aware value functions, and dueling networks. Although the integration of these technique is meaningful in some sense, authors do not provide sufficient evidence why previous methods cannot achieve similar functions and improvement with direct combinations.
-
Theoretical guarantees for the proposed algorithm are not sufficiently established. The provided theoretical analysis only establishes Bellman formulation, and the convergence or performance guarantees are not provided. In particular, the convergence of the coupled actor–critic updates is not provided. Besides, there is no theoretical guarantee that the proposed residual-based learning consistently improves estimation accuracy.
-
Althought HTB achieves significant improvements on AntFall, its benefits are less convincing on other tasks. For example, on Reacher-v1 sparse without relabeling, HTB achieves 96.1% success compared with 95.4% for the multi-step-only baseline. On sparse Maze tasks, even the simpler baseline can outperform HTB. These raise concerns whether the components in HTB offer consistent benefits across different control problems.
-
This paper does not provide sufficient comparisons with other related previous methods. Although HTB is compared against HIRO and several internal variants, it lacks a direct comparison with the method of intra-option learning [1] which incorporates remaining-time conditioning and the method of conventional action-conditioned critic. Conducting these comparisons is important to determine whether HTB's residual decomposition provides benefits beyond existing off-policy temporal abstraction techniques. Moreover, comparisons with more recent hierarchical RL methods are also limited.
-
In this paper, off-policy evaluation is based on restrictive assumptions. The theoretical formulation has strong assumptions, such as Markovian state representations, the known mapping from state to goal space, and sufficient action coverage. These assumptions may not hold in practical environments with partial observations.
[1] Intra-Option Learning about Temporally Abstract Actions. R. Sutton, et al.
[2] Time Limits in Reinforcement Learning. F. Pardo, et al.
Questions for authors
-
Could the authors clarify the fundamental methodological distinction between HTB and the combinations of previously established techniques, such as intra-option off-policy [1] and remaining-time conditioning [2]? In particular, which capability of specified-subgoal evaluation cannot be achieved through such a combination?
-
Proposition 1 is dependent on assumptions, such as the Markovian state representations and sufficient primitive-action coverage. However, the paper acknowledges that the Push/Fall observations omit object states, which may violate thses assumptions. Could the authors clarify how these violations affect the validity of the proposed off-policy updates?
-
The experiments demonstrate that the shared V-A architecture improves value-prediction accuracy for actions excluded from training. However, lower prediction error does not necessarily imply better decision-making, such as high-level policy improvement. Can authors provide more evidence that improved value estimation leads to better decision-making?
-
Why is the V-A residual decomposition needed instead of a conventional action-conditioned critic?
-
On Reacher-v1 sparse, removing relabeling increases the success rate of HTB from 32.38% to 96.13% at . Why can removing subgoal relabeling produce such dramatic performance improvements?
5 comments
W3. Robustness relative to HIRO and the internal mixture control
Althought HTB achieves significant improvements on AntFall, its benefits are less convincing on other tasks. For example, on Reacher-v1 sparse without relabeling, HTB achieves 96.1% success compared with 95.4% for the multi-step-only baseline. On sparse Maze tasks, even the simpler baseline can outperform HTB. These raise concerns whether the components in HTB offer consistent benefits across different control problems.
Response. The main comparison is HTB's ability to learn the required subgoal value under a changing worker, evaluated against HIRO across tasks with different correction demands. On the complex Push/Fall tasks, the matched five-seed success rates improve from 0% to 14.00% and from 9.60% to 32.80%. The benefit also appears on Reacher: without correction, Reacher-v1 sparse improves from independently implemented HIRO's 76.00% to HTB's 96.13% over eight paired seeds.
The 95.38% multi-step-only result is a different comparison: it is the λ=1 endpoint of the remaining-time architecture, not the separate HIRO implementation. A task on which that endpoint is already strong does not require the short branch to create another large gain. For transparency, the four sparse fixed-configuration comparisons are:
| Sparse task | HIRO, no correction | Multi-step only, λ=1 | Full HTB, λ=.95 |
|---|---|---|---|
| AntReacher-v0 | 32.13 ± 32.14 | 54.25 ± 37.78 | 48.63 ± 41.81 |
| AntReacher-v1 | 76.00 ± 30.27 | 95.38 ± 7.93 | 96.13 ± 3.44 |
| AntMaze-v1 | 46.88 ± 33.05 | 35.00 ± 39.91 | 19.38 ± 31.78 |
| AntMazeW-v2 | 14.00 ± 11.90 | 14.75 ± 15.11 | 10.38 ± 11.41 |
All entries use no correction, eight seeds and the same 1M tail-window metric. The sparse Maze rows delimit the benefit of this fixed mixture. The clearer component evidence is on corrected Fall: multi-step only gives 13.60%, adding the full mixed update gives 32.80%, and removing next-step residual recursion gives 0%. We will keep the complete task results while organizing the contribution around the supervision capability and the settings where the corresponding practical components matter.
W4. Direct-Q controls and transfer to a reachability-constrained hierarchy
This paper does not provide sufficient comparisons with other related previous methods. Although HTB is compared against HIRO and several internal variants, it lacks a direct comparison with the method of intra-option learning [1] which incorporates remaining-time conditioning and the method of conventional action-conditioned critic. Conducting these comparisons is important to determine whether HTB's residual decomposition provides benefits beyond existing off-policy temporal abstraction techniques. Moreover, comparisons with more recent hierarchical RL methods are also limited.
Response. We have added a time-aware, parameter-matched direct-Q estimator with the same continuous subgoal interface, one-step/multi-step targets and correction setting. It is the direct control for the conventional action-conditioned evaluator in this task, rather than a transplant of a tabular discrete-option implementation. On AntFall at 10M, five-seed success is 0% versus HTB's 32.80% with correction, and 6.80% versus 24.40% without.
For external coverage, we completed Maze/MazeW HIRO controls, task-adapted official HRAC and flat TD3. HAC's path-independent hindsight-action rewards [4] and HiTS's timed-goal interface [5] address different settings. HRAC and HIGL improve reachability and exploration [6,7], whereas HTB changes how replay updates the required subgoal. These mechanisms can be combined. We therefore also inserted the HTB estimator/update into official-code HRAC, keeping its worker and reachability learner unchanged. The complete 1M transfer matrix uses three fresh paired training seeds and λ=.95:
| Task | Correction | HRAC | HRAC+HTB |
|---|---|---|---|
| AntReacher-v1 | on | 95.33 ± 2.52 | 97.33 ± 2.89 |
| AntReacher-v1 | off | 99.67 ± 0.58 | 98.67 ± 1.53 |
| AntMaze-v1 | on | 73.33 ± 7.64 | 100.00 ± 0.00 |
| AntMaze-v1 | off | 90.00 ± 5.00 | 81.67 ± 10.41 |
Values are mean±SD success (%) over the last twenty native evaluations. Corrected dense AntMaze improves from 73.33% to 100%, demonstrating a concrete benefit within a reachability-constrained hierarchy. The complete matrix keeps the task-dependent effects visible. This transfer test is more informative about the update mechanism than attributing every difference between two complete exploration architectures to non-stationarity correction.
W5. State sufficiency, goal transport and action coverage
In this paper, off-policy evaluation is based on restrictive assumptions. The theoretical formulation has strong assumptions, such as Markovian state representations, the known mapping from state to goal space, and sufficient action coverage. These assumptions may not hold in practical environments with partial observations.
Response. These conditions identify what an off-policy update can learn from the available data. HTB uses the stationary primitive environment law, a consistent goal-transport map and coverage of the actions required by the evaluated worker. The state-sufficiency and coverage requirements are the usual requirements for identifying an off-policy Bellman expectation; explicitly choosing the supervised subgoal does not remove them. The goal map is part of the goal-conditioned interface, as in HIRO-style relative subgoals.
The construction therefore gives direct supervision for a specified command on the identified action support. Remaining time removes the ambiguity about continuing versus invoking the next high-level decision. It does not assert that an arbitrary partial observation is Markov or that an unsupported action value can be inferred without data. We will place these conditions alongside the proposition so that the theoretical guarantee and the benchmark observation interface are clearly distinguished.
Q1. Why the one-step construction is not the complete practical algorithm
Could the authors clarify the fundamental methodological distinction between HTB and the combinations of previously established techniques, such as intra-option off-policy [1] and remaining-time conditioning [2]? In particular, which capability of specified-subgoal evaluation cannot be achieved through such a combination?
Response. Adding an execution clock to an intra-option-style primitive evaluator gives the one-step starting point of HTB, not the complete practical learning procedure. It does not by itself solve repeated propagation of intermediate estimation errors or poor coverage of newly selected primitive actions under a changing worker.
We test whether this one-step supervision is sufficient by setting λ=0 while retaining the HTB architecture. The controlled results are:
| Task | Reward | Correction | One-step only, λ=0 | Full HTB, λ=.95 |
|---|---|---|---|---|
| AntReacher-v1 | sparse | off | 6.50 ± 3.02 | 96.13 ± 3.44 |
| AntMaze-v1 | dense | off | 0.00 ± 0.00 | 73.75 ± 9.54 |
| AntFallPaper-v0 | dense | on | 0.00 ± 0.00 | 32.80 ± 21.10 |
Navigation uses eight seeds at 1M; AntFall uses five seeds at 10M. Values are success (%), mean±SD. These results explain the subsequent design choices: multi-step suffix returns replace learned intermediate continuations with observed rewards, nonnegative residuals filter upward corrections, and V−A shares information across actions whose sampling distribution changes. A suitably extended option framework can express the conditional evaluator; expressing it is different from obtaining a reliable continuous-control learner. HTB contributes the specific goal-supervision construction and the practical procedure supported by these controls.
Q2. Partial observations and the reason for mixed backups
Proposition 1 is dependent on assumptions, such as the Markovian state representations and sufficient primitive-action coverage. However, the paper acknowledges that the Push/Fall observations omit object states, which may violate thses assumptions. Could the authors clarify how these violations affect the validity of the proposed off-policy updates?
Response. Yes, omitting object state can violate the sufficient-state assumption. In that case, conditioning only on the critic's observation can mix hidden physical states in a policy-dependent way; Proposition 1 should not be applied literally to that observation as if it were a complete Markov state.
This is also a practical reason not to rely entirely on repeated one-step bootstrapping. Under function approximation and off-policy replay, an aliased or poorly generalized value can be propagated repeatedly through learned continuations. HTB trades this dependence against multi-step suffix returns: intermediate rewards are observed, and the learned bootstrap is deferred to the high-level boundary. The single-step path retains supervision for the specified subgoal, while the long path reduces the number of intermediate estimates. This mitigates an estimation problem; it does not turn partial observations into a fully observed process or supply a convergence theorem for the neural learner. We will make the distinction explicit.
Response to Reviewer AbCw
Response to Reviewer AbCw
Thank you for identifying the importance of specified-subgoal evaluation and the validity of the time-aware formulation. The method follows a problem-driven design chain: construct the required off-policy update, reduce repeated bootstrapping, share value information across changing primitive actions, and control upward target amplification. We clarify this chain below and add the requested direct-Q and end-to-end evidence.
W1. A problem-driven construction rather than independent ingredients
Response. HTB was constructed to solve one concrete problem: likelihood relabeling can remove direct supervision from the subgoal the manager needs to evaluate. Its design follows the obstacles encountered in making that update practical.
First, the remaining time is the time left in an already scheduled H-step subgoal invocation. At its boundary the task continues under a new manager decision; this differs from an episode time limit [2] and from choosing a timed subgoal as a high-level action [5]. Combined with primitive-action conditioning and goal transport, it gives an intra-option-style evaluator for continuously parameterized, changing workers [1].
Second, using that evaluator alone repeatedly propagates learned intermediate estimates. Multi-step suffix returns and their matching truncated relabeling replace intermediate estimates with observed rewards; nonnegative residuals filter upward corrections relative to a learned reference. Third, the changing worker shifts the action distribution. The dueling-inspired V−A structure [3] supplies a shared value offset rather than learning each newly queried action entirely through direct-Q extrapolation.
These choices have direct controls: one-step-only HTB (λ=0) reaches 6.50% on Reacher-v1 sparse and 0% on dense Maze-v1, versus 96.13% and 73.75% at λ=.95 without correction. On corrected AntFall, the matched time-aware direct-Q control reaches 0% versus 32.80% for full HTB. Together, these controls tie the targeted off-policy learning procedure to the original supervision problem and show why its practical estimation design matters.
W2. What the one-step theory establishes
Response. The theoretical guarantee concerns the off-policy one-step construction, not just the presence of a time input. Proposition 1 identifies the conditional Bellman expectation for a recorded primitive transition, independently of the collecting worker, when the specified subgoal is chosen consistently with the sampling conditions. Remaining time then provides the correct continuation or manager-boundary case. The appendix additionally establishes the signed-residual reconstruction for fixed policies and boundary values. This is the intra-option-style policy-evaluation foundation extended to the task, subgoal and primitive-action value granularities.
The multi-step branch uses HIRO-style approximate behavior correction, and the practical algorithm couples learned actors, target networks and nonlinear estimators. The paper does not identify their global convergence with the fixed-policy Bellman results. The residual architecture's role is supported both by its shared-offset identity and by the matched end-to-end controls: on AntFall, HTB versus direct Q is 32.80% versus 0% with correction and 24.40% versus 6.80% without. We will distinguish these operator-level guarantees and empirical learning results explicitly.