acceptodds
Under review as a conference paper at ICLR 2027

Hierarchical Reinforcement Learning with Temporally-extended Latent Action World Models

15 views

Abstract

World Models enable counterfactual reasoning and planning but struggle in long horizon tasks because prediction errors compound over the horizon. Temporal abstraction can mitigate these errors by simply requiring fewer consecutive predictions for a given horizon. We propose a Temporally-extended Latent Action World Model (Te-LAWM) and present a hierarchical model-based framework for goal-conditioned reinforcement learning that leverages it to generate subgoals for a value-based model-free policy. Our temporally-extended latent action model learns a multi-step action abstraction, a multi-step dynamics model conditioned on these abstract actions (the Te-LAWM), and a prior policy over the actions within a single objective. Subgoal planning is done by rolling out the prior policy within the Te-LAWM. We demonstrate superior performance of our framework compared to flat model-based and hierarchical model-free approaches on benchmark navigation and manipulation tasks. Ablations show that autoregressive subgoal generation allows the agent to stitch together experience from different trajectories, and that temporal action abstraction is essential for scaling world model predictions to long horizons and large action spaces.

Primary area:
reinforcement learning
Keywords:
Deep Reinforcement Learning, Goal-conditioned Reinforcement Learning, World Models, Latent Action Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_3BmvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a hierarchical framework for offline goal-conditioned RL in which a Temporally-extended Latent Action World Model (Te-LAWM) generates subgoals for a value-based model-free policy. Te-LAWM is a latent action model trained on multi-step (K-step) next-state transitions rather than single-step transitions. The authors evaluate Te-LAWM on OGBench navigation and manipulations tasks, and perform ablation and latent action space analysis to study the contribution of its individual components to performance.

Critical strengths

The authors design the baselines to isolate the key aspects of interest from confounding factors and show that in this setting Te-LAWM matches or outperforms the baselines, especially on long-horizon maze and goal-stitching tasks. The ablations (single-step, action-chunk and latent-action-chunk variants) provide evidence that multi-step autoregressive prediction and temporal action abstraction drive these gains.

Critical weaknesses

The shared-implementation protocol is a reasonable experimental design, but it does not entirely support the paper’s claims of superior performance over hierarchical model-free methods. The hierarchical baselines (HIQL and SD) have been adapted and the methods whose contribution lies in the value function (SHARSA and OTA) can not be represented under this protocol.

The motivation for the learned state-conditioned latent action prior is not sufficiently supported by the empirical results. In the background and related work, the authors distinguish Te-LAWM from prior multi-step LAMs partly on the basis that these methods use a fixed latent action prior. However, the Fixed Prior ablation performs largely on par with the full model, suggesting that the learned prior may not be necessary to achieve the reported performance. This raises questions about the significance of the proposed state-conditioned prior and whether it delivers the benefits motivating its introduction in Section 3.1. The visualisation of the learned state-conditioned prior in Figure 6 demonstrates that it captures interpretable, state-dependent structure in the latent action space. However, it remains unclear whether this property provides a practical advantage over a fixed prior.

Questions for authors
  1. Please could the authors benchmark performance of Te-LAWM with the original implementations of HIQDL, SD, SHARSA and OTA?

  2. Please could the authors clarify the significance of the learned state-conditioned prior?

8 comments

Thank you for your thoughtful and constructive review. We appreciate your acknowledgement of the soundness of our experimental and baseline design, the informativeness of our ablation study and the strong performance of our method on long-horizon and stitching tasks.

Shared-implementation Protocol and Comparison to Hierarchical Baselines

The aim of our study is not to achieve state-of-the-art performance on a benchmark but rather to make empirically-backed scientific statements about the relationship between properties of methods, tasks and datasets to the method’s capabilities and performance. Our claims of superior performance over hierarchical model-free methods are within the context of our study (Table 1 summarizes the properties of interest to our study). Since there may be many sources of variation in results, the shared implementation protocol allows us to make a clean comparison between representative approaches and point to the factors that contribute to performance more precisely. Comparing TD-MPC to Te-LAWM contrasts planning with world models in different temporal timescales, supporting our suggestion in the introduction that world models are better fit for high-level planning in terms of both performance and test-time compute (fewer forward passes). Comparing HIQL to Te-LAWM contrasts model-free subgoal generation with multi-step model-based subgoal planning, showing that the latter enables “subgoal stitching” which leads to better performance in long-horizon tasks that require stitching.

Could the authors benchmark performance of Te-LAWM with the original implementations of HIQL, SD, SHARSA and OTA?

HIQL: We implemented HIQL in our pytorch codebase based on the OGBench reference JAX implementation and tested it on representative tasks. The results compared to the HIQL modification in our paper are presented in the following table:

TaskHIQL (our paper)HIQL (OGBench)
pointmaze-medium-navigate97 ± 283 ± 8
antmaze-large-stitch79 ± 681 ± 6
cube-double-play60 ± 23 ± 2
cube-double-noisy89 ± 41 ± 0

HIQL (OGBench) achieves similar results to the ones in the OGBench paper, where HIQL struggles in pointmaze and state-based manipulation tasks. Our modified implementation of HIQL is a stronger baseline which performs better on pointmaze and significantly better on manipulation tasks. We hypothesize the worse performance on pointmaze is caused by the goal representation which increases the dimensionality of the goal space to 10 compared to the 2-dimensional pointmaze state space.

Regarding the state-based manipulation environments we cite the OGBench paper as reference (Section 9, page 14, bottom):

“Somewhat weirdly, HIQL struggles much more with state-based observations than pixel-based observations on manipulation tasks. We suspect this is related to subgoal representations, given that HIQL uses an additional learning signal from the policy loss to further train subgoal representations only in pixel-based environments (which we found does not help in state-based environments).”

SD: SD is not based on a specific algorithm but on a family of methods, and thus does not have an original implementation. Its role in our study is to represent diffusion-based goal-conditioned world models.

SHARSA and OTA: SHARSA and OTA are equivalent to SD and HIQL respectively with separate high-level value functions for the high-level policies. In SHARSA’s case, the high-level value function training target uses n-step returns while in OTA it omits the n-step returns and directly bootstraps to an n-step target. These modifications are orthogonal to our study in the sense that our focus is on the subgoal generation mechanism (policy vs. planning with a world model). They are also orthogonal to the contribution of our method in the sense that Te-LAWM could also use the SHARSA / OTA value functions for high-level planning. We therefore kept a single value function for all methods and did not compare directly to SHARSA and OTA. We have added a sentence clarifying this in the “Hierarchical Goal-conditioned Reinforcement Learning” paragraph of the related work section.

AuthorsOpenReview ↗

Response continued

Significance of the Learned Prior

We agree that the learned prior was not critical for the performance of our method on our evaluation suite in terms of success rate, as shown by the ablation study. The purpose of the ablation study was precisely to point to the components that were critical for performance, and it was effective in identifying Te-LAWM’s temporal action abstraction and multi-step autoregressive prediction as the main factors (as we highlighted in the paper).

The purpose of the latent action space analysis was partly to distinguish the effects of training with the learned prior. As you mentioned, Figure 6 demonstrates that it captures interpretable, state-dependent structure in the latent action space. We have added a latent action transfer analysis of the fixed prior variant in the new revision (Appendix Figure 11), which shows much lower transfer rates than that of training with the learned prior. The significance of the learned prior, as we see it, is in shaping the latent action space to be more interpretable and transferable across the state space, which have their respective benefits. Transferability for example, can lead to benefits in online adaptation and generalization, a study which is left for future work.

Thank you for the rebuttal. I appreciate the additional benchmarking against the original HIQL implementation. However, I am still not convinced that the manuscript conveys its main aim.

"The aim of our study is not to achieve state-of-the-art performance on a benchmark but rather to make empirically-backed scientific statements about the relationship between properties of methods, tasks and datasets to the method’s capabilities and performance."

If the aim of the study is "to make empirically-backed scientific statements about the relationship between properties of methods..." then this framing needs to be made much clearer throughout the manuscript. For instance in Section 4.1, it does not currently explain what each comparison isolates, or how the results “point to the factors that contribute to performance more precisely." Table 1 also shows that several models share algorithmic properties, so it is unclear how clean comparisons across models can be drawn. The section reads more as a performance benchmark against other methods, and the manuscript reinforces that reading with claims such as “We demonstrate superior performance of our framework compared to flat model-based and hierarchical model-free approaches on benchmark navigation and manipulation tasks” (L21).

I would encourage the authors to state the motivations behind the study more explicitly. In particular, they should explain why they compare different models: is it as a performance benchmark, as a way to identify which factors contribute to performance on this task suite, or both? What does the superior performance tell us about Te-LAWM?

Question: Would it be more effective to apply the approach you used for the learned prior (Figures 6, 11, 12) to other components of the Te-LAWM architecture? I understand the current model comparisons as an attempt to tell the same story, but controlled ablations of individual components seem like a more direct way to support claims about which properties drive performance.

Reviewer_HZLfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Te-LAWM is an offline goal-conditioned RL method that uses a hierarchy for planning and control. The authors learn a latent action model from state pairs a fixed number of time steps apart, along with a prior over latent actions conditioned on the current state. During deployment, rollouts in this model produce possible future states. A goal-conditioned value function scores them, and the selected state becomes a subgoal for a low-level policy trained separately. By making predictions over longer time intervals, the planner needs fewer model steps to look further ahead and can combine behaviors found in different offline trajectories. The experiments use OGBench navigation and manipulation tasks, including tests with visual inputs and ablations of the architecture. Several difficult tasks requiring trajectory stitching show large improvements. What remains less clear is how much these results depend on the amount and type of data used for training.

Critical strengths

Predicting several steps at once seems a sensible way to reduce the number of autoregressive model calls in a long plan. The state-conditioned prior also helps keep the proposed transitions close to what is represented in the dataset. Together, these choices give a reasonable way to use temporal abstraction when choosing subgoals, and the component ablations help assess what the abstraction and prior each contribute. The navigation results are encouraging. On antmaze-large-stitch, for example, success reaches 91%, compared with 79% for HIQL. Testing manipulation as well as navigation, and including visual observations, makes the empirical study broader.

Critical weaknesses

I am most concerned about how little we know about the method's dependence on the training dataset. Both the world model and the latent action prior are fitted to offline experience. If relevant transitions are poorly covered, the planner may propose a subgoal based on an inaccurate prediction. The value function has a related problem: it is learned from offline data too, so a high score for a predicted state need not be reliable. Errors in proposing candidates and choosing among them could therefore arise from the same gaps in the data. It is not clear how often this happens.

Section 4.1 does discuss coverage, and the play/noisy experiments offer some evidence on this issue. Still, those comparisons do not tell us separately what happens when there are fewer trajectories, worse behavior, or fewer examples of particular local transitions. Most of Section 4.2 changes parts of the model while leaving the data question aside. I would have liked a controlled study that follows these data changes through to model prediction errors, whether the policy actually reaches the selected subgoals, and whether the value function ranks candidates correctly.

Questions for authors
  1. Do the authors have results that separate the amount of training data from behavior quality and transition coverage, beyond the play/noisy variants? For the cube tasks in particular, which local manipulation behaviors are present in the data when full trajectories reaching an evaluation goal are missing?

  2. Could the method and baselines be compared using the same subsets of complete trajectories, such as 25%, 50%, and 100% of the original data? It would also help to hold dataset size constant while changing behavior quality or the coverage of local transitions. I am asking about sensitivity where the task remains feasible from the available data, rather than expecting the method to recover behaviors for which it has no examples.

  3. When the data is changed in these ways, how do model error, subgoal success, and the agreement of value rankings with actual outcomes change? Do they get worse together? One useful check would keep the low-level policy and value function fixed and change only the data used to fit the world model. This would make the world model's role easier to assess, and the findings could change my recommendation.

5 comments

We appreciate your in-depth review, questions and suggestions. Thank you for highlighting the effectiveness of our Te-LAWM design choices and their ablation study, our encouraging results on long-horizon tasks that require stitching and for acknowledging the breadth of our empirical study.

Dataset Size Study

We agree that the effects of dataset properties on downstream performance is an important and interesting aspect to study. Our experiments covered the effects of the length and overlap of dataset trajectories (maze navigate vs. stitch) and behavior quality and transition coverage (manipulation play vs. noisy). Based on your suggestions and questions, we have conducted additional experiments that vary the dataset size in representative navigate, stitch, play and noisy dataset settings.

We have added a section in the new revision, Appendix B.4, where we study the effects of the dataset size on the performance of the hierarchical methods. Each method is trained on 100%, 50%, or 25% of the original dataset trajectories. We evaluate varying the data budget only for the high level, only for the IQL low level and for both. For your convenience we summarize our main findings here and refer you to the paper for the full results and analysis:

  • In navigation, Te-LAWM is robust to the data budget, maintaining the highest scores on maze tasks across all settings. Te-LAWM displays little to no degradation when trained on the full data budget, even with an IQL trained on 25% which degrades significantly on its own. We find this result promising as it hints to the potential of Te-LAWM to leverage more abundant action-free data to to make up for less performant low-level policies through subgoal generation.
  • In manipulation, the IQL low level is the main bottleneck in performance.
  • Decreasing only the high level budget hurts performance in cube-double-play but not in noisy. This suggests that dataset state coverage may be more important than behavior quality for learning high-level subgoal generators.
  • HIQL is much less sample efficient than Te-LAWM, degrading significantly with lower data budgets.

To summarize, we have studied varying the dataset size in representative environments and dataset variants that span different types of transition coverage and behavior quality. We believe this study addresses your questions and concerns. Please let us know if there are any questions our study did not cover.

Questions and Answers

When the data is changed in these ways, how do model error, subgoal success, and the agreement of value rankings with actual outcomes change?

Subgoal success: Our planning procedure produces a subgoal that guides the low-level policy for a fixed maximum number of timesteps and then replans in an MPC fashion for robust execution. The policy therefore makes progress towards those subgoals but often does not reach them, depending on how far away they are set. We attempted to measure progress using various heuristics but we believe these do not add insight beyond the overall performance in the dataset study. By observing rollouts we can qualitatively say that in navigation, the low-level policy mostly makes progress towards the generated subgoals. When performance degrades, it is mostly because the subgoals do not make progress towards the goal. In manipulation, the policy makes progress towards the subgoals on the full data budget but when it is trained on less data its ability to follow subgoals quickly degrades.

Value ranking correctness: This is not straightforward to measure because it requires a ground truth global distance measure. In environments with simple state spaces such as pointmaze we can measure this but in others it is difficult. In antmaze, for example, velocity also affects the value of an agent’s state. By observing rollouts and visualizing subgoal candidates as well as the chosen subgoal (see Figure 1 for example visualizations), we can qualitatively say that the value function ranks subgoals reasonably in all of the data budgets and settings.

Model Prediction Error: We measure the Te-LAWM reconstruction error on the validation datasets provided by OGBench. The error increases with smaller training dataset size, although as we see from the performance results, this does not necessarily degrade performance. We believe the reason is that the prediction of the model does not need to be completely accurate in order to constitute a useful subgoal. In our high-level planning procedure, the latent actions are generated from a learned prior, in which case there is no ground-truth future state to compare to. This is in contrast to a low-level world model where one could compare model predictions to executing the low-level actions in the simulator.

AuthorsOpenReview ↗

Response continued

Which local manipulation behaviors are present in the cube data when full trajectories reaching an evaluation goal are missing?

The cube datasets are composed of pick-and-place operations of random cubes to random positions on the table. Due to this and the size of the state space, the probability of the dataset containing a trajectory that takes the cubes from their exact evaluation initial positions to the goal positions is extremely low, especially for increasing numbers of cubes.

Thank you for the response. but my concerns are still not resolved. The new results show that the method is sensitive to the data used for the world model. With less of it, success drops by 22% on antmaze-large-stitch and by 25% on humanoidmaze-large-navigate and cube-double-play, while the low-level policy and value function are unchanged. On cube-double-play the method ends up below flat IQL, so the world model, not only the low-level policy, limits performance there. This does not match the claim that Te-LAWM is robust to the data budget. The claim that coverage matters more than behavior quality is not tested either, since the play and noisy datasets differ in both. The rebuttal also mentions that the reconstruction error was measured and that value ranking can be checked on pointmaze, but reports neither.

My two main concerns therefore remain. First, the method is sensitive to the training data, especially on stitching and manipulation tasks. Second, the overall design is fragile. The world model, the prior, and the value function are all learned from the same offline data, so a gap in the data can affect both how subgoals are proposed and how they are selected, and nothing in the framework detects or corrects such errors. I would like to keep my score.

Reviewer_ExwhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a Temporally-extended Latent Action World Model (Te-LAWM) for subgoal planning in offline goal-conditioned reinforcement learning. An inverse dynamics model infers a latent action distribution from the current state and a multi-step future state. A forward dynamics model predicts the future state from the current state and latent action. These components are trained jointly with a state-conditioned prior, and the forward dynamics model serves as Te-LAWM.

At test time, latent actions are sampled from the prior, and Te-LAWM generates subgoal candidates through autoregressive prediction. A separately trained IQL value function scores the candidates, and a goal-conditioned low-level policy executes actions toward the selected subgoal.

The method is evaluated on OGBench navigation and manipulation tasks, with substantial improvements over the evaluated baselines on large-maze stitching tasks. Ablations examine temporal action abstraction, the abstraction learning procedure, autoregressive subgoal prediction, and the learned prior. Additional analyses investigate whether latent actions have similar effects across states, while the appendix provides results on visual navigation tasks.

Critical strengths
  1. Substantial performance improvements on large-maze stitching tasks. Te-LAWM achieves a success rate of 84 ± 5 on pointmaze-large-stitch, compared with 35 ± 7 for the strongest evaluated baseline. On antmaze-large-stitch and humanoidmaze-large-stitch, it achieves 91 ± 2 and 50 ± 3, respectively, exceeding the highest baseline means of 79 ± 6 and 31 ± 3. These results support the usefulness of high-level subgoal planning with Te-LAWM on tasks that require stitching different behavior segments.

  2. Joint training of the latent action model and a modular hierarchical framework. The inverse dynamics model, forward dynamics model, and state-conditioned prior are trained jointly with a single objective. This simplifies training relative to the Latent Action Chunk World Model (LACWM), which learns latent representations of action chunks and then trains a separate forward dynamics model. The high-level and low-level components are trained independently and combined at test time, avoiding the need to train all components jointly.

  3. Latent action transfer experiments help characterize the latent action space. The paper applies latent actions inferred from different transitions to the same starting state and compares the decoded transitions. Latent actions transferred from transitions involving the same event reproduce the grasp event in 94% of decoded transitions, compared with 12% for those transferred from different events. Although these results concern decoded transitions rather than low-level policy execution, they support the interpretation that latent actions can be reused and have similar effects across different states.

Critical weaknesses
  1. Limited comparison with closely related prior work. The paper explains differences from prior work, including HiGoC, in terms of the state-conditioned prior and planning procedure. However, without a direct comparison with HiGoC, it is difficult to assess the benefits of these differences in success rate and planning cost. The improvements over the evaluated baselines are encouraging, but further comparison would help establish the contribution relative to this closely related method.

  2. Some ablations change both the planning depth and how far into the future the model predicts. On pointmaze-large-stitch, Te-LAWM uses K = 100 and d = 2, whereas Single-step uses K = 100 and d = 1. The predictions therefore extend nominally 200 and 100 environment steps into the future, respectively. This comparison does not separate the benefit of autoregressive subgoal prediction from that of predicting further into the future. The humanoidmaze experiments match K × d and still show improvements, providing supporting evidence for autoregressive prediction. Nevertheless, the reasons for these different comparison settings and their implications need clarification.

  3. The subgoal selection procedure is described inconsistently. The main text and Algorithm 1 score subgoal candidates using V(g̃, g), whereas Appendix A specifies V(sₜ, g̃). Furthermore, Algorithm 1 selects from the final predicted states, while the experimental settings include cases that use the first waypoint. These differences make it difficult to reproduce the subgoal selection procedure and understand how subsequent autoregressive predictions contribute to selecting the subgoal.

Questions for authors
  1. What benefits does the proposed method provide over HiGoC? Given the differences in the prior and planning procedure, could the authors compare success rates and planning costs on representative tasks? If a direct comparison is difficult, please explain the relevant constraints and clarify which advantages over HiGoC are supported by the current experiments.

  2. How was the subgoal distance K selected for Single-step? K × d is matched on some tasks but differs on pointmaze-large-stitch. Does the improvement persist when Single-step uses K = 200 and d = 1 on this task? Please also specify the number of subgoal candidates and the subgoal selection procedure used in this comparison.

  3. What subgoal scoring and selection rules are used in the implementation? Are candidates scored using V(g̃, g) or V(sₜ, g̃)? When the first waypoint is selected as the subgoal, how do subsequent predicted states affect its selection? Please reconcile the descriptions in the main text, appendix, and Algorithm 1.

  4. How do the modifications to HIQL affect its performance? The motivation for sharing value functions and low-level policies is understandable. However, several changes are applied together, including training the low-level policy on the goal distribution rather than the subgoal distribution and removing goal representation networks. Could the authors compare with the OGBench reference HIQL implementation under the same evaluation protocol on representative tasks, or provide other results assessing these changes?

4 comments

Thank you for your detailed and thoughtful review and questions. We appreciate your acknowledgement of the simplicity of Te-LAWM's training objective, the benefits of the modularity of our overall framework, the efficacy of our subgoal planning approach in stitching compared to baselines and the informativeness of our latent action transfer analysis.

1. Comparison to HiGoC

Our problem setting is offline goal-conditioned RL while HiGoC is designed for standard offline RL. This makes some aspects of HiGoC not directly comparable to Te-LAWM. Nevertheless, we have summarized the core method differences below and performed additional experiments to clarify the advantages of our design choices compared to those of HiGoC which are comparable.

Modularity: HiGoC trains the low-level policy on goals generated by the VAE which creates a dependency between the hierarchies during training. Our framework is modular and allows training the levels independently.

State-conditioned Prior: HiGoC trains their VAE with a fixed Normal prior while we train with a learned state-conditioned prior. This design choice has been ablated in Section 4.2 and further analyzed in Section 4.3. Figure 6 demonstrates that the learned prior captures interpretable, state-dependent structure in the latent action space. We have added a latent action transfer analysis of the fixed prior variant in the new revision (Appendix Figure 11), which shows much lower transfer rates than that of training with the learned prior. The learned prior thus provides advantages in terms of interpretability and transferability of the latent actions.

Planning Procedure: HiGoC plans with the cross-entropy method over the VAE latent space and adds a penalty on the magnitude of the latent actions in the optimization objective to keep them close to the Normal prior. We simply roll out our learned latent action policy within the Te-LAWM to produce subgoal candidates without iterative refinement. Our approach generates in-distribution candidates by design and does not require additional constraints during planning. We ablate this design choice by running MPPI optimization (with the latent action magnitude penalty in the objective) on top of the fixed-prior Te-LAWM ablation, which is the closest setting to the HiGoC planning procedure. We evaluate this variant on representative tasks and report the results in the following table:

TaskFixed Prior + MPPIFixed Prior
antmaze-medium-navigate79 ± 1088 ± 6
antmaze-large-stitch73 ± 883 ± 3
humanoidmaze-large-navigate78 ± 980 ± 7
antsoccer-medium-navigate32 ± 534 ± 3
scene-play66 ± 373 ± 2
cube-double-play81 ± 284 ± 1

Planning with MPPI performs worse or on-par with simply sampling from the fixed prior, while being more computationally expensive. We believe this may be a result of MPPI exploiting inaccuracies in the learned value function.

To summarize, our framework is modular and simpler in terms of training and test-time execution. The learned prior has benefits in shaping the latent actions to be more interpretable and transferable, and our planning procedure is simple while keeping subgoals within the support of the dataset, advantages established by our experiments and analysis.

2. Single-step Ablation Subgoal Distance K Selection

The subgoal distance was chosen such that K×dK \times d is matched, in order to separate the benefit of autoregressive subgoal prediction from that of the prediction horizon, as you mentioned. pointmaze-large-stitch was the only exception since dataset trajectories contain only 200200 transitions and thus do not allow training with K=200K=200 (for K=199K=199 each trajectory would only have 1 valid training (st,st+K)(s_t, s_{t+K}) pair).

We have run additional experiments for the single-step Te-LAWM ablation with K=150,180,199K=150, 180, 199. This brings the prediction horizon closer to that of Te-LAWM while significantly decreasing the effective amount of training data. The success rates are 46±546 \pm 5, 67±1167 \pm 11, and 41±1341 \pm 13 respectively. K=150,199K=150, 199 performs equivalently to K=100K=100. K=180K=180 performs better and seems to be a good balance but still significantly underperforms the standard Te-LAWM, consistent with the results on the other large-stitch maze tasks.

The number of subgoal candidates and subgoal selection procedure is unchanged from the main results and detailed in Appendix Table 5. For Single-step, naturally, the first waypoint from the best trajectory is selected because there is only one prediction to choose from.

AuthorsOpenReview ↗

Response continued

3. Subgoal Scoring and Selection Rules

We appreciate your attention to detail and thank you for pointing us to this inconsistency. Appendix A had a mistake in the notation which understandably caused the confusion. We have now made sure the descriptions are correct and consistent across the entire paper, which you can view in the new revision.

We include a clarification here for your convenience: We always score trajectories using the final predicted state. The subgoal is then chosen to be either the first or last predicted state in that trajectory. In this setup, the final predicted state always informs the subgoal selection. So even when the first state is chosen, it is selected based on the K×dK \times d lookahead. Selecting the first state as the subgoal is analogous to executing the first action from an optimized action sequence in an MPC fashion.

4. Original HIQL Version

We implemented HIQL in our pytorch codebase based on the OGBench reference JAX implementation and tested it on representative tasks in our evaluation setup. The results compared to the HIQL modification in our paper are presented in the following table:

TaskHIQL (our paper)HIQL (OGBench)
pointmaze-medium-navigate97 ± 283 ± 8
antmaze-large-stitch79 ± 681 ± 6
cube-double-play60 ± 23 ± 2
cube-double-noisy89 ± 41 ± 0

HIQL (OGBench) achieves similar results to the ones in the OGBench paper, where HIQL struggles in pointmaze and state-based manipulation tasks. Our modified implementation of HIQL performs better on pointmaze and significantly better on manipulation tasks. We hypothesize the worse performance on pointmaze is caused by the goal representation which increases the dimensionality of the goal space to 10 compared to the 2-dimensional pointmaze state space.

Regarding the state-based manipulation environments we cite the OGBench paper as reference (Section 9, page 14, bottom):

"Somewhat weirdly, HIQL struggles much more with state-based observations than pixel-based observations on manipulation tasks. We suspect this is related to subgoal representations, given that HIQL uses an additional learning signal from the policy loss to further train subgoal representations only in pixel-based environments (which we found does not help in state-based environments).”

Regarding the low-level policy’s training goal distribution, we found that training on the subgoal distribution performed similarly or worse than the goal distribution.

To summarize, our HIQL implementation is a stronger baseline in our setting and additionally enables a cleaner comparison to our method and other baselines.

Thank you for the detailed response. Could you elaborate on why HiGoC cannot be extended to a goal-conditioned version? If such an extension is feasible, I would like to see a comparison between Te-LAWM and the extended version of HiGoC. I am asking again because it appears to be one of the most closely related methods, and I believe this comparison is necessary to understand the main contributions of your work.

Thank you for your continued engagement, we highly appreciate it.

Our response above including a comparison to “Fixed Prior + MPPI” is an extension of the relevant aspects of HiGoC to a goal-conditioned version, which we should have made clearer.

A main component of HiGoC that is not relevant to our goal-conditioned setting is a learned auxiliary goal-conditioned reward, which is essentially another value function. They optimize for this reward along with the environment (non-goal-conditioned) reward. In our goal-conditioned setting we simply have a single sparse goal-conditioned reward.

We could not find code for HiGoC, we believe that the authors have not open-sourced it. The paper does not have an Appendix or in-depth implementation details either. We believe the “Fixed Prior + MPPI” variant captures the key aspects of HiGoC, implemented within our codebase, and provides a solid and clean comparison to Te-LAWM.

Beyond the technical differences, we believe our contribution lies also in our framing and empirical study. Our study focuses on world models, their role in sequential decision-making, their limitations and capabilities compared to model-free approaches. We propose them for high-level planning in a hierarchical system and study the temporally-extended world model design space.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Thank you for the time and effort you have invested in reading and reviewing our work. We highly appreciate your detailed and constructive feedback and questions.

Thank you for acknowledging the simplicity and modularity of our framework (reviewer Exwh), the careful baseline design (reviewer 3Bmv), the breadth of our experiments (reviewer HZLf), the informativeness of our ablations (reviewers HZLf, 3Bmv) and latent action analysis (reviewer Exwh), and our strong results on long horizon tasks that require stitching (reviewers Exwh, HZLf, 3Bmv).

We have uploaded a new revision based on your reviews and have responded individually to each reviewer. We summarize our main additions, modifications and experiments here:

Dataset Size Study: Reviewer HZLf requested experiments on how training dataset size affects the methods' performance under different dataset properties. We have conducted a study varying the training dataset size on representative tasks, varying the data budget only for the high level, only for the IQL low level and for both. We found that Te-LAWM is robust to these variations on the navigation tasks, that it is significantly more sample efficient than HIQL, and that dataset state coverage may be more important than behavior quality for learning high-level subgoal generators. See Appendix B.4 for full results and analysis

Learned Prior Contribution: Reviewers Exwh and 3Bmv requested clarification on the contribution of the learned state-conditioned prior. We have added a latent action transfer analysis of the fixed prior variant to Appendix Figure 11, which shows much lower transfer rates than that of training with the learned prior. This together with our existing results point to the learned prior’s contribution to the interpretability and transferability of the latent actions.

Subgoal Selection Procedure Clarification: Reviewer Exwh requested a clarification due to an inconsistency in our descriptions. We have fixed a notation mistake in the Appendix description and made sure the procedure is described correctly and consistently across the paper.

Comparison to Original Version of HIQL: Reviewers Exwh and 3Bmv requested a comparison of our modified version of HIQL to the original version on representative tasks. We show that our modification performs better while making it easier to compare to other methods in our study (see table and further details in responses to the individual reviewers.)

Comparison to HiGoC: Reviewer Exwh requested a comparison to HiGoC. While not all aspects of the method are directly comparable to ours, we run additional experiments using HiGoC’s planning procedure and show that it performs worse or on par while being more complex and computationally expensive.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.