- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors propose a hierarchical framework for offline goal-conditioned RL in which a Temporally-extended Latent Action World Model (Te-LAWM) generates subgoals for a value-based model-free policy. Te-LAWM is a latent action model trained on multi-step (K-step) next-state transitions rather than single-step transitions. The authors evaluate Te-LAWM on OGBench navigation and manipulations tasks, and perform ablation and latent action space analysis to study the contribution of its individual components to performance.
Critical strengths
The authors design the baselines to isolate the key aspects of interest from confounding factors and show that in this setting Te-LAWM matches or outperforms the baselines, especially on long-horizon maze and goal-stitching tasks. The ablations (single-step, action-chunk and latent-action-chunk variants) provide evidence that multi-step autoregressive prediction and temporal action abstraction drive these gains.
Critical weaknesses
The shared-implementation protocol is a reasonable experimental design, but it does not entirely support the paper’s claims of superior performance over hierarchical model-free methods. The hierarchical baselines (HIQL and SD) have been adapted and the methods whose contribution lies in the value function (SHARSA and OTA) can not be represented under this protocol.
The motivation for the learned state-conditioned latent action prior is not sufficiently supported by the empirical results. In the background and related work, the authors distinguish Te-LAWM from prior multi-step LAMs partly on the basis that these methods use a fixed latent action prior. However, the Fixed Prior ablation performs largely on par with the full model, suggesting that the learned prior may not be necessary to achieve the reported performance. This raises questions about the significance of the proposed state-conditioned prior and whether it delivers the benefits motivating its introduction in Section 3.1. The visualisation of the learned state-conditioned prior in Figure 6 demonstrates that it captures interpretable, state-dependent structure in the latent action space. However, it remains unclear whether this property provides a practical advantage over a fixed prior.
Questions for authors
-
Please could the authors benchmark performance of Te-LAWM with the original implementations of HIQDL, SD, SHARSA and OTA?
-
Please could the authors clarify the significance of the learned state-conditioned prior?
8 comments
Response continued
Significance of the Learned Prior
We agree that the learned prior was not critical for the performance of our method on our evaluation suite in terms of success rate, as shown by the ablation study. The purpose of the ablation study was precisely to point to the components that were critical for performance, and it was effective in identifying Te-LAWM’s temporal action abstraction and multi-step autoregressive prediction as the main factors (as we highlighted in the paper).
The purpose of the latent action space analysis was partly to distinguish the effects of training with the learned prior. As you mentioned, Figure 6 demonstrates that it captures interpretable, state-dependent structure in the latent action space. We have added a latent action transfer analysis of the fixed prior variant in the new revision (Appendix Figure 11), which shows much lower transfer rates than that of training with the learned prior. The significance of the learned prior, as we see it, is in shaping the latent action space to be more interpretable and transferable across the state space, which have their respective benefits. Transferability for example, can lead to benefits in online adaptation and generalization, a study which is left for future work.
Thank you for the rebuttal. I appreciate the additional benchmarking against the original HIQL implementation. However, I am still not convinced that the manuscript conveys its main aim.
"The aim of our study is not to achieve state-of-the-art performance on a benchmark but rather to make empirically-backed scientific statements about the relationship between properties of methods, tasks and datasets to the method’s capabilities and performance."
If the aim of the study is "to make empirically-backed scientific statements about the relationship between properties of methods..." then this framing needs to be made much clearer throughout the manuscript. For instance in Section 4.1, it does not currently explain what each comparison isolates, or how the results “point to the factors that contribute to performance more precisely." Table 1 also shows that several models share algorithmic properties, so it is unclear how clean comparisons across models can be drawn. The section reads more as a performance benchmark against other methods, and the manuscript reinforces that reading with claims such as “We demonstrate superior performance of our framework compared to flat model-based and hierarchical model-free approaches on benchmark navigation and manipulation tasks” (L21).
I would encourage the authors to state the motivations behind the study more explicitly. In particular, they should explain why they compare different models: is it as a performance benchmark, as a way to identify which factors contribute to performance on this task suite, or both? What does the superior performance tell us about Te-LAWM?
Question: Would it be more effective to apply the approach you used for the learned prior (Figures 6, 11, 12) to other components of the Te-LAWM architecture? I understand the current model comparisons as an attempt to tell the same story, but controlled ablations of individual components seem like a more direct way to support claims about which properties drive performance.
Thank you for your thoughtful and constructive review. We appreciate your acknowledgement of the soundness of our experimental and baseline design, the informativeness of our ablation study and the strong performance of our method on long-horizon and stitching tasks.
Shared-implementation Protocol and Comparison to Hierarchical Baselines
The aim of our study is not to achieve state-of-the-art performance on a benchmark but rather to make empirically-backed scientific statements about the relationship between properties of methods, tasks and datasets to the method’s capabilities and performance. Our claims of superior performance over hierarchical model-free methods are within the context of our study (Table 1 summarizes the properties of interest to our study). Since there may be many sources of variation in results, the shared implementation protocol allows us to make a clean comparison between representative approaches and point to the factors that contribute to performance more precisely. Comparing TD-MPC to Te-LAWM contrasts planning with world models in different temporal timescales, supporting our suggestion in the introduction that world models are better fit for high-level planning in terms of both performance and test-time compute (fewer forward passes). Comparing HIQL to Te-LAWM contrasts model-free subgoal generation with multi-step model-based subgoal planning, showing that the latter enables “subgoal stitching” which leads to better performance in long-horizon tasks that require stitching.
HIQL: We implemented HIQL in our pytorch codebase based on the OGBench reference JAX implementation and tested it on representative tasks. The results compared to the HIQL modification in our paper are presented in the following table:
HIQL (OGBench) achieves similar results to the ones in the OGBench paper, where HIQL struggles in pointmaze and state-based manipulation tasks. Our modified implementation of HIQL is a stronger baseline which performs better on pointmaze and significantly better on manipulation tasks. We hypothesize the worse performance on pointmaze is caused by the goal representation which increases the dimensionality of the goal space to 10 compared to the 2-dimensional pointmaze state space.
Regarding the state-based manipulation environments we cite the OGBench paper as reference (Section 9, page 14, bottom):
SD: SD is not based on a specific algorithm but on a family of methods, and thus does not have an original implementation. Its role in our study is to represent diffusion-based goal-conditioned world models.
SHARSA and OTA: SHARSA and OTA are equivalent to SD and HIQL respectively with separate high-level value functions for the high-level policies. In SHARSA’s case, the high-level value function training target uses n-step returns while in OTA it omits the n-step returns and directly bootstraps to an n-step target. These modifications are orthogonal to our study in the sense that our focus is on the subgoal generation mechanism (policy vs. planning with a world model). They are also orthogonal to the contribution of our method in the sense that Te-LAWM could also use the SHARSA / OTA value functions for high-level planning. We therefore kept a single value function for all methods and did not compare directly to SHARSA and OTA. We have added a sentence clarifying this in the “Hierarchical Goal-conditioned Reinforcement Learning” paragraph of the related work section.