- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper investigates adaptive computation in a hierarchical reinforcement learning. They do it through proposing a method called HOMER, which combines two existing methods HiQL (a hierarchical offline RL method) and PonderNet (a method from supervised learning providing an adaptive computation mechanism). HOMER allows the manager and worker to independently learn how much computation to use. HOMER achieves a comparable or better performance than the non-adaptive baselines, while using less recurrent steps. They propose interesting analysis: how do the worker and manager networks behave under increasing time-horizon and of reusing previous computations.
Critical strengths
I find the idea to scale computations at different hierarchy levels very interesting in itself. In this context, the analysis in section 6.2.2, which investigates the amount of computation needed when the horizon increases is the strongest, and best to my knowledge most novel part of this paper.
Additionally, the paper is very well written.
Critical weaknesses
The gain in performance within a given computational budget doesn't look that impressive in most environments, when compared to the adaptive version.
The practical significance seems quite modest. OGBench tasks are on the simpler side as robotics goes and the network sizes are quite small. In this type of task, adaptive computations don't necessarily matter. Even though it is important to demonstrate this in simpler tasks first, this has already been done (and this paper cites that work). And even with that, based on Appendix A.1 it's unclear to me whether this method actually saves on computational cost, see Questions.
The paper doesn't cite previous work that did adaptive computation in forms such as early exits or mixture of experts. In particular some of them also looked into RL and reusing computations [1].
Questions for authors
- Do I understand correctly that the current implementation (as described in Appendix A.1) requires computing all the steps of the RNN each time? If so, could the authors demonstrate that this is not necessary?
- 307-309 What does this oracle representation consist of? Why is learning the representation a problem?
- 346-348 Why is setting the optimal hyperparameter for a non-meta-reasoning agent harder than for a meta-reasoning one?
- 374-375 Could the authors expand on why does the number of k-step subgoals in the dataset matter here?
3 comments
Dear authors,
Thanks for the clarifications. While I believe that the changes strengthen the work, my main concern about wider applicability remains unaddressed for now.
Dear reviewer,
We appreciate your timely response and are glad that the changes have strengthened the work!
Even though it is important to demonstrate this in simpler tasks first, this has already been done (and this paper cites that work).
Regarding the wider applicability based on your initial comments, respectfully, there is perhaps a misunderstanding. Could you please clarify the above quote?
To the best of our knowledge, this claim is not true. Studying adaptive computation in RNNs for hierarchical RL agents in the offline GCRL setting has not been explored before. Based on the components of our work, the closely related work that has been done before is:
- PonderNet [1] which used adaptive computation for RNNs but on supervised learning tasks only, not for RL nor HRL.
- IRU [2], which showed non-adaptive RNN compute agents could help for flat RL agents on offline GCRL. That is, they did not focus on adaptive computation nor HRL. Our FOAMER results are adaptive compute for flat RL, but that is just simply for benchmarking performance, not the focus of the paper.
Given these points, we believe that our empirical analysis on the existing benchmark is an appropriate step for this novel direction, especially given that we have still been able to demonstrate interesting insights and benefits (entropy correlation, caching, comparable success at much fewer FLOPs).
Please let us know if we can clarify anything else. Thank you.
[1] PonderNet: Learning to Ponder. Banino et al. 2021.
[2] On the Role of Computation in Reinforcement Learning. Ghugare et al. 2026.
Thank you to the reviewer for their feedback and effort.
We are glad you find our research problem interesting, our empirical analysis strong, our findings novel, and our writing clear.
Weaknesses
Re: performance gain. As acknowledged in Section 6.2.1, we do find that HOMER achieves a comparable amount of success rate as fixed-compute baselines. However, the benefit of adaptive computation lies in other factors as well. We refer the reader to our common response (Fixed vs. adaptive benefit) for compute speed-up benefits.
Re: OGBench merit. While it is true that there can always be harder and more appropriate tasks, we were primarily interested in understanding adaptive computation in goal-reaching offline RL. The OGBench benchmark serves as a useful testbed for our initial and novel direction. Even in this simple setting, we were able to demonstrate interesting findings and benefits of adaptive agents such as the correlation with entropy and the use of caching to save recurrence steps over fixed-recurrence agents.
We do agree, however, that scaling these algorithms and findings to harder environments is interesting, and it is something we leave to future work.
Re: savings in compute cost. Please see our common response above (Computational benefits). In short, we do not unroll all the steps (the listing was a typo, see updated Listing 2); additionally, we report the FLOPs of our early-exit strategy in Appendix A.2 and show nearly 3x-10x times improvement in FLOPs over fixed compute agents.
Re: MoE paper. Thank you for the reference! We have added it to the related work in the updated manuscript. While this work does indeed share similar motivations, our work is distinct in that it focuses on the hierarchical and offline RL setting.
Questions
[1] HIQL: Offline Goal-Conditioned RL with Latent States as Actions. Park et al. 2023.
[2] Action-Sufficient Goal Representations. Hyeon et al. 2026
[3] Exploiting Local Dynamics Regularity for Reusable Skills in Offline Hierarchical RL. Dayal et al. 2026.
[4] Dual Goal Representations. Park et al. 2026.
[5] Horizon Reduction Makes RL Scalable. Park et al. 2025.
[6] Is Value Learning Really the Main Bottleneck in Offline RL? Park et al. 2024.