acceptodds
Under review as a conference paper at ICLR 2027

HOMER: Rational Meta-Reasoning for Hierarchical Reinforcement Learning

Abstract

A key capability for sequential decision-making agents is *rational meta-reasoning*: the ability to adaptively balance the requisite computing time to solve a problem against the time taken to perform those computations. Rational meta-reasoning agents can compute for the minimum amount of time needed to generate high-quality decisions, thereby avoiding poor decisions from insufficient computation or wasteful computations with marginal utility. In this work, we study meta-reasoning in the context of model-free hierarchical reinforcement learning (HRL) agents, composed of a manager who assigns -step subgoals to a worker. Since each level in the hierarchy optimizes a different objective, at a different temporal horizon, it raises the question of whether the requisite computation at each level is different? Motivated by this question, we first formalize meta-reasoning for HRL agents, and then introduce HOMER a hierarchical rational meta-reasoning agent, that instantiates the meta-reasoning process through iterative computations within model-free recurrent neural networks. We then apply HOMER to the offline goal-conditioned RL setting and demonstrate that it successfully attains goal-reaching success rates on par with non-meta-reasoning agents while using far less computation. We then empirically study the adaptive computations deployed at each level of the hierarchy; we observe that the worker's adaptation of compute correlates with distance to subgoals whereas the manager's adaptation of compute correlates with the entropy of the empirical -step distribution induced by the offline dataset. Finally, we show that a meta-reasoning manager can cache previous computations to improve its goal-reaching success rates and meta-reasoning efficiency.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, hierarchical reinforcement learning, meta-reasoning, offline RL, goal-conditioned RL, computation, adaptive computation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_vF3FOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates adaptive computation in a hierarchical reinforcement learning. They do it through proposing a method called HOMER, which combines two existing methods HiQL (a hierarchical offline RL method) and PonderNet (a method from supervised learning providing an adaptive computation mechanism). HOMER allows the manager and worker to independently learn how much computation to use. HOMER achieves a comparable or better performance than the non-adaptive baselines, while using less recurrent steps. They propose interesting analysis: how do the worker and manager networks behave under increasing time-horizon and of reusing previous computations.

Critical strengths

I find the idea to scale computations at different hierarchy levels very interesting in itself. In this context, the analysis in section 6.2.2, which investigates the amount of computation needed when the horizon increases is the strongest, and best to my knowledge most novel part of this paper.

Additionally, the paper is very well written.

Critical weaknesses

The gain in performance within a given computational budget doesn't look that impressive in most environments, when compared to the adaptive version.

The practical significance seems quite modest. OGBench tasks are on the simpler side as robotics goes and the network sizes are quite small. In this type of task, adaptive computations don't necessarily matter. Even though it is important to demonstrate this in simpler tasks first, this has already been done (and this paper cites that work). And even with that, based on Appendix A.1 it's unclear to me whether this method actually saves on computational cost, see Questions.

The paper doesn't cite previous work that did adaptive computation in forms such as early exits or mixture of experts. In particular some of them also looked into RL and reusing computations [1].

[1] https://arxiv.org/abs/2106.05409

Questions for authors
  • Do I understand correctly that the current implementation (as described in Appendix A.1) requires computing all the steps of the RNN each time? If so, could the authors demonstrate that this is not necessary?
  • 307-309 What does this oracle representation consist of? Why is learning the representation a problem?
  • 346-348 Why is setting the optimal hyperparameter for a non-meta-reasoning agent harder than for a meta-reasoning one?
  • 374-375 Could the authors expand on why does the number of k-step subgoals in the dataset matter here?
3 comments

Thank you to the reviewer for their feedback and effort.

We are glad you find our research problem interesting, our empirical analysis strong, our findings novel, and our writing clear.

Weaknesses

Re: performance gain. As acknowledged in Section 6.2.1, we do find that HOMER achieves a comparable amount of success rate as fixed-compute baselines. However, the benefit of adaptive computation lies in other factors as well. We refer the reader to our common response (Fixed vs. adaptive benefit) for compute speed-up benefits.

Re: OGBench merit. While it is true that there can always be harder and more appropriate tasks, we were primarily interested in understanding adaptive computation in goal-reaching offline RL. The OGBench benchmark serves as a useful testbed for our initial and novel direction. Even in this simple setting, we were able to demonstrate interesting findings and benefits of adaptive agents such as the correlation with entropy and the use of caching to save recurrence steps over fixed-recurrence agents.

We do agree, however, that scaling these algorithms and findings to harder environments is interesting, and it is something we leave to future work.

Re: savings in compute cost. Please see our common response above (Computational benefits). In short, we do not unroll all the steps (the listing was a typo, see updated Listing 2); additionally, we report the FLOPs of our early-exit strategy in Appendix A.2 and show nearly 3x-10x times improvement in FLOPs over fixed compute agents.

Re: MoE paper. Thank you for the reference! We have added it to the related work in the updated manuscript. While this work does indeed share similar motivations, our work is distinct in that it focuses on the hierarchical and offline RL setting.

Questions

  1. Please see the common response above (Computational benefits). In short, we do not roll out all the recurrent steps.
  2. The oracle representation provided by OGBench just consists of the low dimensional state information required to communicate the goal. In the maze tasks, this is just the 2D location. In the puzzles tasks, it is the one-hot vector of on/of configurations. Learning the subgoal representations is challenging and is a separate research direction [1, 2, 3, 4]. To avoid confounding issues due to poor subgoal representations, we chose to use the oracle representations as done in prior work [5].
  3. From an agent designer perspective, we believe setting the optimal hyperparameter for non-meta-reasoning agents is more challenging due to the lack of flexibility since it presumes the designer knows the appropriate amount of compute budget for all situations beforehand. On the other hand, we believe setting the compute prior parameter affords some flexibility since it simply provides a nudge on the rough compute budget preference but with the flexibility for adaptation to take over if necessary.
  4. The primary driver of the adaptive computation is based on the offline RL loss, which in this case, is the AWR loss. At a given kk, there may be many subgoal options and since AWR is a mode-covering objective [6], it is trying to find a single (Gaussian) distribution that covers all the possible modes over these k-step subgoals. If there are many kk-step subgoals, the distribution is harder to model, which results in more computation needed for more model expressivity to minimize the loss.

[1] HIQL: Offline Goal-Conditioned RL with Latent States as Actions. Park et al. 2023.

[2] Action-Sufficient Goal Representations. Hyeon et al. 2026

[3] Exploiting Local Dynamics Regularity for Reusable Skills in Offline Hierarchical RL. Dayal et al. 2026.

[4] Dual Goal Representations. Park et al. 2026.

[5] Horizon Reduction Makes RL Scalable. Park et al. 2025.

[6] Is Value Learning Really the Main Bottleneck in Offline RL? Park et al. 2024.

Dear authors,

Thanks for the clarifications. While I believe that the changes strengthen the work, my main concern about wider applicability remains unaddressed for now.

Dear reviewer,

We appreciate your timely response and are glad that the changes have strengthened the work!

Even though it is important to demonstrate this in simpler tasks first, this has already been done (and this paper cites that work).

Regarding the wider applicability based on your initial comments, respectfully, there is perhaps a misunderstanding. Could you please clarify the above quote?

To the best of our knowledge, this claim is not true. Studying adaptive computation in RNNs for hierarchical RL agents in the offline GCRL setting has not been explored before. Based on the components of our work, the closely related work that has been done before is:

  1. PonderNet [1] which used adaptive computation for RNNs but on supervised learning tasks only, not for RL nor HRL.
  2. IRU [2], which showed non-adaptive RNN compute agents could help for flat RL agents on offline GCRL. That is, they did not focus on adaptive computation nor HRL. Our FOAMER results are adaptive compute for flat RL, but that is just simply for benchmarking performance, not the focus of the paper.

Given these points, we believe that our empirical analysis on the existing benchmark is an appropriate step for this novel direction, especially given that we have still been able to demonstrate interesting insights and benefits (entropy correlation, caching, comparable success at much fewer FLOPs).

Please let us know if we can clarify anything else. Thank you.

[1] PonderNet: Learning to Ponder. Banino et al. 2021.

[2] On the Role of Computation in Reinforcement Learning. Ghugare et al. 2026.

Reviewer_u731OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces HOMER, which combines HIQL with iterative LSTMs and adaptive halting to study computation allocation in hierarchical reinforcement learning. Experiments show comparable task performance with fewer reported recurrent steps, reveal distinct computational needs across hierarchy levels, and explore the benefits and risks of reusing internal states.

Critical strengths
  • The methodology design has good math and engineer intuition. The combination of HIQL, iterative LSTM and PonderNet is technically well-motivated, allowing the manager and worker to learn separate, input-dependent halting policies that balance policy learning objectives against computational cost.
  • A meaningful conclusion I learn is that the worker’s computation increases slightly with the subgoal horizon before plateauing, whereas the manager’s computation exhibits a more pronounced inverted-U pattern. This suggests that different levels of the hierarchy may require distinct computational strategies and constitutes one of the paper’s most distinctive empirical findings.
  • It is insightful to show that reusing internal states can reduce the reported number of recurrent steps, while unrestricted reuse can severely degrade success rates. The periodic reset experiments suggest that computation reuse and cache invalidation should be studied jointly.
Critical weaknesses
  • The actual computational saving is not clear to me. The appendix implementation unrolls all recurrent steps before selecting the output at the step closest to the expected halting time. This demonstrates that useful representations can be obtained at earlier steps, but does not establish that less computation is performed during deployment. Without a demonstrated early-exit execution path and runtime measurements, the paper’s efficiency claim is not sufficiently substantiated.
  • The authors acknowledge that appropriately chosen fixed recurrence counts can achieve similar performance. It is therefore important to establish whether input-dependent adaptive halting outperforms well-tuned fixed-computation baselines under the same actual computational cost. Otherwise, the benefit of adaptive allocation, beyond simply choosing a suitable computation budget, remains unclear.
  • Section 4 introduces the meta-MDP formulation, while Section 5 presents the practical training objectives. However, the connection between the two remains insufficiently established. A theoretical characterization of how the proposed objectives relate to the meta-MDP objective (e.g., through optimality guarantees or a performance–computation tradeoff analysis) would substantially strengthen the paper’s technical contribution.
Questions for authors

This paper is good on method design and emprical analysis. However, it remains unclear under what conditions the proposed approach can still be effective and reliable. Could the authors provide some theoretical analysis on optimality guarantees or a performance–computation tradeoff analysis? This would significantly upgrade the rating of the paper.

2 comments

We thank the reviewer for their feedback.

Thank you for acknowledging the intuition behind our method and the motivation of our approach. We are glad that you found the general findings on adaptive computation at the different levels and the caching of compute states to be meaningful and insightful.

Below, we address specific concerns.

Weaknesses

Re: unrolling all recurrent steps. Please see our common response above (Computational benefits). In short, we do not unroll all the steps (the listing was a typo, see updated Listing 2); additionally, we report the FLOPs of our early-exit strategy in Appendix A.2 (Table 1).

Re: fixed-compute baselines. Please see our common response above (Fixed vs. adaptive benefit).

Re: meta-MDP formulation. Thank you for this valuable feedback! Please see our common response above (Linking Meta-MDP VF and Practical Objective) and the updated manuscript. As you mentioned, we hope the additions have strengthened our technical contribution.

Questions

Yes we have updated the manuscript with a theoretical characterization relating the practical objective to the meta-MDP. Please see our common response (Linking Meta-MDP VF and Practical Objective).

Dear u731.

Thank you again for your strengthening our work. Given the rebuttal timeline and the weekend coming soon, we wanted to gently remind you to please let us know if we have adequately addressed your concerns.

Thank you!

Reviewer_CAfQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether the levels of a hierarchical RL agent need different amounts of computation. It defines a meta-level MDP for each level of a manager–worker agent, where the meta-policy decides when to stop computing. It then builds HOMER by adding PonderNet-style adaptive halting to LSTM policies inside HIQL; the computation being adapted is the number of LSTM recurrence steps. On five offline goal-conditioned OGBench tasks, the paper reports four findings:

  • Success vs. compute: HOMER and its flat counterpart (FOAMER) match the success of fixed maximum-recurrence agents while using fewer recurrence steps.
  • Worker compute: it grows slightly with subgoal horizon kk and then plateaus.
  • Manager compute: it follows an inverted-U shape in kk that tracks the dataset's conditional entropy H(St+k∣St,G)H(S_{t+k} \mid S_t, G) (Spearman 0.84 and 0.97 on two maze tasks).
  • Caching: reusing LSTM states from previous decisions can cut manager compute, sometimes while also improving success.
Critical strengths
  • A well-posed question with careful, interesting analysis. Asking whether a hierarchy of decisions implies a hierarchy of compute needs is a novel and natural question. The manager-compute/entropy relationship is a genuinely interesting, non-obvious observation, and it comes with a plausible explanation: as kk grows the manager must choose among more candidate subgoals, until kk is large enough that it simply outputs the goal. The experiments use 10 seeds with confidence intervals, and the per-(s,g)(s,g) visualizations help interpretation. This analytical contribution is the paper's main merit.
Critical weaknesses
  • Thin method, and the efficiency claim lacks the right baseline. HOMER is HIQL plus PonderNet. The meta-MDP formalism isn't used by the algorithm: PonderNet optimizes imitation loss plus a geometric-prior penalty, not a value-of-computation objective, so "rational" meta-reasoning is a stretch. The paper admits that a fixed reduced step count matches HOMER, and HOMER itself needs θp\theta_p tuned. Without a per-environment tuned fixed-step baseline, or a wall-clock measurement, the efficiency benefit is not established.
  • The level-specific claims are weakly supported. The worker's halting step only varies between 5.0 and 5.8, essentially the prior's mean (1/θp=51/\theta_p = 5), so its "adaptation" may simply reflect the prior. The entropy correlation comes from two maze tasks and is confounded with kk and with success rate, and no intervention tests it causally.
Questions for authors
  1. Does HOMER beat fixed-step agents tuned per environment at matched compute? Is θp\theta_p easier to set than a fixed step count?
  2. Does the worker's halting go beyond the prior's mean across different θp\theta_p? Does forcing fewer steps on high-compute inputs hurt success?
  3. Does the entropy correlation hold when entropy is changed at fixed kk, or after controlling for kk and success?
3 comments

We thank the reviewer for their feedback and assistance in making the work stronger.

We appreciate that the reviewer acknowledges the clarity and novelty of our research question, our interesting empirical findings, and rigor of empirical results.

Below, we address the concerns raised:

Weaknesses

Re: relation to meta-MDP formalism. This is great feedback. Thank you! Please see our common response (Linking Meta-MDP VF and Practical Objective).

Re: fixed-step baseline. Please see our comments in the common response (Fixed vs. adaptive benefit).

Re: worker’s halting steps. The primary driver of the adaptive compute is the offline RL loss, whose objective is to learn a policy that fits the (advantage-weighted) distribution over actions (or subgoals). In general, we found that the action distribution is similar across all states/goals on average. Therefore, it is not surprising that the worker’s chosen amount of recurrence hovers closer to the prior parameter.

Re: entropy correlation The adaptive compute is driven purely by the difficulty in fitting the offline RL loss. Therefore, the entropy correlation is independent of success, and is linked to the difficulty in modeling the distribution over subgoals at different kk.

Questions

  1. As noted in Section 6.2.1, we found that HOMER performs on-par with fixed-compute agents. Yes, from an agent designer perspective, we believe setting the prior affords some flexibility than setting a fixed recurrence step since the former provides a nudge on the rough compute budget preference but with the flexibility for adaptation to take over if necessary. The fixed recurrence step, on the other hand, dictates the compute budget used in all situations, which can be harder to gauge.
  2. No, we generally found that the worker’s recurrence stayed close to the geometric prior mean. Yes, forcing fewer steps could hurt performance since the LSTM would have less expressivity in modeling the action distribution.
  3. Yes, the entropy correlation generally holds. In preliminary analysis, where we used expert datasets (optimal datasets used for behavior cloning, not RL), where the kk-step empirical entropy is lower, we found that the manager’s compute to still correlate with the smaller entropy at the same kk. For more details, please see above re: entropy calculation.

Dear CAfQ.

Thank you again for your strengthening our work. Given the rebuttal timeline and the weekend coming soon, we wanted to gently remind you to please let us know if we have adequately addressed your concerns.

Thank you!

I appreciate the authors' reply, but my concerns in Critical Weaknesses still exist. I'm going to keep my score.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Common Response to Reviewers

We thank the reviewers for their efforts in making our paper better and for appreciating our work. We are glad that the reviewers consider our research question interesting and novel, empirical analysis insightful, and writing clear.

We address the common concerns here and respond to each reviewer individually below.

We have uploaded an updated manuscript with the feedback provided. All changes are marked in blue. The changes are located in: 1) Appendix A.1. and Section 5; 2) Appendix A.2.; 3) Appendix A.3 (Listing 2). Please review them when you get a chance.

We request the reviewers to please consider re-evaluating your review if we have adequately addressed your concerns.

Linking Meta-MDP VF and Practical Objective (CAfQ and u731)

See: Proposition 5.1 and Appendix A.1 of updated manuscript.

Thank you for the suggestion! We have made the connection between the practical objective of PonderNet with advantage-weighted regression (AWR) and the meta value function explicit in Proposition 5.1 (followed by detailed discussion and proofs in Appendix A.1). This result justifies the use of our practical objective for meta-reasoning in the offline setting.

In short, our method: 1) replaces value maximization in Equation 1 with a closeness-to-optimal policy maximization term, 2) uses a specific choice of the cost penalty in terms of the geometric prior parameter, and 3) adds an entropy regularization term similar to that used in maximum-entropy RL [1].

Computational benefits (CAfQ, u731, and vF3F)

See: A.2. and A.3 (Listing 2) of updated manuscript.

Thank you for raising this concern. The first main concern seems to be due to an unfortunate typo (which we have corrected in the updated Listing 2 in the Appendix) and the second is easy to address. To elaborate:

  1. Rolling out all steps and early exit strategy (u731 and vF3F). Rolling out all steps is only required during training since PonderNet minimizes the expected AWR loss across all steps (weighted by the probability of halting). However, it is not needed nor used during deployment. At deployment, the algorithm samples the halting step sequentially, where at each recurrent step, the halting head outputs a value λ∈[0,1]\lambda \in [0,1] and then a sample from a Bernoulli distribution with parameter λ\lambda is drawn. If the sample is 1, the recurrence halts and the policy acts from the current step; otherwise, the process repeats, with halting forced at the last recurrence step. So there is indeed a computational benefit since the meta-policy can terminate at an earlier recurrence step compared to a fixed recurrence agent.
  2. Runtime benefits (u731 and CAfQ). Thank you for this suggestion! We ran this evaluation for HOMER and the fixed-compute baselines in terms of FLOPs averaged over different inputs, and do indeed see a benefit during deployment. We refer the reader to Appendix A.2 (Table 1). We see that HOMER achieves computational efficiency with 3x-10x fewer average FLOPs than the maximum fixed-recurrence agent at comparable levels of success, depending on the environment.

Fixed vs. adaptive benefit (CAfQ, u731, and vF3F)

As noted in Section 6, we did observe that fixed-compute baselines performed comparably in terms of goal-reaching success rates with HOMER. Despite that we want to stress the following important benefits of adaptive computation that go beyond success rates:

  1. Our results in Section 6.2.3 (warm-starting computation) are a huge advantage for adaptive compute agents like HOMER. On the other hand, fixed-computer agents cannot benefit from this caching. While success rates may be comparable, HOMER is able to exploit caching to maintain its success rate (and even improve upon it) at far fewer recurrence steps.
  2. Additionally, thanks to the suggestion by the reviewers on measuring runtime efficiency, we also see that HOMER achieves its success with nearly 3x-10x fewer average FLOPs at comparable levels of success as fixed-recurrence agents.

Finally, our goal is not to maximize performance on a given benchmark. Instead, our goal is to use HOMER as a vehicle to understand adaptive computation in the offline hierarchical RL setting (such as PonderNet’s relation with AWR, the relation to entropy of the dataset, and caching states to speedup computation), while still demonstrating that HOMER performs competitively on a given benchmark. As such, we hope that the community can benefit from our insights in this novel direction of hierarchical meta-reasoning.

[1] Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review. Levine. 2018.

Dear reviewers. As the end of rebuttal window is approaching, it would be great to know if we have adequately addressed your concerns. Please let us know when you can. Thank you! Once again, we appreciate your time and feedback.

Dear reviewers, The rebuttal window is coming to a close. We have put in our best efforts to target the raised concerns. Please let us know if they make sense and address your concerns. It would be of great help. Thanks again for your feedback and for making the paper stronger!

AuthorsOpenReview ↗

Follow up and summary of changes

Dear reviewers.

Thanks again for the feedback to make our paper stronger! We believe we have addressed the main concerns, but would love to know if we have actually done so. Today is the last day so please let us know.

Since the rebuttal window is coming to a close, we have updated the pdf to remove the blue-colored changes. Below are the changes that try to address the main concerns from your review.

  • Soundness of our practical objective as a meta-reasoning objective. Proposition 5.1 (page 6).
  • HOMER is computationally more efficient than non-adaptive agents in terms of FLOPs. Table 1 (page 21).
  • All steps of the recurrence are not unrolled. Listing 2 (page 23).

These concrete changes address the following main issues raised by the reviewers:

  • Is there a relation between meta-reasoning objective and practical objective (yes, there is, which we make explicit).
  • Are there computational benefits to HOMER during deployment? (It was presumed that all steps were unrolled, which is not the case; related: FLOPs speedup).
  • Are there benefits to adaptive agents if non-adaptive agents perform comparably? (Yes, FLOPs speeds up, warm-starting to further save compute and boost performance).

The common response below details how we have addressed these. Additionally, we have addressed each reviewer directly.

Thanks again.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.