acceptodds
Under review as a conference paper at ICLR 2027

Learning to Act While Thinking

Abstract

Operating in real-time, dynamic environments requires agents to respond quickly to sudden changes while also planning over long horizons to solve difficult tasks. In such settings, plans take time and energy to compute and the environment does not pause while an agent thinks. As a result, planning should only be done when necessary and plans may arrive stale, as they are computed from a prior state of the world. Such an agent therefore must decide when to request a plan, how to act while waiting for it, and to what extent to follow it once it arrives. We formalize this setting as a real-time meta-reasoning MDP, in which an agent can request a plan from a slow, asynchronous planner but incurs a cost and delay before the plan is returned. Using this formalism, we show when requesting a plan improves an agent's policy given a cost and delay. We then describe our method Acting While Thinking (AWT) to train reinforcement learning agents to learn to request plans and act while they are computed. We then evaluate AWT in five real-time, dynamic tasks, and show that learned policies achieve a higher return and use fewer plans than fixed meta-reasoning and heuristic baselines. Finally, we analyze the behavior of the resulting agents and show when they learn to query additional computation.

Primary area:
reinforcement learning
Keywords:
Reinforcement learning, meta-reasoning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_CAWoOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper formalizes the problem of when to request a plan and how to act while waiting for it and once receiving it, with a specific cost and delay as an MDP. They train RL agents to get modest improvements compared to various heuristic baselines.

Critical strengths

The problem of when to plan and when to act is an interesting one for sure, and the formulation is clean.

Critical weaknesses

The paper overstates its novelty compared to prior literature. The problem is well known and its formulation is not particularly novel. The formulation of when to plan is very similar to the framing in When2Ask (Hu et al. 2024) and Learning when to plan (Paglieri et al. 2025), the first of which is not referenced and the second one is only mentioned at a high-level. As far as I'm aware, the main novelty here comes from the delay term and acting while waiting for the requested plan to arrive, while other methods usually stop acting while waiting for the plan to be formed.

A limitation is that the planner here is a perfect oracle, which to be fair the authors mention, however the training of the planner is as important of a problem as the acting policy.

Finally, the improvements over the heuristic baselines are quite modest.

Questions for authors

Can the authors better compare and contrast their work and the actual novelty with the existing literature, both in the framing and in the baselines where possible? How will AWT work when there is no optimal planner?

3 comments

Thank you for your feedback!

We have included When2Ask in the revised version. The key novelty of our work in comparison to these papers is the need to 1) strategically ask to plan even though environment change results in a partially stale plan and 2) learning what to do while waiting for the plan. This fills a critical gap in the literature as there are many domains (e.g., emergency response robotics, autonomous vehicles) where the world will not pause while the agent plans.

We agree that training of models for planning is also an important problem and there are interesting questions around how a learned model would interact with meta-reasoning in dynamic environments, however, we chose to first study the meta-reasoning questions which specifically relate to plan delay for this paper. We noted this in the global rebuttal, but we have modified the text to clarify that an oracle planner meant an observation of the full state, not a perfect model of transition dynamics as our previous description was imprecise. If anything, learning agent behavior models/world models would improve the plan if it provided an accurate projection.

As far as limited improvement over baselines, we don't claim that our method is the best possible solution to this problem and instead, our paper provides a formalism for studying this novel problem and a method to train an agent in it. We believe that further improvement is possible and are interested in continuing that direction.

Hello reviewer CAWo,

As today is the last day of the rebuttal period, we were hoping you would let us know if we have addressed your concerns.

Thank you!

Thank you for the response and clarifications. I will review these points carefully and take them into account during the reviewer–AC discussion period.

Reviewer_zbMAOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Acting While Thinking (AWT), a method that learns when to request for a plan and how to act while it is generated, in dynamic environments where states are in constant flux and planning incurs real-time costs that could be costly. Experiments demonstrate that jointly learning in this manner yields greater average returns in their benchmarks, and more intuitively that agents can learn to meta reason and be reactive in dynamic environments.

Critical strengths
  • Straightforward method with broad applications in robotics and other dynamic environment needs. In scenarios where you have split action-planner models, this method shows how you can jointly learn to manage both effectively. As mentioned this can have broad applications which has merit, and experiments are promising to validate the technique.
Critical weaknesses
  • Comparison to real-time planning systems. A critical question is whether this system design is needed, and why it would be preferred over an agent that plans using a real-time decision making model/world model) or even a direct learned policy. Reported gains over the period request/heuristic waiting baselines do not give a complete value analysis of the proposed design over other capable systems.
  • Oracle planner. Given that all experiments use an oracle planner, it is further unclear how robust this method is to noisy/suboptimal plans. While the authors highlight this as a limitation, noting that this paper isolates the meta-reasoning problem, the strength of the paper is greatly weakened by the unknown value of the work in the absence of an oracle planner. It is also unclear whether the model would still be able to meta-reason over a non-oracle plan, and whether the introduced noise would unbalance the decision making system, for example. It remains unclear whether the learned policy could adequately allocated compute to request additional plans.
Questions for authors
  1. How does AWT compare to agents that plan with real-time world models? (e.g. https://github.com/danijar/dreamerv3)
  2. Does AWT retain its performance with non-oracle planners?
  3. How does AWT compare to direct policies?
3 comments

Thank you for your review!

Response to specific questions:

  1. This is a reasonable topic to study but in this work we are considering a different question. Instead, we are operating under the assumption that we do not have a real-time world model for the given domain. Many works such as JEPA (https://arxiv.org/pdf/2512.24497) or VLAs can provide high-quality plans in very complex settings but require 30 seconds or more to produce such plans. You are also correct that there are other ways to use a world model such as how Dreamer (like you mention) performs standard MBRL to train an agent to act in real-time. However, Dreamer and similar DYNA-like methods don't plan in their world models so if one did use them to plan it is uncertain if it would be real-time or not. Regardless, in this paper we focus on enabling these highly performant but slow planners in real-time settings.

  2. Discussed in global rebuttal.

  3. We have added additional experiments to the paper that train the agent without access to the plan to Table 1; these results show that direct policies fail to learn the tasks in the gridworlds and that in Waymax it performs only as well as the worst baseline. To study the question of an agent learning to use slow planners, we created environments in which a policy without the performs substantially worse without a planner and now Table 1 shows this.

  1. Agreed
  2. My remaining concern is the planner receives fully observed state, which in practical partially observed environments, it is unclear how such a system would behave. While I understand that this may be out of the scope of the paper, it should clearly distinguish these limitations.
  3. Thank you

I am raising my score as much of my concerns have been addressed, and I would like to see some additional discussion on limitations to partially observed states discussed.

Thank you for your response and feedback!

In the updated draft, we have added more commentary to Section 6 Discussion and Future Work on our use of a fully observable planner state. The summary of our changes are that we discuss how adding a more realistic (learned) planner would possibly improve or worsen plan quality by either decreasing plan staleness with learned agent models or by adding model error into the plans as more realistic planners would rely on a local observation and world model. We note that this is an interesting and important question for future work.

Let us know if you have more feedback on this topic!

Reviewer_9ftrOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper describes a real-time metareasoning MDP in which an agent can request a plan from a slow, asynchronous planner that is not received until some number of steps in the future, and which incurs a cost. The authors train neural nets via deep RL (PPO) to act (i.e. act in the real world and request plans) in environments using this formalism. They show that this works well.

Critical strengths

Critical strength: The paper is simple, clear, and does what it says. It does seem to fill in a gap in the real-time RL / metareasoning literature. The research setting is uncontrived: like in "real-time RL", planning (compute) is not instant, and the environment changes while time is taken to compute; like in metareasoning, planning also has intrinsic cost and we want to learn/reason how to optimally use our compute; instead of simple models or heuristics, we want to learn possibly-complex behavior via deep reinforcement learning. However, despite the simplicity of the setting, previous research seems to hit upon one or two of these features, but not all three.

The authors lay out their reasonable formalism, their experimental design, and results showing that the agents are able to learn behavior better than the unlearned baselines. They perform ablation experiments and provide some commentary.

Critical weaknesses

I find three critical weaknesses with this work:

  1. Planning delay δ\delta is explained and discussed confusingly and is unrealistic: In lines 164-165, the planning delay δ\delta is introduced: it "may be fixed or drawn from a distribution at the time of each request, which models variable planner execution time". I have a few qualms about this:
    (A) I don't get what we are modeling by having δ\delta be drawn at the time of request -- when I think of planners with variable execution time, I don't think that the plan times are known a priori to finishing the plan (for example, a planner that ends when it finds an optimal solution, or can't improve its solution for more than X seconds, or finds a solution at least over some threshold Y -- none of those times would be known a priori). However, if variable-length planning isn't known at the time of request, then it doesn't fit in the given metareasoning MDP formalism. Therefore, I find the modeling choice to include δ\delta to be limiting if one wishes to extend it to be more practical/realistic.
    (B) Is unfixed δ\delta used anywhere in the paper? In the experiments? It should be clarified whether unfixed δ\delta is part of what is being discussed in this paper, or if it is a possible extension that is not seriously interrogated in this paper.
    (C) What does "drawn ... at the time of each request" mean, exactly? Is it known by the agent before making the request, or only after? If only after, then some mentions of δ\delta in the paper seem to need rewording from referencing a single δ\delta to the distribution of δ\delta, like Equation 6, which should have an expectation over δ\delta in that case. Further, in that case, does δ\delta correlate to the expected strength of the plan?

  2. Missing references to metareasoning and real-time RL research:
    (A) The paper discusses its "meta-reasoning MDP", but does not cite the seminal metareasoning work of Eric Wefald and Stuart Russell. The work most often cited is the paper "Principles of Metareasoning" (1991) or the book "Do the Right Thing: Studies in Limited Rationality" (1991).
    (B) Further, the paper "Dynamic thinker: Optimizing decision-time planning with costly compute" by Wang et al. (2025) should be cited as related work in "Variable Computation" as it also defines a similar metareasoning MDP, uses deep reinforcement learning instead of simple models or learned heuristics, and learns acting and thinking together. However, it does not involve real-time domains.
    (C) Another good related work to cite for "real-time reinforcement learning" or "real-time planning" is "Thinking While Moving: Deep Reinforcement Learning with Concurrent Control" by Xiao, et al. (2020). Although they consider action-sequences that take time to execute instead of plans that take time to be received (Figure 4 in their appendix is a good illustration), the model is similar, and in both, a deep RL model is conditioned on both the current state and either the plan or "concurrent knowledge", so it should be cited.

  3. I was confused about Section 3.3: it's stated that the state has "no request in-flight, so d = 0". I think the rest of the subsection is written as if it is assumed that the agent requests a plan at that timestep, but this assumption is not stated anywhere. Is my understanding correct? For example, the sentence immediately following, as well as Assumption 2 and Proposition 1.

(Also, should the subsection title have a question mark in it?)

Questions for authors
  1. Section 3.3, Assumption 1: For elucidation, it would be helpful to have some examples of the two assumptions in Assumption 1 and why we think they are generally true, as well as examples of cases where they are not true.
  2. In Section 5.3, it is stated that "Evade is the only one in which a baseline had a higher success rate". What about Warehouse, in which Short achieves 92 +/-1 compared to AWT's 90+/-1?
3 comments

Thank you for your feedback on our paper! We will first answer your questions and then respond to the weaknesses proposed.

Questions:

  1. The assumptions in section 3.3 may not hold generally during real learning when a policy performs periodic plan requests but the agent does never condition on the plan state, instead that is only used to keep the state Markov. They are made so that we can compare the return of an agent after the delay and have the only difference in return be from the updated plan. We have updated the beginning of 3.3 to clarify this motivation.
  2. You are correct that in Warehouse the Short Baseline has a higher mean success rate but the CIs overlap and thus, we did not include it in that part of the analysis. However, this brought two missing overlapping bolds on the Warehouse Table 1 to our attention that we have fixed.

Weaknesses:

  1. Thank you for pointing out that our description of the planning delay is unclear. We originally attempted to make a general formalism for this problem setting by including a stochastic δ\delta as a possibility but see that the stochastic case (which we don’t use) still needs some additional refinement. We have simplified the definition of δ\delta to only a fixed value so that the formalism more closely matches our empirical evaluation.
  2. We have added the references presented to related work, thank you for mentioning these. We quickly note that our work has novelty in comparison to these works due to the combination of real-time, dynamic tasks and delayed computation.
  3. The proposition states "Suppose the policy improvement step of policy iteration sets π′(s~)←(ae,request)\pi'(\tilde{s}) \leftarrow (a^e, \text{\texttt{request}}), so that the improved policy requests a plan but does not change the control action”. The intuition of the proposition is that we are initially in a state where the agent does not request a plan and under what conditions will policy improvement update the policy to request a plan? We added this intuition to the proposition discussion to make it more clear, however, if you have further recommendations for improvement let us know!

Thank you for answering my questions 1 and 2.

Thank you for addressing Weakness 1 -- that clarifies some confusion I had while reading the paper.

Thank you for addressing Weakness 2.

For weakness 3: I understand it now, but I do still think you could make it more clear in the introductory paragraph of Section 3.3 that we are interested in comparing the no-plan-request case with the yes-plan-request case. (Just my opinion!)

Overall, I feel good maintaining my recommendation of weak acceptance (3).

Thank you for your response, we appreciate your feedback!

We have reworded the paragraph of 3.3 to more specifically state what the proposition does. With the added sentence, it now says: “We answer this question by analyzing when policy iteration would update a policy from using a \texttt{no-request} plan action to using a \texttt{request} plan action \citep{sutton2018reinforcement}. Concretely, we compare the value of a \texttt{no-request} plan action to the value a \texttt{request} plan action in the same state \tilde{s}. We derive…”

If possible, let us know if this addition further improves the clarity!

Comments on the paper

4 comments
AuthorsOpenReview ↗

Global Rebuttal

We thank all reviewers for taking the time to provide feedback on our paper! We are excited to hear that the reviewers found the problem was interesting, has real-world applications, and that the formalism was presented clearly.

Reviewers zbMA and CAWo both raise the issue of using a fixed world model in our work. While we agree that the use of learned world models would be an interesting addition to the study, our work focuses on the question of how an agent can learn to use delayed plans and act reactively in dynamic domains. However, we agree that the description of the world models and planners was imprecise in the prior draft. Previously, we described the planners as “oracle” planners, however, the planner is an oracle in that it can see the full state of obstacles (walls, doors, etc) in the world, not in that it uses an exact future projection of object movement. The gridworld planners use the full state at the timestep plan was called and return a path around objects from that state but do not predict other agent movement. In Waymax, we use a future projection of vehicle movement, but that projection assumes constant speed and thus, does not guarantee a correct plan. We have added a paragraph discussing this to the empirical setup and adjusted the wording in the limitations to better reflect our experiments.

While we believe the paper is complete as-is (without learned model experiments), we agree this could be a valuable orthogonal axis to study meta-reasoning on, and thus, we are currently setting up and running experiments with planning with learned agent models. If we are able to complete these in the space of the response period, we will update our comments and the paper and can otherwise include them for the camera-ready.

Second, reviewers CAWo and 9ftr mention a few citations to be included in our related work. We agree with their recommendations and have included the references in the updated draft. However, we note that none of the additional references decrease the novelty of our work and are included to more broadly cover the topic.

We have updated the document with a latexdiff of our changes.

To all reviewers,

Thank you again for your time spent reviewing our paper!

We have addressed your questions and critiques in our followup comments and our updated draft. We wanted to gently remind you to let us know if we have resolved your concerns or if you have additional followup questions.

Hello reviewers,

We wanted to send a quick reminder to please check out our response to your comments and let us know if we resolved your feedback.

Thanks again!

To the reviewers and AC, thank you once again for your time spent helping improve our paper!

As the rebuttal period is coming to a close, we note that two of the three reviewers support accepting our paper and the concerns of the reviewer who did not respond are resolved in our response to their review.

Additionally, we have removed the latexdiff highlighting from the paper to present the work as is.

Thanks again!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.