acceptodds
Under review as a conference paper at ICLR 2027

How Far Should an Agent Imagine? Decision-Stable Planning with Learned World Models

Abstract

Model-based reinforcement learning uses learned world models to evaluate future consequences, yet deeper imagination brings both delayed decision information and accumulating model error. Existing methods often adapt rollout depth using prediction uncertainty alone, although uncertainty matters only when it can change the preferred action. We introduce MarginPlan, which selects planning depth according to decision stability. For each action, it constructs uncertainty-aware return intervals over plausible dynamics models and identifies whether the same action remains optimal across them. Planning is declared decision-stable when the lower return bound of the preferred action exceeds the upper bounds of all alternatives. MarginPlan extends imagination only when the expected reduction in action ambiguity justifies the additional model-induced uncertainty. We evaluate controlled tasks that independently vary model uncertainty, action-value margins, and delayed rewards, including matched-uncertainty settings requiring different planning depths, together with computation-matched baselines. This formulation shifts adaptive planning from uncertainty-aware rollout truncation to decision-stable imagination, where planning depth is determined by whether additional lookahead can reliably improve the action decision.

Primary area:
reinforcement learning
Keywords:
model-based reinforcement learning, world models, adaptive planning, uncertainty quantification, decision stability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_MpfSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper aims to adaptively balance the benefits of long-horizon planning with accumulating model errors in model-based reinforcement learning. For each candidate action, their proposed method uses split-conformal calibration to construct a calibrated return interval over plausible dynamics models for each planning horizon level and executes the action whose lower bound exceeds the upper bound of all others. When no such action exists, on each iteration, they predict the influence of extending the rollout and whether that will allow any action to be certified for stability enough to justify the additional interval uncertainty and the cost of computation. The resulting algorithm gives an adaptive planning horizon for each state that reflects the uncertainty in estimated returns and separation between competing actions. This is evaluated on synthetic tasks that are designed to isolate the horizon selection rule.

Critical strengths
  • The principle behind this paper that planning depth should depend on the uncertainty that matters for selecting the best action is interesting and well aligned with decision directed planning. Using calibrated return intervals to certify decision stability makes sense as a stopping criterion.
  • The experiments are designed to vary the action-value margins, uncertainty, and delayed rewards while keeping the predictions fixed across methods. This is a well controlled analysis that demonstrates how the appropriate planning horizon depends on how clearly the candidate actions can be differentiated in terms of their expected returns.
Critical weaknesses
  • The theoretical guarantee relies on conformal calibration using the environment return targets for all candidate actions. In the DMC experiments, these are obtained by resetting the simulator and executing each candidate action separately, which is generally infeasible for realistic online RL. In addition, there is limited empirical validation as most experiments use synthetically generated return predictions rather than a learned world model. This method would be most useful if it could be effectively applied in more practical settings even with some approximations.
  • Proposition 1 appears to bound the probability of incorrectly certifying an action but does not guarantee the correctness of decisions made through the fallback mechanism. In the experiments, only 42% of decisions are certified and 50% use the fallback, making this an important gap to address.
Questions for authors
  • How could the method be calibrated in a realistic RL setting? How could calibration be maintained as the world model and policy change? Perhaps an offline-to-online RL setting where the offline data is used for initial calibration could be possible.
  • It would be very interesting to see if the adaptive decision-directed planning mechanism also improves training sample efficiency compared with fixed-horizon or uncertainty-based planning, could this be shown?
  • How much improvement comes from the learned continuation predictor?
Reviewer_Huu1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes MarginPlan, a rule for choosing the imagination depth in world-model-based planning. For each candidate action, conformal return intervals are built from an ensemble of world models and calibrated jointly over actions and horizons. Planning stops when the lower bound of the leading action exceeds all upper bounds of the rest. Otherwise a small regressor decides whether one more imagination step is worth its cost. The main results come from a fully synthetic testbed (App.A). A small DeepMind Control Suite study is in App.G.

Critical strengths
  1. The stopping criterion is clear and well motivated. The depth depends on whether more lookahead can still change the chosen action, not on uncertainty alone.

  2. The calibration is nice. The conformal score takes the maximum over actions and horizons, so the certificate stays valid even though the algorithm chooses the stopping horizon itself. This is the real contribution of the paper, and the rule is simple enough to be plugged into existing planners.

Critical weaknesses
  1. The main evidence comes from a self-designed synthetic generator, not from world models.
  • In App.A (Eq.12), the "predictions" are the true return plus noise, a bias that shrinks with depth, and an error that grows with depth. The 24 "world models" are 24 noise draws. The generator is therefore build to match the paper's premise, and the main text never says so. The photos of robots in Fig.1 are especially misleading.
  • The only real-environment study (App.G) shows no meaningful difference. All returns are within about one s.d., and MarginPlan is not the best on Walker Walk or Finger Spin. The used world model is not named, the "M=24M=24 predictive samples" come from one fixed model.

Since the paper's claims are about planning with learned world models, there is no convincing evidence that MarginPlan helps there.

  1. The novelty is limited and the comparison is weak.
  • Stopping when the leader's lower bound exceeds all competitors' upper bounds look like the LUCB-type rule from best-arm identification, which the authors cite themselves. Max-residual conformal calibration is standard.
  • The continuation rule is similar to "value of computation" in metareasoning (Russell & Wefald 1991, Hay et al. 2012), which is not cited.
  • The baselines are three one-line threshold rules defined by the authors. None of the methods used as motivation (MACURA, DMVE, STEVE, MBPO) is compared.
  • The supplementary code contains only the core rule, so none of the tables can be reproduced easily.

The contribution should have been shown empirically against real competitors in real environment, which is not done.

  1. The presentation makes the method harder to follow than necessary.
  • The introduction interleaves motivation with related-work paragraphs that are too compressed to help the first understanding and are repeated in Sec. 2. It also uses the certificate before saying what is bounded.
  • Fig.1 draws one world model per action, while Eq.(1) and Alg.1 roll out every action through all models. L1,U1L_1, U_1 index the action in Fig.1 but the horizon in Eq.(3). Also, the standard deviation sh(a)s_h(a) reads like something about environment state for readers familiar to RL.
  • Figs.2,3,4 only repeat Tabs.1,2,5 in very uncommon, unnecessary playful styles (bubble areas, "q rail", "calls dial").
Questions for authors
  1. How are the world models of the "family" derived? In the main experiments they seem to be noise draws, see Eq.(12). Did you do the main comparisons with an actual learned ensemble, e.g. DreamerV3 or TD-MPC2?

  2. Table 1 reports only Fixed-2, 4, 8 and 12. Fixed-8 is the best of these and Fixed-12 is already worse, so the best fixed horizon may lie in between. Can you report at least Fixed-6, 7, 9 and 10? If one of them reaches .944, the accuracy gain of MarginPlan disappears.

  3. In App.G, the 24 samples come from one fixed model. Where does their spread come from, and why is that representing epistemic uncertainty? In my opinion, this is aleatoric only.

  4. The differences in Table 1 are small (.944 vs. .930 for Consistency-stop, with std .008 and .010 in Tab.11). Since the seeds are paired, can you report paired differences with confidence intervals or a significance test? Also for App.G?

  5. Why is there no comparison to a published adaptive-horizon method like MACURA, DMVE, STEVE, MBPO?

  6. Will you release the testbed, the baselines and the experiment scripts?

Reviewer_eVxkOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2(+1)
Soundness
1
Read review
Paper summary

The paper proposes a modification to Monte Carlo planning algorithms to adaptively decide the planning depth. This work introduces a method, using best arm identification methods from bandits, to decide when a deeper rollout is unlikely to change the current estimate of the best action. In principle, this means that the lower bound on the return for the current optimal action should be higher than the upper bound on the return of any other action. More specifically, the rollout depth is based on a mixture of increased confidence in action ranking, loss of compounded model accuracy and a penalty for compute cost. Practically, the confidence bound on action returns is based on a percentile of a multiple of the standard deviation on rollout returns from an ensemble of approximate transition functions This multiplication factor is assumed to be a constant and calibrated on a precomputed set of rollouts. The paper also proposes to use linear regression to predict the expected changes in these rollout-depth criteria when the rollout would be continued.

The submission evaluates the optimality of planning according to this dynamic depth on a controlled task where the transition function is known, and the ensemble artificially adds noise to each model to simulate information loss due to epistemic uncertainty at longer rollouts. Proportionally larger fractions of the q function are revealed for deeper rollouts. Results might demonstrate some improvement compared to ablations such as fixed rollout depths or simplified stopping criteria.

Critical strengths
  • The idea of a trade-off between the information gained by encountering more rewards vs. information lost due to compounding model error is an interesting view on the problem.
  • The paper performs ablations over a significant number of design choices/hyperparameters. The highly isolated experimental setting can be useful to gain insights into specific properties of the method.
  • Partial code snippets are provided, which provide some clarity and reproducibility.
Critical weaknesses

I see two fundamental weaknesses with the work: A) the problem setting of the main experiments does not reasonably transfer to the real problem setting that the paper tries to tackle, and B) the evaluation method contains significant weaknesses that do not provide confidence in the correctness of the results.

A) In the true problem setting that the paper situates itself (rollout depth in model-based RL), large parts of the difficulty of estimating an optimal rollout depth come from issues such as an imperfect q function, o.o.d. (off-policy) usage of the transition function, sampling variance, etc... These issues are fully ignored in the isolated setting of the experiments of this work. I do not believe that there are indications in the paper that lead to a reasonable argument that these results can be seen as a first step toward the full setting. This argument also transfers to the methodology of the works in several areas. Some of the core mistmatches are: 1) the experiments assume a smooth evidence delay where deeper unrolls reveal a larger portion of the return, this is a totally different setting from actual model-based RL rollouts where the final states in the unroll have a value function estimate; 2) the gap between q(a*,s) and the other q values is an assumption that seems difficult to motivate, as this in not always the case in an MDP and the method fully relies on this; 3) The work limits the rollouts based on predicted information gain, learned from previous data. But this contradicts the core motivation for planning, i.e., discovering a policy that unexpectedly provides an advantage to the current policy. The paper's own experiments seem to support this mismatch, as there is no statistically significant improvement for the experiments in the slightly less restricted setting of Appendix G.

B) Within the experimental setting of the paper, I am not convinced that the results support the claimed improvements sufficiently.

  • The work relies on seemingly extremely arbitrary parameters for the MDP. E.g., the gap ranges from 0.04->0.5 and the standard deviation from 0.04 to 0.34. The remaining q values are sampled from Unif[0.45, min(0.78, 1 − g)]. Why was this seemingly very specific setting chosen?
  • No comparison with any related work is provided, combined with the non-standard problem setting, it is impossible for the reader to asses how this approach compares to any other work.
  • On the wide gap, the method does not outperform fixed-8, on the narrow gap, it is the best method with a very small margin (0.5%), I do not believe the paper convincingly shows that the results are significant. No confidence metrics are provided. The question of what happens at small gaps (much smaller than 12%, which are realistic in many settings) remains.

Mainly due to these reasons, I believe the paper sits below the bar for acceptance and this is unlikely to be altered with small additions or improvements. I am open to changing my recommendation if the authors can demonstrate there has been a serious overarching mistake in my interpretation of the submission.

Here are some recommendations to improve the work:

  • Since the work is largely empirical and is a combination of existing methods, it is important to demonstrate that the method can scale to more common RL planning settings.
  • The work must compare these results to other state-of-the-art methods (such as InfoProp and others). Since it relies on many assumptions and approximations, these comparisons are needed to demonstrate that the approximations and assumptions were warranted and led to a useful approach.
  • Although notation and clarity are strongly personal, I believe that the clarity of the work needs to be improved. The underlying method is rather straightforward, but much time is needed to understand this from the paper. Some notational choices are confusing, such as using s for standard deviation (instead of state) and using C in equation 10 when it is also used for the action space.
  • Figures such as figure 4 and figure 2 are hard to interpret, I would recommend sticking to simple and common methods to represent your results.
Questions for authors
  • How is the continuation policy determined in the experiments in table 16?
  • Also in the experiments of table 16: what was the performance without any planning, how are the candidate actions selected? If the candidate actions are selected from the policy, what is the stddev of these actions? Are they actually significantly different?
  • Is the unroll depth actually strongly related to compute time, as these often happen in batch and are therefore limited by the deepest unroll? Do you have results on this?
  • How was \kappa_{comp} determined, can this not strongly and arbitrary bias the results?
  • How would the work be able to deal with a changing policy and an approximate/changing q function during training?
Reviewer_Tkm3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The manuscript studies an interesting topic. It proposes an adaptive planning framework for model-based reinforcement learning that determines imagination depth according to decision stability.

Critical strengths

The central idea is to construct calibrated return intervals for candidate actions and terminate planning when the lower return bound of the preferred action exceeds the upper bounds of all alternatives. When the action ranking remains uncertain, a lightweight continuation predictor estimates whether additional imagination can sufficiently reduce decision ambiguity to justify the increased model uncertainty and computational cost. The authors introduce a simultaneous conformal calibration procedure to provide a probabilistic guarantee on incorrect certification under exchangeability assumptions.

Critical weaknesses
  1. Limited novelty of the proposed decision-stability criterion and insufficient comparison with closely related methods. The underlying principle of stopping when confidence intervals separate competing actions has been well established in best-arm identification and sequential decision-making. The ideas are also closely related to value-of-information reasoning and metareasoning approaches to adaptive computation.
  2. The main experiments use artificially generated action returns, evidence delays, and model errors rather than actual trajectories produced by learned dynamics models. The experimental evaluation relies heavily on a synthetic testbed designed around the proposed mechanism. The baseline comparisons do not fully establish the advantage of decision-aware stopping.
  3. The theoretical guarantee is relatively narrow and depends on strong calibration assumptions. The authors should clarify the practical scope of the guarantee and investigate how calibration and certification behave under more realistic distribution shifts and nonstationary world models.
  4. Computational efficiency is evaluated using an incomplete cost model.
Questions for authors

please see the weaknesses above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.