acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Scalable Planning Paradigms for High-Dimensional Control

Abstract

Test-time planning with learned world models has become a common practice in model-based reinforcement learning in recent years, yet mixed empirical results and a principled limitation arising from actor–planner divergence have also been reported. We revisit whether test-time planning should be a default component of high-dimensional control and re-examine policy improvement through background planning on imagined rollouts. By combining on-policy background planning with an annealed curiosity reward, our method outperforms the strongest prior high-dimensional control method under the matched environment interaction budget, improving mean performance by 8.57% on DMC and 9.99% on HumanoidBench. We further show that curiosity-driven exploration yields larger gains in high-dimensional and sparse-reward settings than either test-time planning or planner-action imitation. In contrast, the benefits of test-time planning and action imitation depend strongly on the task and temperature hyperparameters and are not consistently positive. We conduct an extensive hyperparameter sweep over planner and imitation temperatures. Across the 84 task–configuration cells involving a test-time planner, 21 and 15 yield relative improvements above 10% and relative degradations below -10%, respectively. Based on a correlational analysis, we formulate a causal hypothesis consistent with these observations: early intervention by the test-time planner, when the world model and value function remain underfitted, may degrade action quality and reduce experience diversity in the replay buffer. Together, these results suggest that, under our unified setting, on-policy background planning with curiosity-driven exploration provides a more reliable route to high-dimensional policy improvement than test-time planning.

Primary area:
reinforcement learning
Keywords:
high-dimensional continuous control, imagination-driven MBRL, test-time planning, curiosity-based intrinsic reward, curse of dimensionality

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_tLrEOpenReview ↗
Rating
3
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper revisits the value of test-time planning for continuous control. Its main agent combines a Gaussian-latent world model, imagined actor–critic learning, and annealed curiosity, obtaining strong final results on five DMC and six HumanoidBench tasks at four million environment steps. A planner and imitation temperature sweep shows that planning can help or hurt depending on the task. The results are interesting, but I have concerns about differences between the paper and code, the explanation of the planning diagnostics, and how broadly the comparison can be interpreted.

Critical strengths
  1. The final performance is strong across the evaluated tasks, particularly on humanoid-run and H1hand-hurdle. Five-seed results and learning curves help assess the gains and reveal the slower start of the proposed method (Table 2; Figure 5).
  2. The controlled framework includes planner-only settings as well as planner imitation. I appreciate that the full sweep shows large positive results alongside failures, which makes the task dependence clear (Figure 3).
  3. Using model prediction losses to supply imagined curiosity is a practical design. The large categorical/Gaussian difference is also a useful observation, even though its cause needs more investigation (Section 3.1; Figure 2).
Critical weaknesses
  1. I could not fully reconcile the method description with the anonymous code. Eq. 6 anneals curiosity using the median estimated extrinsic return, but the available implementation uses the midpoint of percentile bounds from a normalizer updated with curiosity-shaped returns. The imitation implementation also uses running normalization, detaches the policy standard deviation, and has a separate loss coefficient whose setting for the sweep is unclear. Please provide the resolved configurations for the reported runs and update the equations to match them. Unfortunately, these differences make it difficult to tell which precise recipe supports the main conclusions. They do not, on their own, show that the reported results are invalid. See the agent implementation, normalizer, and configuration.
  2. The interpretation of ESS needs to account for the normalization in Eq. 8. Scaling all candidate return differences by a positive constant leaves the weights and ESS unchanged unless the variance floor becomes active. A flatter absolute value scale therefore does not by itself explain higher ESS. Please examine return spread, standardized distribution shape, and floor activation. Higher trajectory ESS also need not mean more diverse actions or replay states, and gradient norms alone do not show conflicting updates. A delayed-planning experiment with model-error and diversity measurements would help test the proposed explanation. Until then, the causal language should stay explicitly tentative (Section 4.3; Figure 4).
  3. The planning conclusion is broader than the controlled comparison. The tested planner uses one iteration, horizon eight, policy-prior proposals, and stochastic trajectory selection. The temperature sweep is informative within that design, but it does not establish how another reasonable planning budget would behave. More directly, Figure 2 does not show planning added to the full curiosity agent. A small factorial comparison on representative high-dimensional tasks would clarify whether curiosity replaces planning or works well with it. Please state the reward used for planning and consider whether favorable sampled model outcomes affect candidate ranking (Sections 3.2 and 4.2; Table 3).
  4. The categorical-latent result needs a more cautious explanation. Figure 2 shows a severe failure of the tested categorical configuration, but a similar parameter count does not match latent capacity, regularization, or optimization. Please provide the full categorical settings and some evidence on prediction quality and latent usage. A modest capacity or KL/free-nat sensitivity check would help establish whether an information bottleneck is responsible. Without that evidence, the claim should be limited to the configurations tested (p. 2; Figure 2).
  5. The compute comparison remains incomplete. The four-million-step results support strong final performance under a fixed interaction budget, but the runtime is reported only for the proposed method. The optimizer-step entry in Table 3 also needs clarification against the ratio-based training implementation. Please report actual updates, imagined transitions, total training time, and action-selection latency, and identify the source and evaluation settings of the baseline results. This would make the claim about computational scalability easier to assess. See the training loop and configuration.
Questions for authors
  1. Which exact configurations produced the reported results, and how should the annealing and imitation equations be revised to match them?
  2. What explains the ESS changes after accounting for return standardization and the variance floor? Does delaying planning improve the degraded runs in the way the proposed mechanism predicts?
  3. What happens when planning is added to the full curiosity agent, including a modest search-budget or rollout-averaging check?
  4. Do the categorical results persist under a reasonable capacity and regularization comparison, and what measurements support the bottleneck interpretation?
  5. What are the actual training and inference costs for each method, and how were the baseline scores obtained and evaluated? Please clarify the optimizer-step count in Table 3.
Reviewer_nedxOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This work investigates the utility of background planning, test-time planning, and planner imitation for high-dimensional continuous control of robots. The authors develop and evaluate a model-based RL approach that leverages imagination, Gaussian latents, and annealed curiosity to enhance performance when compared to prior model-based approaches. Their results show that on average their methods outperform prior works in the DMC and HumanoidBench environments. Furthermore the authors provide ablations of the various components alongside a planner sensitivity analysis to show that test-time planning and planner imitation alone are not sufficient for consistent performance improvements.

Critical strengths

The paper provides useful insights into why various components of their method are necessary and what their relative utility is via their ablation studies. Of particular note is the demonstration that the choice of stochastic latents being used for imagination-based policy are quite important for performance. Specifically they show that using their Gaussian latent substantially improves relative to the categorical stochastic latents of prior approaches like were used in Dreamer.

Included in the results is a relatively expansive systematic empirical investigation of test-time planning and planner imitation across 84 task configurations. The investigation both shows that the gains or losses of including these methods are conditional depending on task and hyperparameters, but also delves deeper into what the potential underlying mechanisms are that drive these differences. I found this result particularly valuable in challenging the idea that the inclusion of test-time planning broadly improves model-based RL.

Critical weaknesses

The headline of 8.5% and 9.9% average improvements is very nice, but it doesn't tell the whole story when comparing BOOM to the author's work for both DMC and HumanoidBench tasks. Specifically the averaged gains are disproportionately driven by two tasks out of eleven, Humanoid run and H1hand-hurdle. In both cases their method drastically improves over BOOM, which is commendable, but if we exclude these two tasks, the averaged performance gains over the remaining nine tasks drop to less than 2%. As a result, I question whether the methods introduced in this work really provide as much utility as the authors claim and would consistently outperform prior work in a larger suite of high dimensional continuous control tasks. In addition to this, the central claims of this work all center around high dimensional tasks, but in Section 4.3, the authors use a set of generally lower dimensional continuous control tasks to conduct their planner sensitivity analysis without sufficient motivation as to why they diverge from high dimensional tasks.

The authors do show that their curiosity mechanism improves over their base method in the ablations, what isn't established is whether the curiosity formulation that they introduce as a contribution is better than existing approaches. There is some motivation for why to use these methods, but I am not convinced as to why experimentally we should prefer this over prior epistemic uncertainty-based, ensembled methods, or other signals. Furthermore, not including comparisons to other curiosity-based exploration methods seems like a major oversight in showing readers that their method is an improvement over more established curiosity methods.

Questions for authors

The planner sensitivity analysis in Section 4.3 is conducted on DMC tasks that are different from those used in Section 4.1. More importantly they are generally low dimensional and share no overlap with those in Section 4.1 where everything is high dimensional. What evidence is there to support the idea that the conclusions here generalize to the high dimensional environments that are the focus of this method and paper?

Why were more traditional curiosity mechanisms such as ensemble disagreement excluded from ablations and comparisons in the paper? What motivated the use of dynamics and reward prediction losses as curiosity signals over existing curiosity methods?

Given that two tasks drive most of your reported performance improvements over BOOM, what evidence is there to support the idea that your method would outperform BOOM on additional high dimensional continuous control tasks beyond those reported here?

Reviewer_GdzWOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies scalable planning paradigms for high-dimensional model-based reinforcement learning. It compares background planning, MPPI-style test-time planning, and advantage-weighted planner imitation within a unified world-model framework. The authors propose an annealed curiosity reward based on dynamics- and reward-prediction surprise, and evaluate these components on DMC and HumanoidBench under a matched 4M-step interaction budget. Their results suggest that curiosity-driven background planning provides the most consistent gains, while the benefits of test-time planning and planner imitation depend strongly on the task and temperature settings.

Critical strengths

The paper provides a useful empirical study of planning and exploration choices for high-dimensional model-based RL. Its comparisons across multiple control tasks suggest that curiosity-driven exploration can provide substantial gains, while the benefits of test-time planning and planner imitation depend on the task and hyperparameter settings. These findings offer practical insight into when these components may or may not be useful.

Critical weaknesses
  1. The paper’s positioning is not fully clear. It presents itself both as a systematic study of planning paradigms and as a method paper claiming strong benchmark performance and an annealed curiosity reward as a contribution. The study provides useful empirical observations, but the methodological contribution is less clearly established. The paper would benefit from distinguishing more clearly which components constitute the proposed method and which are mechanisms being evaluated.

  2. The curiosity component is not compared with closely related exploration methods. The ablation shows that the proposed curiosity reward improves over a version without intrinsic reward, but the paper does not compare it against methods such as Plan2Explore-style latent disagreement or learning-progress-based exploration, which the paper itself discusses. Thus, the experiments support the usefulness of adding curiosity in the tested setting, but do not establish the relative advantage or distinctiveness of this particular design.

  3. The benefits of test-time planning and planner imitation are highly configuration-dependent. The planner sweep shows both substantial gains and substantial regressions across tasks and temperatures, while the aggregate gains are influenced by a small number of tasks. The subsequent explanation of why some configurations degrade is based on post-hoc correlations, and the authors acknowledge that its causal mechanism remains unvalidated.

Questions for authors
  1. Could the authors clarify whether the primary contribution is intended to be a systematic empirical study of planning paradigms or a novel method for high-dimensional MBRL?

  2. Could the authors compare the proposed curiosity mechanism with closely related intrinsic-exploration methods, such as Plan2Explore-style latent disagreement or learning-progress-based exploration, under a matched backbone and interaction budget? This would help clarify its relative effectiveness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.