acceptodds
Under review as a conference paper at ICLR 2027

Reinforced Planning with Latent World Models

Abstract

Humans solve complex problems by constructing plans and mentally simulating their outcomes with an internal model of the world. Machine learning has made substantial progress in learning world models that predict the consequences of action sequences, yet the procedures used to plan with these models remain largely hand-designed. Most planners rely on fixed search or optimization rules; approaches that learn aspects of search typically imitate a predefined optimizer or use planning to train an amortized policy, rather than learning how to iteratively improve a complete multi-step plan. We introduce Reinforced Planning, a method for learning the plan-update rule itself through a pretrained world model. Our implementation, RP1, learns a critic for imagined outcomes by temporal-difference learning, and a neural operator that iteratively revises multi-step action sequences, trained by differentiating the critic's value through the world model. RP1 requires no optimizer demonstrations and is trained without environment interaction, from imagined rollouts of a pretrained differentiable latent world model; environment episodes are used only for checkpoint selection. Across visual navigation, arm reaching, and robotic manipulation on two world-model backbones, RP1 matches or exceeds hand-designed and learned planners, achieving near-perfect success in several settings while using fewer world-model rollouts than the strongest alternative (CEM) and planning up to faster under concurrent planners inference.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Model-based Reinforcement Learning, World Models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_7hgiOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose Reinforced Planning, a plan update framework that uses a frozen world model to predict an n-step rollout latent and use a cost function to evaluate the proximity to a goal state latent. The framework iteratively revises the multi-step action sequences, trained by differentiating the critic’s value through the world model.

Critical strengths
  • The authors motivate the benefits of making the planning update rules themselves learnable, theoretically showing that different tasks may have different optimal hyperparameter configurations for search and planning, hence an expressive learned planner can outperform every hand-tuned planner.
Critical weaknesses
  • The name “Reinforced Planning” is somewhat misleading. Apart from the TD critic, training is by analytic gradients through the model, rather than reinforcement learning. The framework is closer to learned optimization, and related work such as differentiable MPC (Amos et al., 2018) and learned optimizers in planning should be discussed in more depth.
  • Despite the paper being about planning with world models, the literature review for world modeling is incomplete and incomprehensive. It only covers the Dreamer models and some JEPA-style world models, but disregards many MCTS approaches with learnable policies and latent state transitions (e.g. MuZero, EfficientZero), MPPI-like methods with a learned policy prior (e.g. TD-MPC2), and the entire progress in recent generative world models (e.g. DIAMOND, IRIS, Genie, Cosmos), which are not “latent” world models but are still relevant, especially considering their downstream applications for (emergent) planning. The literature review also lacks the lines of work on “emergent planning”, such as Value Iteration Networks and Value Prediction Networks. Hence, the claims that existing planners use “fixed search rules” (L043) and “hand-designed update rules” (L050) are overstated.
  • The authors criticise the “hand-designed” aspect of other competing methods to differentiate their method. It is true that data-driven methods are desirable because rule-based methods cannot adapt to every new task domain and hinder performance. However, making every single component learnable result in an unconstrained solution (at which point it is just a model-free method. The RP1 method works only because the world model itself is not learnable and fixed), so there is a balance between the two. Whether to make a component learnable or fixed is a design / configuration choice, so using a “hand-designed optimizer” is not a strong argument to disregard an entire family of algorithms. The focus of the argument should be more on the actual planning and task-execution capabilities of said algorithms, and argue for a learnable planner only if that is the main reason for sub-optimal performance on particular benchmarks.
  • The definition of a “world model” given in LL108-115 is limiting, since usually the latent representation ztz_t is obtained by encoding a sequence of observations o1:to_{1:t} (maybe with a cut-off via sliding window) to reduce the ambiguity that arises from the partially observable nature of said observations.
  • The notation z^N\hat{z}_N is confounded because the authors have defined the hat notation as the predicted latent of the next state, so z^N=hϕ(zN−1,aN−1)\hat{z}_N = h_\phi(z_{N-1}, a_{N-1}), which is not the same as the N-step rollout estimate of the latent defined in equation 1 (L113).
  • The setting that this work considers where the goal-condition cost function Vψ(zt,zg)V_\psi(z_t, z_g) is optimized is very limiting in scope, since it assumes that the immediate observation oto_t contains enough information to encode into latent ztz_t that can determine whether the goal has been reached or not. This is true for navigation and object reaching tasks but is not the case when there are delayed rewards involved, or if taking a series of actions in the correct sequence gives the ultimate reward (e.g. reaching the goal prematurely would result in no rewards). The term “value function” is also not appropriate since they use a minimization objective. They should use the term “cost function” instead for consistency.
  • None of the experiments actually test the planning ability of RP1, since all of the environments are simple RL environments that can be solved with a model-free policy. Real planning tasks involve replanning and utilization of persistent memory, where an agent explores a partially observable environment, updates its hypothesis based on new evidence, and modifies its plan in a way that is more drastic (tree-search-like backtracking) rather than a smooth adjustment to its proposed trajectory. The paper is better situated as a trajectory optimization work rather than work on learning to plan.
  • References [51] and [52] are duplicates.
Questions for authors
  • Can RP1 demonstrate planning behaviors that involve backtracking or major revisions to the plan given new evidence that refutes a previous hypothesis?
  • Why are the results in parentheses (the one that uses the learned value critic) worse in Table 2 for Reacher? As discussed in Theorem 1, a distance / cost metric defined by a learned critic seems more sensible than an arbitrary Euclidean distance so this is counter-intuitive.
Reviewer_6e1mOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes RP1, a learned reinforcement planner built on the Actor-Critic paradigm. The critic is trained offline with TD learning to estimate the remaining cost toward the target from current state, whereas the actor optimizes explicit action sequences. During training, the world model and critic are frozen. Candidate trajectories are rolled out by the pretrained world model to in latent space, giving values and gradients to guide iterative trajectory refinement. The proposed method avoids online environment interaction and enables the planner to learn trajectory update. Authors evaluated RP1 on TwoRoom, Reacher, and OGBench Cube, with LeWorldModel (LeWM) and PLDMtwo as world-model backbones and reported promising results.

Critical strengths
  1. The claim and proof that latent-space distances are not equivalent to true temporal reachability distances is insightful. Many world model works are learning a critic directly from the latent states, assuming that latent-space distances guarantee temporal reachability distances in real world.
  2. The proposed RP1 planner is modular and pluggable. It can be attached to any pre-trained world model and goal-conditioned value function.
Critical weaknesses
  1. It seems like the authors didn't claim their main contribution of their work clearly enough as a research work instead of engineering. At the end of the introduction section, the authors should list out their novel contribution in this work.
  2. The critic only evaluates terminal latent states and lacks explicit supervision over intermediate constraints, so trajectories may contain invalid intermediate states.
  3. RP1 does not deal with epistemic uncertainty of critic in a pessimistic manner. It trusts gradients uniformly across in-distribution and out-of-distribution regions. Thus, the planner may come up with overconfident plans.
  4. The authors claimed that RP1 takes the gradient as input explicitly and can use gradient information adaptively, but there is no ablation experiments supporting this.
Questions for authors
  1. Does RP1 suffer from mode collapse compared with CEM and other methods that samples more candidates? If not, how does RP1 deal with it?
  2. In the world model section, you denote the observation encoder as EϕE_{\phi}, and the transition model as hϕh_{\phi}, does this mean they share a same set of parameters? Usually, the encoder and the transition model have their own set of parameters.
  3. Could you please add ablation experiments w/wo the gradient of value w.r.t. the trajectory as input?
Reviewer_RRjPOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper is about learning an update rule, essentially learning an algorithm for planning on a frozen pre-trained latent world model. They do so in a goal-conditioned setting. They also use a critic that is conditioned on the goal, and they parameterize their update rule using a residual MLP.

That work is very much in line with the line of work on learning optimizers, the line of work on learning gradient descent by gradient descent, for example, the work by Luke Metz. It's revisiting those ideas under the lens of the more recent interest around world models.

The proposed algorithm is presented in synthetic domains, OG bench, and a two-room domain. In that sense, experiments are not very strong or particularly challenging.

They compare against CEM, MPPI, and direct backpropagation through the model, and show improvement with their methods. There is some theory that tries to motivate the form of the update rule. Essentially, it's something about the fact that, in that latent space, the latent space need not be representing a Euclidean metric. I think the theorem is trying to say something about the fact that using a linear parameterization for the update rule is just as good. Although, to be honest, I'm not quite sure of the relevance of that theorem. I think it's more aesthetic rather than a foundational piece of the paper.

Critical strengths

I think the paper is doing a good job at using baselines and properly comparing against other algorithms, so this is definitely a good strength. They use five baselines with two objectives each, two backbones. They also use held out validation sets when scoring their checkpoints when training RP1.

Critical weaknesses

I think the main problem is that the paper doesn't quite show that RP1 learn-and-update rule. I think there could be a simpler interpretation of what RP1 ends up learning.

The first one is that RP1 could simply end up learning a policy that maps the start and the goal to an actual plan. Since the first input that it receives is the gradient with the zero plan, it's already somehow encoding the task. And if that's the case, it kind of goes against the idea that RP1 is doing improvement to a plan; the counterpart to the amortization approach that the paper talks about and uses as motivation.

It could also be the case that RP1 ends up just doing gradient descent, where it ends up learning a step size. I had a look at the planner weights that were released to see if I could investigate this behavior experimentally. And the conclusion looks like there would be a natural baseline with green descent that perhaps hasn't been considered just yet. What I try to see is what that correction network is actually doing in RP1. Because of the MLP residual form, you would expect a natural solution would be to learn to basically just output the identity if you were to feed it the gradient, right?

If RP1 were to be plain gradient descent, you'd expect that it would output a correction where it's the actual input gradient, but where you'd flip the sign and you'd scale it by a step size. It somehow looks like, when you probe that network, at the first step, it ignores mostly the gradient you give it as input, except for the sum over time across all the action coordinates. It somehow looks like it outputs a plan that repeats the same action at every step.

Overall, it looks like, from my limited investigation, it's learning a gradient descent update, but restricted to plans that have a very simple structure: you'd keep the same constant action, and you'd be learning the step size. That would be something, I guess, I would expect for this simple four room navigation task, especially for these short tasks where you output like 25 actions and you're moving in 2D coordinate space. Since you're replanning, even if that's a restricted plan, you can compensate along the way.

Anyway, that could be a good baseline to try against and kind of probe at the structure of the learned update rule. So there might also be an issue in the way that the baselines are computed. In particular, there seems to be a discrepancy in what the caption says. The table caption says that the baseline all use the critic that RP1 is actually deploying in its experiment. But what I understand of the release code is that it seems to be actually using an offline critic for the baseline, while it allows for RP1 to use a critic that is trained along with the planner. We're not quite comparing apples to apples here. In fact, it looks like, from the numbers reported in the papers, that when CM uses an offline critic, the performance goes from 98 to 88. This is in a setting where RP1 would otherwise reach 100.

I think the right way to do things is actually either to take the cold-trained critic that RP1 learns and use it for the baselines, or you could also do the other way around and run RP1 with one of those offline critics. That way, you could really compare all these methods on the same basis, and that would help you also disentangle the effect of a learned planner versus the effect of learning a critic online or offline.

Questions for authors

I think it'd be interesting if you can report numbers with RP1. You could just take one refinement step, but feed it the start and goal as goal states as inputs. You could also compute the cosine between RP1's update and the negative gradient across something like 1, 2, 4, 8, or 16 steps of gradient descent, and add them with the tuned step size.

Now, more importantly, I think it is the comparison using the critic. Make sure to check what happens as you go from an offline critic in RP1 or vice versa.

Now, there's another important experimental detail, too, that maybe shouldn't be overlooked: it looks like the RP1 score comes from the checkpoints. The score that we see in the paper is actually obtained while essentially tuning RP1 on the simulator itself, and the baselines don't get to have the same privileged access to the simulator.

From what I understand, the simulator access gives RP1 access to something like 43,200 episodes, which is a pretty big number, and of course gives the method a chance to be tuned better than the baselines. Maybe there'd be a way to actually try to compare against the same budget of tuning for the baselines and see which one does well.

Finally, going back to the point that maybe, at least in the room domain, RP1 outputs a constant-velocity plan. If you happen to be replanning often, even with a crude, very imperfect, and limited plan, you would still probably be able to compensate and account for the limitation of that crude plan. I'm not sure if RP1 is actually benefiting from this, and we could see a difference if you were to instead use RP1 and execute the whole plan rather than doing it every chunk.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.