- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors propose Reinforced Planning, a plan update framework that uses a frozen world model to predict an n-step rollout latent and use a cost function to evaluate the proximity to a goal state latent. The framework iteratively revises the multi-step action sequences, trained by differentiating the critic’s value through the world model.
Critical strengths
- The authors motivate the benefits of making the planning update rules themselves learnable, theoretically showing that different tasks may have different optimal hyperparameter configurations for search and planning, hence an expressive learned planner can outperform every hand-tuned planner.
Critical weaknesses
- The name “Reinforced Planning” is somewhat misleading. Apart from the TD critic, training is by analytic gradients through the model, rather than reinforcement learning. The framework is closer to learned optimization, and related work such as differentiable MPC (Amos et al., 2018) and learned optimizers in planning should be discussed in more depth.
- Despite the paper being about planning with world models, the literature review for world modeling is incomplete and incomprehensive. It only covers the Dreamer models and some JEPA-style world models, but disregards many MCTS approaches with learnable policies and latent state transitions (e.g. MuZero, EfficientZero), MPPI-like methods with a learned policy prior (e.g. TD-MPC2), and the entire progress in recent generative world models (e.g. DIAMOND, IRIS, Genie, Cosmos), which are not “latent” world models but are still relevant, especially considering their downstream applications for (emergent) planning. The literature review also lacks the lines of work on “emergent planning”, such as Value Iteration Networks and Value Prediction Networks. Hence, the claims that existing planners use “fixed search rules” (L043) and “hand-designed update rules” (L050) are overstated.
- The authors criticise the “hand-designed” aspect of other competing methods to differentiate their method. It is true that data-driven methods are desirable because rule-based methods cannot adapt to every new task domain and hinder performance. However, making every single component learnable result in an unconstrained solution (at which point it is just a model-free method. The RP1 method works only because the world model itself is not learnable and fixed), so there is a balance between the two. Whether to make a component learnable or fixed is a design / configuration choice, so using a “hand-designed optimizer” is not a strong argument to disregard an entire family of algorithms. The focus of the argument should be more on the actual planning and task-execution capabilities of said algorithms, and argue for a learnable planner only if that is the main reason for sub-optimal performance on particular benchmarks.
- The definition of a “world model” given in LL108-115 is limiting, since usually the latent representation is obtained by encoding a sequence of observations (maybe with a cut-off via sliding window) to reduce the ambiguity that arises from the partially observable nature of said observations.
- The notation is confounded because the authors have defined the hat notation as the predicted latent of the next state, so , which is not the same as the N-step rollout estimate of the latent defined in equation 1 (L113).
- The setting that this work considers where the goal-condition cost function is optimized is very limiting in scope, since it assumes that the immediate observation contains enough information to encode into latent that can determine whether the goal has been reached or not. This is true for navigation and object reaching tasks but is not the case when there are delayed rewards involved, or if taking a series of actions in the correct sequence gives the ultimate reward (e.g. reaching the goal prematurely would result in no rewards). The term “value function” is also not appropriate since they use a minimization objective. They should use the term “cost function” instead for consistency.
- None of the experiments actually test the planning ability of RP1, since all of the environments are simple RL environments that can be solved with a model-free policy. Real planning tasks involve replanning and utilization of persistent memory, where an agent explores a partially observable environment, updates its hypothesis based on new evidence, and modifies its plan in a way that is more drastic (tree-search-like backtracking) rather than a smooth adjustment to its proposed trajectory. The paper is better situated as a trajectory optimization work rather than work on learning to plan.
- References [51] and [52] are duplicates.
Questions for authors
- Can RP1 demonstrate planning behaviors that involve backtracking or major revisions to the plan given new evidence that refutes a previous hypothesis?
- Why are the results in parentheses (the one that uses the learned value critic) worse in Table 2 for Reacher? As discussed in Theorem 1, a distance / cost metric defined by a learned critic seems more sensible than an arbitrary Euclidean distance so this is counter-intuitive.