- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper investigates the consequences of training a value function using a policy, but improving and excecuting the policy using search within model-based RL (e.g MPPI). The authors provide theoretical support for this approach to be problematic due to the divergence of the two “policies” and suggest an approach (MBDPO) to resolve it. They suggest to learn an implicit energy function as a regularization term. With very extensive experiments and ablation studies the authors show strong performances across well known benchmarks and model-based baselines.
Critical strengths
- The regor of the experimental Section is very impressive - including all the ablations and supporting results in the appendix. The authors did a great job comparing to state-of-the-art methods in model-based RL, and using well known benchmarks in the field.
- The empirical results are very appealing - showing big gaps in some cases from strong baselines.
- The proposed approach is clearly supported by theoretical analysis. Although I did not go over all the proofs - the theorems make a lot of sense and give strong motivation for the empirical part.
Although I found a few weaknesses, listed below, I believe this paper studies an important topic and within it, an important problem. I have not found any critical problems that cannot be fixed in a revision, and so I recommend acceptance.
Critical weaknesses
- Theoretical Analysis in Section 3.1 is Not Specific to “Search” and Lacks Justification for Suboptimimality: The theoretical analysis conducted in Section 3.1 (Problem 1 and Problem 2) seem standard for off-policy / offline RL and should hold for any two policies and . Nothing in that Section is specific to MPPI or search policies. A missing piece that could tie everything together is showing that the search policy should diverge from the the base policy. This is non-trivial because search over an accurate world model can find the optimal policy directly.
- Comparison to Simpler KL-Regularized Search or Gaussian Policies: The authors did not discuss alternatives for enforcing KL regularization to the search induced policy. For example, KL regularization can be directly enforced in MPPI and gaussian policies.
Questions for authors
- How does Dreamer fit into your thesis regarding search-value misalignment (as Dreamer dows not use search)? Shouldn’t the preliminaries state that you target search-based world model methods (like TD-MPC)?
- The authors state in multiple locations in the paper that their approach lead to “scalability”. Can you clarify what scalability means in this context and how is it motivated / supported?
4 comments
W1 — Why search diverges from the base policy
We thank the reviewer for this observation. Problems 1 and 2 serve as the analytical lens of §3.1 and hold for any pair of policies by design. What ties them to search is the piece the reviewer asks for, namely why the search policy diverges from the base policy even with an accurate model. It is supplied by the three arguments below.
(i) Nothing anchors search to its proposal. One MPPI re-weighting step from a proposal gives , with
This is strictly positive whenever varies on the support of , and no term in the procedure bounds it: its size is set by the landscape of , including the errors of .
(ii) An accurate model does not remove the dependence on the learned value function. Search over an accurate model finds the optimal policy directly only when the horizon is long enough. These methods plan over a short horizon ( in TD-MPC2 and in our experiments), so the return in Eq. (7) still relies on the approximate terminal value . In TD-MPC2, is trained with TD targets whose next action comes from the policy prior. Search therefore evaluates at actions that the prior rarely produces, even when the dynamics are exact.
(iii) The divergence is observed. Cross-TD error is the mean absolute difference, at the same next state, between the TD target built with the planner's action and the TD target built with a policy-prior action; it is zero when search and prior select actions of equal value. For TD-MPC2 it stays well above zero throughout training (Figs. 2 and 17). Chang et al. (2026) reach the same diagnosis independently: comparable model accuracy does not determine whether search helps (their Table 1); the value target evaluates the policy network although the data are gathered by search, which makes the queried actions out-of-distribution and introduces overestimation (their §4.3 and Fig. 3); and actions selected by search change faster during training than those of the policy network (their Fig. 4, the measure we report as action drift).
W2 — KL-regularized MPPI and Gaussian policies
We thank the reviewer for this suggestion. We ran KL-regularized MPPI as a new baseline: it reproduces neither the diagnostics nor the return of MBDPO (table below; Figs. 39 and 40 in App. I.1 of the revised paper).
The baseline (KL-regularized MPPI). The KL regularizer toward the behavior policy is enforced directly in the MPPI of TD-MPC2. Because the density of the behavior policy is intractable, the regularizer is implemented as in MBDPO, through the behavior energy added to every sampled trajectory return; together with the entropy of the sampling distribution, this term yields the KL-regularized objective (Theorem 1). The regularizer and its weight are therefore the same as in MBDPO, and only the search procedure differs. The baseline is run with three seeds for 500k environment steps per task, and is labeled MPPI + Energy in Figs. 39 and 40.
Result.
- Cross-TD error (Fig. 39). MBDPO ends with the lowest cross-TD error on both tasks. The regularizer lowers the cross-TD error of MPPI on Acrobot Swingup, where it still remains clearly above MBDPO, and does not help on Pick Place, where KL-regularized MPPI ends above TD-MPC2.
- Action drift (Fig. 40). In the second half of training, MBDPO is below both MPPI-based agents on both tasks. KL-regularized MPPI stays at the level of TD-MPC2 on Button Press Wall and falls between TD-MPC2 and MBDPO on Acrobot Swingup only towards the end of training.
- Return. The regularizer does not improve the return of MPPI either. On Acrobot Swingup, KL-regularized MPPI does not exceed TD-MPC2, while MBDPO more than doubles it. On the two Meta-World tasks, which are close to saturation, MBDPO is the highest or tied for the highest.
| Method | Acrobot Swingup (return) | Button Press Wall (success, %) | Pick Place (success, %) |
|---|---|---|---|
| TD-MPC2 | 361.07 ± 76.02 | 93.33 ± 11.55 | 76.67 ± 23.09 |
| KL-regularized MPPI | 273.64 ± 148.19 | 100.00 ± 0.00 | 83.33 ± 14.70 |
| MBDPO | 831.50 ± 90.58 | 100.00 ± 0.00 | 90.00 ± 10.00 |
Scores at 500k environment steps for all methods; mean ± standard deviation over three seeds.
Enforcing KL regularization directly in MPPI is therefore not sufficient: its effect depends on the task, whereas diffusion-based optimization lowers both diagnostics on every task, and does so even without the regularizer (MBDPO at in Fig. 40).
Why a Gaussian search falls short. The optimum of the KL-regularized objective is a Gibbs distribution, which is in general non-Gaussian. MPPI refits a Gaussian to an elite set at every iteration, so adding the regularizer to its return changes which samples are favored but not the family it can represent. MBDPO targets the Gibbs distribution itself through diffusion. The existing Gaussian-framework methods, TD-M(PC)² (Lin et al., 2025), BMPC (Wang et al., 2025) and BOOM (Zhan et al., 2025), share this restriction.
Consistent with the scores in the paper. On the 80-task benchmark at comparable model size, MBDPO scores 75.29 without the regularizer and 79.15 with it, against 57.1 for TD-MPC2 (Figs. 9 and 20).
Q1 — Dreamer and the scope of the paper
As the reviewer suggests, the paper targets search-based world-model methods, and the Preliminaries now state this scope explicitly.
Scope. Search–value misalignment concerns methods that act by decision-time search over a latent world model while learning the value function with a separate, non-search policy: TD-MPC, TD-MPC2 and Newt, whose TD targets all take the next action from a policy prior.
Dreamer. Dreamer does not use search; its actor and critic are trained jointly on imagined rollouts of the same actor, so this mismatch does not arise. It appears in the paper as a world-model baseline (Fig. 3) and as background.
Q2 — Meaning of scalability
Definition. Scalability means that the performance of a single multi-task agent improves consistently as model capacity increases, at the fixed data and interaction budgets of the benchmark. This is the sense used by TD-MPC2 and Newt.
Motivation. A larger critic trained on multi-task data is queried by the planner on a growing set of out-of-distribution actions, so misalignment can limit the return on additional capacity. Anchoring the search to the data-supported distribution is designed to remove this limit.
Support.
- Offline, 80 tasks (Fig. 9). MBDPO scores 61.8, 72.1, 79.4, 87.2 and 89.5 from 1.7M to 340M parameters, against 16.0, 49.5, 57.1, 68.0 and 70.6 for TD-MPC2. It improves monotonically and is higher at every size, by 19 to 46 points; the 6M MBDPO model already exceeds the largest TD-MPC2 model.
- Online, MMBench, 200 tasks (Fig. 4 right). MBDPO improves from 0.33 to 0.50 from 2.4M to 1.82B parameters, and is competitive with Newt at matched size (0.45 versus 0.44 at about 20M parameters).
- Cost. Training time as a function of model size is in Tables 1(b) and 4.
The claim is therefore consistent improvement with capacity and a higher score at every size, with the largest margin for small models.
We thank the reviewer for the positive assessment of the experimental rigor and of the importance of the problem, and for the constructive suggestions. We respond to each weakness and question below. We hope these responses adequately address the reviewer's concerns and that the reviewer will be willing to increase the score accordingly.