acceptodds
Under review as a conference paper at ICLR 2027

Scaling World-Model Reinforcement Learning Through Diffusion Policy Optimization

Abstract

Model-based reinforcement learning (RL) can be effectively supported at scale through the use of world models. However, scaling such approaches remains fundamentally limited because policy improvement often relies on value functions induced by a separate non-search policy, creating training inconsistency and ultimately leading to suboptimal learning. To address this limitation, we propose Model-Based Diffusion Policy Optimization (MBDPO) in world models, a framework that unifies search and policy optimization through diffusion policy representations. Instead of constructing an explicit planner over a learned world model, we reformulate policy optimization as a diffusion process over searched trajectories in latent world models. In this view, we extract an implicit energy function from the collected transitions that anchors the policy, enabling MBDPO to refine the score field for policy optimization while mitigating misalignment. We evaluate MBDPO across several complementary settings, including offline pretraining, online learning, offline-to-online fine-tuning, and massively multi-task online learning. MBDPO achieves strong multi-task performance and exhibits consistent scaling behavior with increasing model size.

Primary area:
reinforcement learning
Keywords:
model-based reinforcement learning, world models, diffusion policies, policy optimization, offline reinforcement learning, multi-task reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_5iw7OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates the consequences of training a value function using a policy, but improving and excecuting the policy using search within model-based RL (e.g MPPI). The authors provide theoretical support for this approach to be problematic due to the divergence of the two “policies” and suggest an approach (MBDPO) to resolve it. They suggest to learn an implicit energy function as a regularization term. With very extensive experiments and ablation studies the authors show strong performances across well known benchmarks and model-based baselines.

Critical strengths
  1. The regor of the experimental Section is very impressive - including all the ablations and supporting results in the appendix. The authors did a great job comparing to state-of-the-art methods in model-based RL, and using well known benchmarks in the field.
  2. The empirical results are very appealing - showing big gaps in some cases from strong baselines.
  3. The proposed approach is clearly supported by theoretical analysis. Although I did not go over all the proofs - the theorems make a lot of sense and give strong motivation for the empirical part.

Although I found a few weaknesses, listed below, I believe this paper studies an important topic and within it, an important problem. I have not found any critical problems that cannot be fixed in a revision, and so I recommend acceptance.

Critical weaknesses
  1. Theoretical Analysis in Section 3.1 is Not Specific to “Search” and Lacks Justification for Suboptimimality: The theoretical analysis conducted in Section 3.1 (Problem 1 and Problem 2) seem standard for off-policy / offline RL and should hold for any two policies π\pi and β\beta. Nothing in that Section is specific to MPPI or search policies. A missing piece that could tie everything together is showing that the search policy should diverge from the the base policy. This is non-trivial because search over an accurate world model can find the optimal policy directly.
  2. Comparison to Simpler KL-Regularized Search or Gaussian Policies: The authors did not discuss alternatives for enforcing KL regularization to the search induced policy. For example, KL regularization can be directly enforced in MPPI and gaussian policies.
Questions for authors
  1. How does Dreamer fit into your thesis regarding search-value misalignment (as Dreamer dows not use search)? Shouldn’t the preliminaries state that you target search-based world model methods (like TD-MPC)?
  2. The authors state in multiple locations in the paper that their approach lead to “scalability”. Can you clarify what scalability means in this context and how is it motivated / supported?
4 comments

We thank the reviewer for the positive assessment of the experimental rigor and of the importance of the problem, and for the constructive suggestions. We respond to each weakness and question below. We hope these responses adequately address the reviewer's concerns and that the reviewer will be willing to increase the score accordingly.

W1 — Why search diverges from the base policy

We thank the reviewer for this observation. Problems 1 and 2 serve as the analytical lens of §3.1 and hold for any pair of policies by design. What ties them to search is the piece the reviewer asks for, namely why the search policy diverges from the base policy even with an accurate model. It is supplied by the three arguments below.

(i) Nothing anchors search to its proposal. One MPPI re-weighting step from a proposal p0p_0 gives π1∝p0exp⁡(G/λ)\pi_1 \propto p_0\exp(G/\lambda), with

DKL(π1∥p0)=Eπ1[G]λ−log⁡Ep0[eG/λ].D_{\mathrm{KL}}(\pi_1 \Vert p_0) = \frac{\mathbb E_{\pi_1}[G]}{\lambda} - \log\mathbb E_{p_0}\left[e^{G/\lambda}\right].

This is strictly positive whenever GG varies on the support of p0p_0, and no term in the procedure bounds it: its size is set by the landscape of GG, including the errors of Q^\hat Q.

(ii) An accurate model does not remove the dependence on the learned value function. Search over an accurate model finds the optimal policy directly only when the horizon is long enough. These methods plan over a short horizon (H=3H = 3 in TD-MPC2 and in our experiments), so the return in Eq. (7) still relies on the approximate terminal value Q^\hat Q. In TD-MPC2, Q^\hat Q is trained with TD targets whose next action comes from the policy prior. Search therefore evaluates Q^\hat Q at actions that the prior rarely produces, even when the dynamics are exact.

(iii) The divergence is observed. Cross-TD error is the mean absolute difference, at the same next state, between the TD target built with the planner's action and the TD target built with a policy-prior action; it is zero when search and prior select actions of equal value. For TD-MPC2 it stays well above zero throughout training (Figs. 2 and 17). Chang et al. (2026) reach the same diagnosis independently: comparable model accuracy does not determine whether search helps (their Table 1); the value target evaluates the policy network although the data are gathered by search, which makes the queried actions out-of-distribution and introduces overestimation (their §4.3 and Fig. 3); and actions selected by search change faster during training than those of the policy network (their Fig. 4, the measure we report as action drift).

W2 — KL-regularized MPPI and Gaussian policies

We thank the reviewer for this suggestion. We ran KL-regularized MPPI as a new baseline: it reproduces neither the diagnostics nor the return of MBDPO (table below; Figs. 39 and 40 in App. I.1 of the revised paper).

The baseline (KL-regularized MPPI). The KL regularizer toward the behavior policy is enforced directly in the MPPI of TD-MPC2. Because the density of the behavior policy is intractable, the regularizer is implemented as in MBDPO, through the behavior energy added to every sampled trajectory return; together with the entropy of the sampling distribution, this term yields the KL-regularized objective (Theorem 1). The regularizer and its weight are therefore the same as in MBDPO, and only the search procedure differs. The baseline is run with three seeds for 500k environment steps per task, and is labeled MPPI + Energy in Figs. 39 and 40.

Result.

  • Cross-TD error (Fig. 39). MBDPO ends with the lowest cross-TD error on both tasks. The regularizer lowers the cross-TD error of MPPI on Acrobot Swingup, where it still remains clearly above MBDPO, and does not help on Pick Place, where KL-regularized MPPI ends above TD-MPC2.
  • Action drift (Fig. 40). In the second half of training, MBDPO is below both MPPI-based agents on both tasks. KL-regularized MPPI stays at the level of TD-MPC2 on Button Press Wall and falls between TD-MPC2 and MBDPO on Acrobot Swingup only towards the end of training.
  • Return. The regularizer does not improve the return of MPPI either. On Acrobot Swingup, KL-regularized MPPI does not exceed TD-MPC2, while MBDPO more than doubles it. On the two Meta-World tasks, which are close to saturation, MBDPO is the highest or tied for the highest.
MethodAcrobot Swingup (return)Button Press Wall (success, %)Pick Place (success, %)
TD-MPC2361.07 ± 76.0293.33 ± 11.5576.67 ± 23.09
KL-regularized MPPI273.64 ± 148.19100.00 ± 0.0083.33 ± 14.70
MBDPO831.50 ± 90.58100.00 ± 0.0090.00 ± 10.00

Scores at 500k environment steps for all methods; mean ± standard deviation over three seeds.

Enforcing KL regularization directly in MPPI is therefore not sufficient: its effect depends on the task, whereas diffusion-based optimization lowers both diagnostics on every task, and does so even without the regularizer (MBDPO at η=0\eta = 0 in Fig. 40).

Why a Gaussian search falls short. The optimum of the KL-regularized objective is a Gibbs distribution, which is in general non-Gaussian. MPPI refits a Gaussian to an elite set at every iteration, so adding the regularizer to its return changes which samples are favored but not the family it can represent. MBDPO targets the Gibbs distribution itself through diffusion. The existing Gaussian-framework methods, TD-M(PC)² (Lin et al., 2025), BMPC (Wang et al., 2025) and BOOM (Zhan et al., 2025), share this restriction.

Consistent with the scores in the paper. On the 80-task benchmark at comparable model size, MBDPO scores 75.29 without the regularizer and 79.15 with it, against 57.1 for TD-MPC2 (Figs. 9 and 20).

Q1 — Dreamer and the scope of the paper

As the reviewer suggests, the paper targets search-based world-model methods, and the Preliminaries now state this scope explicitly.

Scope. Search–value misalignment concerns methods that act by decision-time search over a latent world model while learning the value function with a separate, non-search policy: TD-MPC, TD-MPC2 and Newt, whose TD targets all take the next action from a policy prior.

Dreamer. Dreamer does not use search; its actor and critic are trained jointly on imagined rollouts of the same actor, so this mismatch does not arise. It appears in the paper as a world-model baseline (Fig. 3) and as background.

Q2 — Meaning of scalability

Definition. Scalability means that the performance of a single multi-task agent improves consistently as model capacity increases, at the fixed data and interaction budgets of the benchmark. This is the sense used by TD-MPC2 and Newt.

Motivation. A larger critic trained on multi-task data is queried by the planner on a growing set of out-of-distribution actions, so misalignment can limit the return on additional capacity. Anchoring the search to the data-supported distribution is designed to remove this limit.

Support.

  • Offline, 80 tasks (Fig. 9). MBDPO scores 61.8, 72.1, 79.4, 87.2 and 89.5 from 1.7M to 340M parameters, against 16.0, 49.5, 57.1, 68.0 and 70.6 for TD-MPC2. It improves monotonically and is higher at every size, by 19 to 46 points; the 6M MBDPO model already exceeds the largest TD-MPC2 model.
  • Online, MMBench, 200 tasks (Fig. 4 right). MBDPO improves from 0.33 to 0.50 from 2.4M to 1.82B parameters, and is competitive with Newt at matched size (0.45 versus 0.44 at about 20M parameters).
  • Cost. Training time as a function of model size is in Tables 1(b) and 4.

The claim is therefore consistent improvement with capacity and a higher score at every size, with the largest margin for small models.

Reviewer_mym2OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

While world models enable scalable model-based RL, their practical scaling is limited by model bias and, more critically, a structural misalignment between search and value learning, where policy improvement relies on value functions from a separate, non-search policy. To address this, this paper proposes Model-Based Diffusion Policy Optimization (MBDPO), which unifies search and policy optimization via diffusion policy representations. Rather than using an explicit planner, MBDPO reformulates optimization as a diffusion process over latent trajectories, leveraging an implicit energy function to refine the score field while mitigating misalignment. Extensive evaluations across offline, online, offline-to-online, and massively multi-task settings demonstrate that MBDPO achieves strong multi-task performance and consistent scaling with model size.

Critical strengths
  1. The paper asks how world-model capabilities translate into control, rather than focusing only on prediction accuracy. Constructing action-score targets from model-based returns and learning a network to approximate this computation is a coherent and practically relevant design. Combining behavior anchoring with trajectory-level optimization also addresses the risk of exploiting unreliable value estimates. This is the main methodological consideration supporting acceptance.
  2. The evaluation includes independent single-task training, joint online learning across 200 tasks, offline multi-task scaling, and adaptation to previously unseen tasks. Beyond returns, the paper provides misalignment diagnostics, ablations of energy weights and sampling parameters, and runtime measurements. MMBench comparisons with Newt share demonstrations, frozen language representations, and online interaction budgets, and include approximately matched model capacities. These results warrant serious consideration of the method.
Critical weaknesses
  1. The bridge from behavior energy to the KL-optimal policy is not established. This concern directly affects whether Theorem 1 and Equation (13) describe the implemented algorithm. One issue is that Equation (9) uses actions from other states as negatives and argues that a sufficiently large and diverse dataset makes their distribution approximately uniform over the action space. But dataset size and diversity do not by themselves imply uniformity. If negatives follow a nonuniform q(a), the standard contrastive-discrimination interpretation involves a behavior-to-negative density ratio, rather than log β(a|z) alone. The sampling assumption, a density correction, or an approximation-error analysis is needed.
  2. Equation (4) is an upper bound on an operator discrepancy. Its growth with policy divergence does not prove that actual error necessarily increases, that systematic overestimation occurs, or that errors amplify over iterations. The lower bound in Equation (5) is not a monotonic-improvement guarantee for the implemented algorithm. Figure 2 is consistent with reduced misalignment, but does not independently establish its causal contribution to performance or scaling. Figure 20 supports moderate energy regularization, but removing it retains diffusion generation and score learning, leaving the contributions incompletely separated.
  3. Section 3.3 states that Monte Carlo world-model evaluation is used only for score-network training, while the MMBench description and Table 7 refer to a planner, elite samples, and policy-prior trajectories. These may be training-only components, but their mapping to algorithmic steps is unclear.
Questions for authors
  1. What distribution do the negatives in Equation (9) actually follow? How are nonuniform q(a) and the state-dependent energy offset addressed? Please specify conditions or corrections sufficient for Equation (75), given that adding c(zt) leaves the contrastive loss unchanged but can alter trajectory preferences.
  2. Please reconcile η, κ, and the KL multiplier, and derive the distribution sampled with η = 0.1 and κ = 0.5. If it differs from Equation (13), how should Theorem 1 and its equivalence claim be revised? Please distinguish finite-step sampling error from the exact-score statement.
  3. Please provide training and inference pseudocode matching the reported experiments, specifying whether elite samples and policy-prior trajectories are used during action generation and how next actions in TD targets are obtained. Please report denoising steps for each configuration and single-action generation latency under fixed hardware.
  4. Under comparable computational budgets, can the authors jointly report returns and misalignment diagnostics for original search with the same energy penalty, diffusion without the energy term, and full MBDPO? Please also specify independent seeds and uncertainty for the main large-scale results. If these comparisons already exist, please identify them.
7 comments

We thank the reviewer for the careful and constructive review, and for recognizing the practical relevance of the method and the breadth of the evaluation. We respond to each weakness and question below. We hope these responses adequately address the reviewer's concerns and that the reviewer will be willing to increase the score accordingly.

W1 — Behavior energy and Theorem 1

We thank the reviewer for raising this point. The energy in the reported experiments is trained with a binary contrastive loss and learns the log-ratio log⁡(β/n)\log(\beta/n), where nn is the negative-sampling density, rather than log⁡β\log\beta itself; the revised PDF states this explicitly and no longer relies on the uniformity argument. We provide the density correction and state the assumption explicitly below, and we ran an additional experiment that accounts for the negative-sampling density: performance is essentially unchanged (76.97 ± 1.39 versus 77.87 ± 1.43 on a 10-task subset; see the table under (i) below), indicating that the reported results are not sensitive to this difference.

What the energy learns. The experiments reported in the paper trained the energy EE with a binary contrastive loss, with replay actions as positives and negatives generated from the policy prior and a uniform distribution over the action space. Eq. (9) did not accurately describe this implementation; Eq. (9) and the accompanying description are corrected in the revised PDF. At the optimum of the binary loss,

−E⋆(z,a)=log⁡β(a∣z)−log⁡n(a∣z)+const,-E^{\star}(z,a) = \log\beta(a\mid z) - \log n(a\mid z) + \text{const},

where n(a∣z)n(a\mid z) is the negative density and the constant is global. The loss learns this log-ratio log⁡(β/n)\log(\beta/n) directly, and training requires neither knowing nor estimating nn; the negative density is needed only when the energy is interpreted as a behavior log-density, which requires adding log⁡n\log n back.

(i) Density correction. Although the reported experiments used the learned log-ratio directly, the density correction can be carried out exactly, and we ran it. The negatives are generated by us, so their density is under our control and no density estimation is required. Fixing the negative density nn to a known distribution and adding log⁡n(a)\log n(a) to −E-E before it enters the return is standard noise-contrastive estimation (Gutmann & Hyvärinen, 2010): at its population optimum, the corrected energy recovers the behavior log-density, −E⋆(z,a)+log⁡n(a)=log⁡β(a∣z)-E^{\star}(z,a) + \log n(a) = \log\beta(a\mid z), up to a global constant and with no state-dependent offset (Q1). In practice, the corrected energy is still globally standardized and clipped before it enters the return.

In this experiment, we kept the 21M model of the η\eta sweep (Fig. 20) frozen (world model, encoder, critic and policy prior), retrained only the energy network with the corrected objective, and evaluated planning with the Monte Carlo score of Eq. (15) at η=0.1\eta = 0.1 and κ=0.5\kappa = 0.5 on a fixed subset of 10 of the 80 tasks (5 DMControl and 5 Meta-World tasks, with 3 and 6 episodes per task, respectively). In the corrected objective, each action coordinate of a negative is drawn independently from a fixed distribution on [−1,1][-1, 1] that depends on neither the state nor the policy prior, so log⁡n(a)\log n(a) is available in closed form. The original energy was evaluated under the same setting. Both use three seeds.

Energy10-task meanDMControlMeta-World
Original77.87 ± 1.4365.73 ± 1.2990.00 ± 0.72
Corrected76.97 ± 1.3966.16 ± 1.0487.78 ± 1.92

± is the standard deviation across the three seeds. The difference between the two (−0.90) is within the seed-to-seed variation of either (1.43 and 1.39): performance is essentially unchanged, and accounting for the negative-sampling density brings the implementation in line with the theory at no measurable cost. Scores are averages over this 10-task subset and are not directly comparable with the 80-task averages in Fig. 20.

(ii) Assumption. Theorem 1 is restated with its assumption made explicit: the energy equals −log⁡β-\log\beta up to a global constant, which the corrected objective satisfies at its population optimum, and the partition function is finite (full statement under Q2 below).

Theorem 1 and the implemented algorithm. With the corrected energy, the restated Theorem 1 (stated in full under Q2 below) characterizes the target of the algorithm, π⋆∝exp⁡(G/κ)∏hβ(ah∣zh)bh/κ\pi^{\star} \propto \exp(G/\kappa)\prod_h \beta(a_h\mid z_h)^{b_h/\kappa}. With the energy used in the reported experiments, the same target holds with β\beta replaced by the ratio β/n\beta/n. As with any sampling-based planner, the practical estimator approximates this target with a finite number of samples and an elite set.

The main results hold without the energy term. Even without the energy term (η=0\eta = 0), MBDPO scores 75.29 on the 80-task benchmark, against 57.1 for TD-MPC2 at the comparable model size; the energy term adds a further gain (79.15 at η=0.1\eta = 0.1; Figs. 9 and 20).

W2 — Reading of Eqs. (4)–(5); attribution of the gains

Eqs. (4) and (5) are bounds, and their role is to motivate behavior anchoring. We thank the reviewer for this careful reading. Eq. (4) shows that the operator discrepancy is controlled by the policy divergence and vanishes when π=β\pi = \beta; Eq. (5) shows that the estimated improvement is offset by the same divergence. Both are used to motivate anchoring the search to β\beta, not as guarantees on the error dynamics, and we have reworded the corresponding sentences in §3.1 and §3.2 to make this role explicit. That the mismatch occurs in practice is an empirical observation, reported independently by Chang et al. (2026) and visible in the cross-TD error of TD-MPC2 (Figs. 2 and 17).

A new comparison separates the two contributions. Fig. 2 compares the two complete methods, and the η\eta sweep (Fig. 20) removes the energy term while keeping diffusion. To separate the two factors directly, as the reviewer suggests, we added a further arm, MPPI with the same energy penalty (Figs. 39 and 40 in App. I.1 of the revised paper; details in Q4). Adding the penalty to MPPI reproduces neither the diagnostics nor the return of MBDPO, and MBDPO attains low action drift even without the energy term. The reduction in the diagnostics is therefore attributable to the diffusion-based search rather than to the energy term, while the energy term contributes in return (75.29 at η=0\eta = 0 versus 79.15 at η=0.1\eta = 0.1, three seeds, Fig. 20).

W3 — Planner, elite samples and policy-prior trajectories

As the reviewer anticipates, these are training-time components, and the deployed policy uses none of them. §3.3 and App. G.3.1 now state the two score sources and the step in which each component is used.

Two score sources. MBDPO generates actions by reverse diffusion (Eq. 12) with one of two score sources.

  • Monte Carlo estimator (Eq. 15), used during training. At every denoising step, candidate action sequences are rolled out in the world model and re-weighted by their energy-regularized return. This is the method of §3.2 and the "planner" of App. G.3.1.
  • Distilled score network (Eq. 16), used at deployment. After training, a score network is fitted once to the scores of the Monte Carlo estimator. It performs no world-model rollout.

Mapping of the components.

Component in Table 7Used inRole
Monte Carlo samples (T=512T = 512), world-model rolloutsMonte Carlo estimatorEvaluate G~\tilde G for the candidates
Policy-prior trajectories (24)Monte Carlo estimator24 of the 512 candidates are proposed by the policy prior
Elite samples (64)Monte Carlo estimatorHighest-return candidates, retained before re-weighting
Score networkDeploymentAmortizes the estimator; uses none of the above

Why this design. The estimator depends only on the world model, the value function and the energy. Distilling the score network after training therefore does not affect data collection or value learning, and the final models give it a stationary target.

What each result uses. Learning curves are produced by the Monte Carlo estimator, so MBDPO and the baselines (TD-MPC2, Newt) are compared as planners that all use world-model rollouts at decision time. The aggregate scores in Figs. 3 and 4 are produced by the distilled score network, so the main results already reflect the deployed policy.

Reviewer_xUayOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes MBDPO, a diffusion policy optimization method built on a latent world model. A contrastive energy model is trained on replay data and used to regularize imagined trajectory returns. These returns provide Monte Carlo score targets for an amortized diffusion policy. The paper motivates this design through a mismatch between search and value learning and evaluates it across online control, offline multitask learning, offline-to-online adaptation, and MMBench.

Critical strengths
  1. The paper addresses a relevant failure mode in which planning exploits inaccuracies in a learned model or critic. Training a score network from imagined-trajectory evaluations offers a practical way to amortize trajectory optimization.

  2. The evaluation spans online control, offline multitask learning, adaptation, and MMBench, including approximately size-matched Newt comparisons. The results with zero energy weight also suggest a potentially useful diffusion-optimization effect beyond behavior anchoring.

Critical weaknesses
  1. The energy objective does not establish the trajectory-level behavior prior used in the analysis.

    Equation (9) uses actions from other states in the batch as negatives. These actions follow the dataset's action marginal, say q(a)q(a). A large and diverse dataset does not imply that this marginal is uniform, as assumed in lines 245–249. Under the usual population interpretation of this contrastive objective, the optimal score identifies

    −E(z,a)=log⁡β(a∣z)−log⁡q(a)+C(z),-E(z,a)=\log\beta(a\mid z)-\log q(a)+C(z),

    rather than the behavior log density alone. For example, if actions are independent of states but have a nonuniform marginal, positive and negative actions have the same conditional distribution. A score constant in the action is then optimal, although the behavior density is nonuniform.

    There is a separate normalization problem even if the negatives are uniform. Equation (9) is invariant to replacing E(z,a)E(z,a) with E(z,a)+c(z)E(z,a)+c(z) for any state-dependent offset. The trajectory objective is not: its return changes by −η∑h=tt+Hc(zh)-\eta\sum_{h=t}^{t+H}c(z_h). Future states depend on the candidate actions, so this term can change trajectory preferences. Appendix F.4 acknowledges the state-dependent constant but treats it as approximately constant along short trajectories (lines 1404–1416). Continuous dynamics and a short horizon do not constrain the variation of an otherwise unidentified offset. The behavior-prior interpretation therefore requires both a treatment of the negative-sampling density and a normalization across states.

  2. The target in Equation (11) differs from the distribution in Theorem 1.

    Even granting exact recovery of −E=log⁡β-E=\log\beta, the score weights in Equation (11) target

    \propto \exp(G/\kappa) \left[\prod_{h=t}^{t+H}\beta(a_h\mid z_h)\right]^{\eta/\kappa},$$ with states determined by the world model. Equation (13) instead uses the behavior product to the first power. The stated configurations have $\eta=0.1$ and $\kappa=0.5$ (Tables 5 and 7), giving an exponent of 0.2. As a simple check, with one action step, zero return, and behavior probabilities $(0.9,0.1)$, the two expressions give approximately $(0.608,0.392)$ and $(0.9,0.1)$, respectively. Appendix F.4 derives the regularized optimum using $\eta$, then replaces the temperature by $\kappa$ at the end of the proof. This replacement does not preserve the claimed equivalence while the energy coefficient remains $\eta$. Independent entropy and energy coefficients can define a valid objective, but it is necessary to derive that objective and revise Theorem 1 accordingly. This is a discrepancy in the target distribution even before finite-sample score estimation or diffusion discretization enters the analysis.
Questions for authors
  1. Is the negative-sampling density corrected in the implementation, and how are the state-dependent energy offsets normalized before trajectory returns are compared?

  2. What objective is optimized with η=0.1 and κ=0.5, and how should Theorem 1 change to match that objective?

5 comments

We thank the reviewer for the constructive review, and for recognising the practical relevance of the method and the breadth of the evaluation. We respond to each weakness and question below. We hope these responses adequately address the reviewer’s concerns and that the reviewer will be willing to increase the score accordingly.

W1 / Q1 — Energy objective, negative-sampling density, and state-dependent offsets

We thank the reviewer for raising this point. The energy in the reported experiments is trained with a binary contrastive loss and learns the log-ratio log⁡(β/n)\log(\beta/n) (derived below), where nn is the negative-sampling density, rather than log⁡β\log\beta itself; the revised PDF states this explicitly and no longer relies on the uniformity argument. We also ran an additional experiment that accounts for the negative-sampling density: performance is essentially unchanged (76.97 ± 1.39 versus 77.87 ± 1.43 on a 10-task subset; details below), indicating that the reported results are not sensitive to this difference.

Implemented objective. The reviewer's analysis applies to the in-batch objective stated in Eq. (9), and the dependence of the energy on the negative-sampling density carries over to the objective we implemented. The experiments reported in the paper used a binary contrastive loss to train the energy EE, with replay actions as positives and negatives generated from the policy prior and a uniform distribution over the action space. Eq. (9) did not accurately describe this implementation; Eq. (9) and the accompanying description are corrected in the revised PDF.

No per-state normalization is needed. Eq. (9) compares actions only within a state, which is why it is invariant to E→E+c(z)E \to E + c(z). A binary loss classifies each state–action pair against an absolute label and is not invariant to such shifts: at its optimum,

−E⋆(z,a)=log⁡β(a∣z)−log⁡n(a∣z)+const,-E^{\star}(z,a) = \log\beta(a\mid z) - \log n(a\mid z) + \text{const},

where n(a∣z)n(a\mid z) is the negative density and the constant is global. There is therefore no unidentified per-state offset. The only remaining state dependence is through n(a∣z)n(a\mid z), which the density correction below removes. The only processing applied to the energy is a global standardization followed by clipping, the same map for all states. The short-horizon argument of App. F.4 is therefore no longer needed and is removed.

Accounting for the negative-sampling density. As the expression above shows, the binary loss learns the log-ratio log⁡(β/n)\log(\beta/n) directly, and training requires neither knowing nor estimating nn; the negative density is needed only when the energy is interpreted as a behavior log-density, which requires adding log⁡n\log n back. Although the reported experiments used the learned log-ratio directly, the density correction can be carried out exactly, and we ran it. Unlike the action marginal q(a)q(a) in Eq. (9), the negatives here are generated by us, so their density is under our control and no density estimation is required. Fixing the negative density nn to a known distribution and adding log⁡n(a)\log n(a) to −E-E before it enters the return is standard noise-contrastive estimation (Gutmann & Hyvärinen, 2010): at its population optimum, the corrected energy recovers the behavior log-density, −E⋆(z,a)+log⁡n(a)=log⁡β(a∣z)-E^{\star}(z,a) + \log n(a) = \log\beta(a\mid z), up to a global constant. This provides the behavior-density interpretation assumed in Theorem 1; in practice, the corrected energy is still globally standardized and clipped before it enters the return.

Additional experiment: performance is essentially unchanged. We kept the 21M model of Fig. 20 frozen (world model, encoder, critic and policy prior), retrained only the energy network with the corrected objective, and evaluated planning with the Monte Carlo score of Eq. (15) at η=0.1\eta = 0.1 and κ=0.5\kappa = 0.5 on a fixed subset of 10 of the 80 tasks (5 DMControl and 5 Meta-World tasks, with 3 and 6 episodes per task, respectively). In the corrected objective, each action coordinate of a negative is drawn independently from a fixed distribution on [−1,1][-1, 1] that depends on neither the state nor the policy prior, so log⁡n(a)\log n(a) is available in closed form. The original energy was evaluated under the same setting. Both use three seeds.

Energy10-task meanDMControlMeta-World
Original77.87 ± 1.4365.73 ± 1.2990.00 ± 0.72
Corrected76.97 ± 1.3966.16 ± 1.0487.78 ± 1.92

± is the standard deviation across the three seeds. The difference between the two (−0.90) is within the seed-to-seed variation of either (1.43 and 1.39): accounting for the negative-sampling density brings the implementation in line with the theory at no measurable cost in performance. Scores are averages over this 10-task subset and are not directly comparable with the 80-task averages in Fig. 20.

The main results hold without the energy term. Even without the energy term (η=0\eta = 0), MBDPO scores 75.29 on the 80-task benchmark, against 57.1 for TD-MPC2 at the comparable model size; the energy term adds a further gain (79.15 at η=0.1\eta = 0.1; Figs. 9 and 20).

W2 / Q2 — Objective with independent η\eta and κ\kappa; revised Theorem 1

We thank the reviewer for this careful derivation. With independent η\eta and κ\kappa, Eq. (11) indeed targets the distribution the reviewer derives, and, as the reviewer anticipates, this defines a valid and more general objective: η\eta sets the strength of behavior anchoring and κ\kappa the overall temperature, with the original Theorem 1 recovered as the special case κ=η\kappa = \eta. We have restated Theorem 1 for this general case and corrected the last step of its proof, which had replaced η\eta by κ\kappa. The algorithm and all reported results are unchanged.

Revised Theorem 1. Let βH(at:t+H∣zt)=∏hβ(ah∣zh)\beta_H(a_{t:t+H}\mid z_t) = \prod_h \beta(a_h\mid z_h) under the latent dynamics, and let the energy enter the return with step coefficients bh≥0b_h \ge 0, i.e. G~=G−∑hbhE(zh,ah)\tilde G = G - \sum_h b_h E(z_h, a_h). Assume that the energy equals −log⁡β-\log\beta and that the partition function is finite. Then the clean distribution whose noised marginals have the score in Eq. (10) is

π⋆(at:t+H∣zt)∝exp⁡(G/κ)∏hβ(ah∣zh)bh/κ,\pi^{\star}(a_{t:t+H}\mid z_t) \propto \exp(G/\kappa)\prod_h \beta(a_h\mid z_h)^{b_h/\kappa},

and π⋆\pi^{\star} is the unique maximiser of J(π)=Eπ[G]+∑hbhEπ[log⁡β(ah∣zh)]+κH(π)J(\pi) = \mathbb E_\pi[G] + \sum_h b_h \mathbb E_\pi[\log\beta(a_h\mid z_h)] + \kappa \mathcal{H}(\pi). The proof is one line: J(π)=−κKL(π∥π⋆)+constJ(\pi) = -\kappa \mathrm{KL}(\pi \Vert \pi^{\star}) + \text{const}. With bh=ηb_h = \eta at every step,

J(π)=Eπ[G]−ηKL(π∥βH)+(κ−η)H(π).J(\pi) = \mathbb E_\pi[G] - \eta \mathrm{KL}(\pi \Vert \beta_H) + (\kappa - \eta) \mathcal{H}(\pi).

Setting κ=η\kappa = \eta removes the entropy term and recovers Eq. (13).

Objective at η=0.1\eta = 0.1 and κ=0.5\kappa = 0.5. Under this assumption, the algorithm maximises

J(π)=Eπ[G]−0.1KL(π∥βH)+0.4H(π),J(\pi) = \mathbb E_\pi[G] - 0.1 \mathrm{KL}(\pi \Vert \beta_H) + 0.4 \mathcal{H}(\pi),

whose maximiser is π⋆∝exp⁡(G/0.5)βH0.2\pi^{\star} \propto \exp(G/0.5) \beta_H^{0.2}: a KL-regularised objective with temperature 0.5 towards the tempered behaviour prior βH0.2\beta_H^{0.2}. As with any sampling-based planner, the practical estimator approximates this target with a finite number of samples and an elite set.

The conclusions do not depend on the exponent. In the η\eta-sweep of Fig. 20, performance on the 80-task benchmark is stable across a tenfold range of the energy weight (78.84–81.39 for η\eta between 0.1 and 1.0), and remains at 75.29 with η=0\eta = 0.

Exact target versus estimation. The revised theorem characterises the target distribution itself and involves neither the number of Monte Carlo samples nor the number of diffusion steps, so the discrepancy the reviewer identifies is resolved at the level of the target. The errors from Monte Carlo score estimation, score-network regression and NN-step discretisation are treated separately.

References

[1]. Gutmann, M. and Hyvärinen, A. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics (AISTATS), PMLR 9:297–304, 2010.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.