acceptodds
Under review as a conference paper at ICLR 2027

DPMPC: Learning Diffusion Priors for Model Predictive Control

Abstract

Model predictive control (MPC) has achieved notable success in continuous control. MPC often uses a policy prior to propose candidate trajectories and to learn the value function. When the prior differs from the executed planner, this actor divergence biases the value estimates of candidate trajectories. The commonly used Gaussian prior cannot represent the non-Gaussian action distributions induced by stochastic planning, leaving a residual divergence. To address this, we propose DPMPC (Diffusion Priors for Model Predictive Control), which trains an expressive diffusion prior to match the planner's action distribution. Generalized lazy reanalyze estimates this distribution from multiple recent MPC realizations stored per replay state, without rerunning MPC at every policy update. An advantage-weighted score-matching objective then fits the prior to this estimate while emphasizing high-value actions. DPMPC achieves state-of-the-art asymptotic performance and sample efficiency across 14 continuous-control tasks from DMControl, MyoSuite, and HumanoidBench. Further analysis shows that DPMPC yields more accurate value estimates than Gaussian-prior baselines.

Primary area:
reinforcement learning
Keywords:
Reinforcement learning, model predictive control, diffusion policy

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_MP6hOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes DPMPC, which uses a diffusion policy prior to guide MPPI planning and reduce the mismatch between the prior and the executed planner. The prior is trained through advantage-weighted denoising using periodically refreshed planner distributions stored in replay. Benchmarking experiments show improved performance, sample efficiency, and lower value-estimation errors.

Critical strengths
  1. The method is straightforward and well motivated, and the paper is clearly written.

  2. The empirical results are good across the three benchmarks, with particularly notable gains on HumanoidBench. I like the value-estimation analysis, it provides useful evidence beyond the main performance comparisons. But I think you might need to check TD-MPC-square since it also provides a much simpler and effective solution to reduce value estimation error.

  3. Using a diffusion policy as an MPPI prior is a sensible design choice. Policy-guided planning centered on a single nominal behavior can struggle to discover alternative behavioral branches efficiently through Gaussian perturbations alone (like Residual-MPPI). A multimodal diffusion prior offers a way to propose distinct high-value action candidates directly, potentially broadening the planner's search beyond local variations of one behavior. I find this motivation compelling independently of the value-estimation argument. (However, I think this is not discussed much and might lead to a methodolody-wise limitation below)

Critical weaknesses

Limitations

  1. The conceptual advance feels incremental. Replacing a Gaussian policy prior with a diffusion model is a natural extension of expressive policy learning. The empirical gains are good but not a leap, I am not yet convinced that they sufficiently justify the additional training machinery and iterative sampling cost. The paper would be more compelling if it established a planning capability enabled by this design beyond improved benchmark performance.

  2. As in the strength, my main concern is how the multimodal prior interacts with MPPI's aggregation step. Section 2.1 describes a single Gaussian search distribution and execution of its optimized mean during evaluation. I could not find an explanation of how candidates from distinct behavioral branches are aggregated without producing an undesirable compromise between modes. For example, in a classic PushT scenario with two incompatible but similarly valuable action modes, their weighted mean can be a poor action even when both modes are useful (Please correct me if I have missed a relevant mechanism; if one exists, it should be made explicit in the main text.)

This also limits what I take away from the experimental analysis. A focused analysis or controlled experiment with incompatible modes, tracking candidates through aggregation and final action selection, would be more informative to me than further parameter sweeps. If cross-mode averaging can produce poor actions and no mechanism addresses it, this would be a substantive methodological limitation. Conversely, a convincing resolution and evidence of effective planning across distinct branches would materially strengthen the contribution and could move my assessment toward acceptance.

Questions for authors

See Weaknesses.

2 comments
AuthorsOpenReview ↗

Official Comment by Authors(1/1)

Many thanks to Reviewer MP6h for the careful assessment and for recognizing the gains on HumanoidBench and the usefulness of the value-estimation analysis. Below are our responses to your concerns.


1. Computational overhead (W1)

Gains are good but not a leap; not convinced they justify the extra training machinery and iterative sampling cost.

Thank you for raising this concern. GTP [1], a concurrent method that likewise trains a diffusion policy on planner-generated targets, is 3.3× slower than DPMPC at action selection under the same planning horizon and GPU. To avoid repetition, we refer to Table R1-T1 in our response to Reviewer 1p1N for the setup. Since GTP also runs this planner to generate policy targets, this overhead extends to training. In comparison, DPMPC achieves strong performance with substantially less computation.

2. MPPI aggregation (W2, W3)

Section 2.1 describes a single Gaussian search distribution and execution of its optimized mean during evaluation. How are candidates from distinct behavioral branches aggregated without producing an undesirable compromise between modes?

Thank you for highlighting these missing details, which we have added to Section 2.1. The optimized mean in Eq. 1 is not the aggregated mean over the elite actions. Instead, after the last MPPI iteration, the planner picks one elite trajectory, sampled according to its MPPI weight. This trajectory serves as the mean, while the aggregated std is retained as the final std. Since the mean is an evaluated elite, it does not compromise between two modes.

We provide an example on Pendulum-Swingup. The result is provided in Appendix F. At a state where the prior places mass on both action bounds and both attain nearly the same return, the selected MPC actions split almost evenly between the two modes, with no compromise action between them. We also trace a single planning call at this state: the randomness of prior sampling and initial Gaussian action sampling can bias the optimization direction toward either mode.

3. Relation to TD-M(PC)2^2

Thank you for pointing us to TD-M(PC)2^2 [2], which also addresses value estimation error. It attributes this error to the mismatch between the planner and the policy prior, and constrains the Gaussian prior to stay close to the planner while maximizing the value. DPMPC follows a similar formulation and extends it to diffusion priors, so our prior-training components are complementary to TD-M(PC)2^2. We have added TD-M(PC)2^2 to Related Work (Section 5).

4. Conclusion

We hope the provided analysis addresses your concerns about our work. We would greatly appreciate your reconsideration of the evaluation in light of these revisions. We would be happy to address any further concerns.

References

[1] Balim et al. Model-Based Online Decision Making via Generative Trajectory Planning. NeurIPS, 2026.

[2] Lin et al. TD-M(PC)2^2: Improving Temporal Difference MPC Through Policy Constraint. arXiv:2502.03550, 2025.

Thanks for the rebuttal and the clearification, I raised my score to 3.

Reviewer_MV1eOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DPMPC, a model-based RL framework that replaces the Gaussian policy prior in MPC with diffusion policy. The motivation is that stochastic MPC can induce multimodal action distributions that Gaussian priors cannot adequately represent, potentially introducing actor-divergence bias into value estimation. DPMPC combines generalized lazy reanalyze, which maintains multiple MPC realizations per replay state, with advantage-weighted denoising to train the diffusion prior. Experiments on 14 continuous-control tasks from DMControl, MyoSuite, and HumanoidBench demonstrate improved performance over baselines. The authors also report lower value estimation errors and moderate additional training costs compared with BMPC.

Critical strengths
  1. The paper highlights an important mismatch between the policy prior used for value learning and the MPC planner actually executed in the environment. This mismatch motivates expressive policy priors and offers an interesting perspective on model-based RL.

  2. The proposed method uses MPC-generated supervision to train diffusion policies without requiring backpropagation through the planner. Generalized lazy reanalyze and advantage-weighted denoising form a practical training framework.

Critical weaknesses
  1. The authors attribute improved value estimation to reduced actor-divergence bias, but Table 3 only measures total value estimation error. Neither the actor-divergence component in Eq. 5 nor the distributional mismatch between the prior and planner is directly measured.

  2. In Figure 5a, the Gaussian baseline follows the BMPC training protocol, while diffusion and CVAE use the DPMPC framework. Therefore, the improvement cannot be attributed solely to the policy representation. A Gaussian prior trained under the same pipeline and a Gaussian-mixture baseline would provide more convincing evidence. Moreover, the main ablations are limited to Dog-Run rather than the more challenging HumanoidBench tasks.

  3. The paper motivates diffusion priors as a way to match the planner distribution, but advantage weighting explicitly shifts probability toward higher-value actions. Figure 5c suggests that this weighting is an important contributor to performance. It remains unclear whether the gains primarily come from improved planner-distribution matching or advantage-weighted policy improvement.

  4. Despite repeated diffusion sampling during planning, value learning, and reanalysis, DPMPC reports only modest overhead over BMPC. No runtime breakdown or planning latency is provided to explain this result.

Questions for authors
  1. Could the authors separately evaluate VpV^p, VπV^\pi, and VϕV_\phi to isolate actor-divergence bias? Could the author also directly measure the distributional mismatch between the prior and planner?

  2. Could Gaussian, Gaussian-mixture, and diffusion priors be compared under identical training pipelines? Can key ablations be extended to HumanoidBench?

  3. Does advantage weighting actually reduce prior-planner divergence? How much improvement comes from distribution matching versus policy improvement?

  4. Could the authors clarify the theoretical assumptions behind the value-matching claim and justify removing Z(z)Z(z) in Appendix A?

  5. Could the authors provide a runtime breakdown for MPC planning, diffusion training, value learning, and reanalysis? In particular, how many diffusion samples and denoising evaluations are required per MPC decision? Reporting planning latency and return-versus-wall-clock curves would help establish the computational trade-off.

  6. The reported BMPC performance on HumanoidBench appears lower than its publicly released results. Could the authors clarify the differences in seeds, environment configurations, and evaluation protocols, and provide the corresponding baseline logs? Additionally, could the authors clarify the training budgets underlying the asymptotic-performance claim?

4 comments
AuthorsOpenReview ↗

Official Comment by Authors(1/3)

Many thanks to Reviewer MV1e for the careful assessment and for recognizing the importance of the mismatch between the policy prior and the MPC planner, as well as the practicality of our training framework. Below, we address your concerns in detail.


1. Measuring actor divergence directly (W1, Q1)

Could the authors separately evaluate VpV^{p}, VπV^{\pi}, and VϕV_\phi to isolate actor-divergence bias? Could the authors directly measure the distributional mismatch between the prior and planner?

Evaluating VpV^{p}. Thank you for this suggestion. Estimating VpV^{p} requires executing the prior without the planner. However, the prior is trained only on replay-buffer states visited by the planner, so the states it would reach on its own never appear in training. This distribution shift is a known limitation of policies trained with behavior-cloning objectives. Empirically, the prior fails to maintain stable locomotion on H1hand-Run and performs substantially worse on Dog-Run when executed without the planner. Monte Carlo estimates of VpV^{p} would therefore reflect this shift rather than the actor-divergence bias. We instead measure the prior–planner mismatch directly as below.

Direct measurement of the prior–planner mismatch. We use the same checkpoints and states as in our value-error analysis. At each state, we run the executed planner independently 100 times and use the mixture of the returned first-action Gaussians to estimate the planner distribution π(⋅∣z)\pi(\cdot\mid\mathbf{z}) (Eq. 4). We then draw 5,000 actions from π\pi and 5,000 from the prior pp, and estimate the total variation (TV) distance between their marginal distributions using histograms with 20 bins on [−1,1][-1,1]. We compute TV separately for each action dimension and report the average over the full action space (38 dimensions on Dog-Run, 61 on H1hand-Run).

Table R2-T1. Prior–planner total variation (TV) distance across tasks and training checkpoints. Values are mean ± std over three seeds; lower is better.

H1hand-Run

Method0.4M1.2M2M
BMPC0.290 ± 0.0090.282 ± 0.0140.288 ± 0.013
BOOM0.906 ± 0.0070.891 ± 0.0110.863 ± 0.022
DPMPC0.212 ± 0.0080.220 ± 0.0070.199 ± 0.017

Dog-Run

Method0.8M2.4M4M
BMPC0.319 ± 0.0120.284 ± 0.0060.297 ± 0.023
BOOM0.819 ± 0.0050.775 ± 0.0070.676 ± 0.082
DPMPC0.255 ± 0.0230.252 ± 0.0160.231 ± 0.008

DPMPC achieves the lowest mismatch at every checkpoint on both tasks. Two factors contribute to the remaining mismatch:

  • Finite sampling introduces variability. Two independent sets of 50 planner calls at the same state yield planner-distribution estimates with TV ≈ 0.08 between them.
  • Training and evaluation use different planner distributions. The prior learns from the lazily refreshed estimate πlazy\pi_{\text{lazy}}: a mixture of MPC realizations stored in the buffer, generated using earlier priors and value functions. Evaluation instead compares the prior with the planner at the current checkpoint.

Larger actor divergence, measured by TV, does not necessarily imply larger value error. On H1hand-Run, BOOM has higher TV than BMPC, yet lower value error at 0.4M and 2.0M (Table 3). BOOM's prior has a much larger standard deviation than the planner distribution but nearly the same mean. Empirically, DPMPC attains both the lowest divergence and the lowest value error on H1hand-Run.

AuthorsOpenReview ↗

Official Comment by Authors(2/3)

2. Distribution matching vs. advantage weighting (W3, Q3)

The paper motivates diffusion priors as a way to match the planner distribution, but advantage weighting explicitly shifts probability toward higher-value actions. Figure 5c suggests that this weighting is an important contributor to performance. It remains unclear whether the gains primarily come from improved planner-distribution matching or advantage-weighted policy improvement.

The results support complementary roles for distribution matching and advantage weighting: matching reduces prior–planner divergence, while advantage weighting provides additional gains in value accuracy and return without further reducing this divergence. Table R2-T2 separates these effects by comparing advantage weighting with uniform weights.

With uniform weights, DPMPC already achieves low prior–planner TV (0.233 ± 0.007). At η=0.5\eta=0.5, advantage weighting yields similar TV (0.231 ± 0.008), but reduces relative value MAE from 2.2% to 1.0% and increases return from 729 to 864. Thus, the additional gains from advantage weighting cannot be explained by tighter distribution matching alone. Advantage weighting favors high-value actions while preserving low prior–planner divergence; empirically, this is associated with less pessimistic value estimates and higher returns.

Table R2-T2. Effect of advantage weighting on prior–planner mismatch, value estimation error, and return on Dog-Run at 4M steps. Relative value MAE denotes the relative mean absolute error, following the same definition as in Table 3 of the paper. Signed value error averages the relative errors while retaining their signs: positive values indicate overestimation on average, and negative values indicate underestimation on average.

SettingPrior–planner TV ↓Relative value MAE (%) ↓Signed value error (%)Return ↑
η = 0.010.244 ± 0.0111.0 ± 0.7+0.9886 ± 23
η = 0.50.231 ± 0.0081.0 ± 0.6−0.5864 ± 18
η = 10³0.235 ± 0.0071.8 ± 0.6−1.8813 ± 4
Uniform weights0.233 ± 0.0072.2 ± 0.8−1.9729 ± 62

3. Controlled prior comparison (W2, Q2)

In Figure 5a, the Gaussian baseline follows the BMPC training protocol, while diffusion and CVAE use the DPMPC framework. Could Gaussian, Gaussian-mixture, and diffusion priors be compared under identical training pipelines? Can key ablations be extended to HumanoidBench?

Gaussian priors. Thank you for raising this point. We have rerun the Gaussian prior within the DPMPC pipeline, so Figure 5a now compares the diffusion, CVAE, and Gaussian priors under identical training conditions: the CVAE and Gaussian priors each replace only the diffusion model, with all other components unchanged. The diffusion prior achieves the highest return on Dog-Run, while the CVAE and Gaussian priors perform comparably. On H1hand-Run, the gap widens substantially (Appendix C, Figure 7a): the diffusion prior reaches a return of about 750 within 0.7M steps, whereas the Gaussian and CVAE priors end training at only about 560 and 420.

HumanoidBench ablations. We repeated the ablations of the policy prior, MM, and η\eta on H1hand-Run (Appendix C, Figure 7). The conclusions are consistent with those on Dog-Run.

4. Theoretical assumptions in Appendix A (Q4)

Could the authors clarify the theoretical assumptions behind the value-matching claim and justify removing Z(z)Z(z) in Appendix A?

Value-matching claim. Eq. 5 identifies two sources of value error: estimation error and actor-divergence bias. If the prior matches the planner's action distribution at all relevant states, the actor-divergence term vanishes. With sufficient training to also reduce estimation error, the learned state-value function provides an accurate estimate of the expected return obtained by executing MPC. This analysis does not assume that either condition already holds; it identifies the conditions under which value matching is achieved. Based on this analysis, we design DPMPC with an expressive diffusion prior to approximate the planner's action distribution and mitigate actor-divergence bias.

Removing Z(z)Z(\mathbf{z}). Following AWR [1], Z(z)Z(\mathbf{z}) is dropped because it depends only on the state, not on θ\theta or the action. At each state, the objective is maximized by p∗(⋅∣z)p^{\ast}(\cdot\mid\mathbf{z}), and scaling it by the positive constant 1/Z(z)1/Z(\mathbf{z}) does not change this maximizer. Hence, if the prior class can represent p∗p^{\ast}, the optimum is unchanged. Dropping Z(z)Z(\mathbf{z}) only reweights states relative to one another, which matters only when the prior cannot fit p∗p^{\ast} at every state. This is a further reason to use an expressive diffusion prior.

AuthorsOpenReview ↗

Official Comment by Authors(3/3)

5. Runtime breakdown (W4, Q5)

Could the authors provide a runtime breakdown for MPC planning, diffusion training, value learning, and reanalysis? How many diffusion samples and denoising evaluations are required per MPC decision? Reporting planning latency and return-versus-wall-clock curves would help establish the computational trade-off.

Decision time. At each MPC decision, the planner generates 24 prior trajectories of horizon H=3H=3 by rolling out the world model with actions sampled from the diffusion prior. This requires 72 action samples, each generated with T=20T=20 denoising steps. Table R2-T3 reports inference time on a single RTX 3090, averaged over 1,000 decisions. DPMPC takes approximately 6.2 ms per decision, corresponding to a control rate above 150 Hz.

Table R2-T3. MPC decision time (ms).

TaskBMPCDPMPC (ours)
Dog-Run3.666.21
H1hand-Run3.706.23

Runtime breakdown. Table R2-T4 reports the runtime breakdown of DPMPC on Dog-Run, measured on a single RTX 3090.

Table R2-T4. Runtime breakdown of DPMPC on Dog-Run.

Componentms / stepShare
MPC planning (action selection)6.5018.7%
Reanalysis12.3135.4%
World model & value learning3.6110.4%
Diffusion policy training7.7722.3%
Environment simulation2.005.8%
Replay buffer & misc.2.597.4%
Total34.78100%

6. BMPC baseline performance and training budgets (Q6)

The reported BMPC performance on HumanoidBench appears lower than its publicly released results. Could the authors clarify the differences in seeds, environment configurations, and evaluation protocols, and provide the corresponding baseline logs? Could the authors clarify the training budgets underlying the asymptotic-performance claim?

Seeds, configuration, and evaluation. We trained BMPC using its official implementation and default HumanoidBench configuration, with four seeds for both BMPC and DPMPC. All methods are evaluated every 50k environment steps over 10 episodes. Our BMPC results closely match the released results, with only small differences on Walk and Slide consistent with seed variance. These differences do not affect our conclusion: DPMPC still achieves approximately 35% higher average return across the seven tasks than the released BMPC results.

Training budgets. All tasks are trained for 2M environment steps, except Dog-Run, which is trained for 4M steps.

7. Conclusion

We have added the requested ablations on H1hand-Run and the decision-time comparison against BMPC. We hope that our responses have addressed the raised concerns. We would appreciate it if you would reconsider your evaluation accordingly. We would be happy to address any further concerns.

References

[1] Peng et al. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv:1910.00177, 2019.

Dear reviewer, as the discussion period comes to an end, please let us know if there are any further questions or if anything in our rebuttal is unclear.

Reviewer_1p1NOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DPMPC, a model-based RL method built on TD-MPC2 and BMPC that replaces Gaussian policy priors with diffusion models to better capture the planner's action distribution. The method introduces generalized lazy reanalyze and advantage-weighted score matching to train the diffusion prior from MPPI-generated actions. Experiments across 14 continuous-control tasks demonstrate improvements in sample efficiency, asymptotic performance, and value estimation accuracy.

Critical strengths

The paper addresses a relevant limitation of Gaussian policy priors in model-based RL and provides empirical evaluations across several challenging continuous-control benchmarks.

Critical weaknesses
  1. Limited methodological novelty: The method largely builds on TD-MPC2 and BMPC, primarily replacing the Gaussian prior with diffusion. Generalized lazy reanalyze and advantage-weighted score matching [1] also build on established techniques, leaving the methodological novelty unclear.

  2. Limited use of expressiveness: While diffusion is motivated by its ability to represent multimodal distributions, MPPI still optimizes a Gaussian search distribution and uses diffusion only for additional proposals. It is unclear how much the planner benefits from this expressiveness.

  3. Unclear diffusion-specific contribution: The authors motivate diffusion through its ability to capture complex planner distributions, yet only learn a single-step action prior rather than a distribution over planned trajectories. It is unclear whether matching single-step action distributions adequately captures the planner's temporal structure, or whether trajectory-level diffusion would better address the claimed actor divergence.

  4. Limited novelty in diffusion planning: Recent works such as MODIP [2] and GTP [3] already combine diffusion policies with learned world models and planning, including planner-supervised policy learning. Moreover, these methods directly model action trajectories rather than single-step actions. While their recency may make empirical comparisons difficult, the authors should at least discuss these works and clarify their methodological contributions.

[1] Ma et. al. "Efficient Online Reinforcement Learning for Diffusion Policy"

[2] Song et. al. "MODIP: Efficient Model-Based Optimization for Diffusion Policies."

[3] Balim et. al. "Model-Based Online Decision Making via Generative Trajectory Planning."

Questions for authors
  1. Trajectory-level diffusion: Why does DPMPC model only single-step action distributions rather than full action sequences similar to most diffusion policy papers in offline settings? Would trajectory-level diffusion better capture the multimodal structure of MPC solutions?

  2. Expressiveness in MPC: The authors argue that MPC requires expressive policy priors, yet the planner ultimately optimizes a Gaussian search distribution, using diffusion only for additional proposals. How does the expressiveness of the diffusion prior translate into better planning, and why is a simpler multimodal prior insufficient?

5 comments
AuthorsOpenReview ↗

Official Comment by Authors(1/2)

Many thanks to Reviewer 1p1N for providing a thorough review and valuable comments. Below are our responses to your concerns.


1. Clarify methodological novelty (W1)

The method largely builds on TD-MPC2 and BMPC, primarily replacing the Gaussian prior with diffusion. Generalized lazy reanalyze and advantage-weighted score matching also build on established techniques, leaving the methodological novelty unclear.

First, we believe the primary contribution of our work is to quantify the value estimation error associated with the mismatch between the policy prior and the planner, and to mitigate this error using an expressive diffusion prior.

Second, while our method builds upon existing techniques, we improve them for this setting on the diffusion prior training. Our generalized lazy reanalyze extends the original method from storing a single MPC realization per state to retaining multiple realizations, enabling multimodal action sampling. For advantage-weighted score matching, the key distinction lies in the training target and the role of reweighting. The maximum-entropy formulation in Ma et al. [1] extracts a diffusion policy from a Q-defined Boltzmann distribution without requiring samples from that target distribution. Our objective instead starts from an available planner distribution and uses advantages to improve it under a KL constraint that keeps the prior close to the planner. Thus, the shared weighted score-matching form serves different purposes: Q-based policy extraction and planner-constrained policy improvement. We further normalize advantages using a running estimate of their spread, allowing a single temperature to accommodate different reward scales and reducing task-specific tuning. Together, generalized lazy reanalyze and normalized advantage-weighted score matching provide practical guidance for future work on training expressive policies.

2. Expressiveness (W2, Q2)

MPPI still optimizes a Gaussian search distribution and uses diffusion only for additional proposals.

DPMPC is not tied to a specific planner. We use MPPI, the planner used by our baselines, to ensure a fair comparison and isolate the effect of the policy class. We acknowledge that other planning algorithms could better harness the multimodal diffusion prior and further improve search, which we leave to future work.

How does the expressiveness of the diffusion prior translate into better planning

The expressiveness of the diffusion prior can improve planning through two pathways. First, a prior that better matches the planner distribution can reduce the actor divergence, improving the terminal value estimates used by MPC. Second, a more expressive prior can provide more diverse action proposals, potentially improving MPC search. Both pathways operate within the same MPPI planner. Isolating and quantifying their individual contributions remains challenging. Our experiments evaluate their combined effect on task performance.

why is a simpler multimodal prior insufficient?

We visualize the action distributions of the CVAE and diffusion priors in Appendix G, providing additional results to help explain the performance difference.

For this visualization, we load the final CVAE and diffusion prior checkpoints on H1hand-Run and collect one 1,000-step episode with each. At five evenly spaced states from each episode, we sample 200 actions from the corresponding prior and plot histograms for six action dimensions.

The CVAE samples concentrate within a narrow interval, typically forming two clusters. By comparison, the diffusion samples span nearly the full action range, with distribution shapes varying across states and action dimensions. These visualizations suggest that the diffusion prior learns richer action distributions and captures more action modes than the CVAE prior.

AuthorsOpenReview ↗

Official Comment by Authors(2/2)

3. Single-step vs. trajectory-level diffusion (W3, Q1)

Why does DPMPC model only single-step action distributions rather than full action sequences, as in most diffusion policy papers in offline settings? Would trajectory-level diffusion better capture the multimodal structure of MPC solutions and the planner's temporal structure?

Thank you for raising this question. Modeling single-step actions is a design choice: in online RL, policies are typically parameterized to predict a single action given the current state. Nevertheless, our framework can be easily extended to model full action sequences. Such an extension could accelerate prior sampling during planning, since the policy would generate action sequences directly, without world-model rollouts. However, it would increase inference latency when the policy acts without the planner, since generating a full sequence is more costly than predicting a single action.

Regarding multimodality, the multimodal structure of the learned policy arises from the multiple MPC realizations stored per state and the advantage-weighted score matching objective, both of which are independent of whether the policy predicts individual actions or full action sequences. Regarding temporal structure, our design follows the closed-loop nature of MPC, which replans from the latest state at every step rather than executing a fixed action sequence. This allows subsequent actions to adapt to updated state information, although it does not explicitly model temporal dependencies within an action sequence.

We also implemented a variant that models full action sequences within our framework and observed no noticeable performance difference in our experiments.

4. Related work in diffusion planning (W4)

Recent works such as MODIP [2] and GTP [3] already combine diffusion policies with learned world models and planning, including planner-supervised policy learning, and directly model action trajectories. The authors should at least discuss these works and clarify their methodological contributions.

Thank you for pointing us to these works; we have added both to the related work section. Similar to our approach, GTP trains a diffusion policy using planner-generated targets, with value estimates used to weight these targets. However, its planner is substantially more expensive. GTP uses a planning horizon of H=4H=4 in their paper; for a fair comparison, we set it to our horizon H=3H=3 and measure the time per action selection at evaluation on Dog-Run (Table R1-T1):

Table R1-T1. Action-selection time per step at evaluation on Dog-Run. Single environment on one RTX 3090, averaged over a 500-step episode; lower is better.

Method (H=3H=3)Time per action (ms)
GTP20.48
DPMPC6.21

GTP is 3.3× slower than DPMPC. This overhead may primarily stem from its ensemble world model and beam search algorithm, which involves iterative diffusion denoising. Since GTP also runs this planner to generate policy targets, the same overhead raises its training cost. In comparison, our work achieves strong performance with significantly less computation.

MODIP focuses on offline-to-online fine-tuning: it pretrains a diffusion policy on offline datasets via behavior cloning and then improves it online, using MPC as the policy improvement operator. In contrast, DPMPC learns entirely online from scratch, without demonstrations or offline data.

5. Conclusion

We hope the additional visualization and the comparison to the most recent works address your concerns, and we would be grateful if you would reconsider your evaluation accordingly. We would be happy to address any further concerns.

Dear reviewer, as the discussion period comes to an end, please let us know if there are any further questions or if anything in our rebuttal is unclear.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Gentle Follow-up as Discussion Ends

Dear Reviewers 1p1N and MV1e,

As the discussion period is coming to an end, we would appreciate your feedback on whether our responses have addressed your concerns.

Thank you again for your valuable time and consideration.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.