- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes DPMPC, which uses a diffusion policy prior to guide MPPI planning and reduce the mismatch between the prior and the executed planner. The prior is trained through advantage-weighted denoising using periodically refreshed planner distributions stored in replay. Benchmarking experiments show improved performance, sample efficiency, and lower value-estimation errors.
Critical strengths
-
The method is straightforward and well motivated, and the paper is clearly written.
-
The empirical results are good across the three benchmarks, with particularly notable gains on HumanoidBench. I like the value-estimation analysis, it provides useful evidence beyond the main performance comparisons. But I think you might need to check TD-MPC-square since it also provides a much simpler and effective solution to reduce value estimation error.
-
Using a diffusion policy as an MPPI prior is a sensible design choice. Policy-guided planning centered on a single nominal behavior can struggle to discover alternative behavioral branches efficiently through Gaussian perturbations alone (like Residual-MPPI). A multimodal diffusion prior offers a way to propose distinct high-value action candidates directly, potentially broadening the planner's search beyond local variations of one behavior. I find this motivation compelling independently of the value-estimation argument. (However, I think this is not discussed much and might lead to a methodolody-wise limitation below)
Critical weaknesses
Limitations
-
The conceptual advance feels incremental. Replacing a Gaussian policy prior with a diffusion model is a natural extension of expressive policy learning. The empirical gains are good but not a leap, I am not yet convinced that they sufficiently justify the additional training machinery and iterative sampling cost. The paper would be more compelling if it established a planning capability enabled by this design beyond improved benchmark performance.
-
As in the strength, my main concern is how the multimodal prior interacts with MPPI's aggregation step. Section 2.1 describes a single Gaussian search distribution and execution of its optimized mean during evaluation. I could not find an explanation of how candidates from distinct behavioral branches are aggregated without producing an undesirable compromise between modes. For example, in a classic PushT scenario with two incompatible but similarly valuable action modes, their weighted mean can be a poor action even when both modes are useful (Please correct me if I have missed a relevant mechanism; if one exists, it should be made explicit in the main text.)
This also limits what I take away from the experimental analysis. A focused analysis or controlled experiment with incompatible modes, tracking candidates through aggregation and final action selection, would be more informative to me than further parameter sweeps. If cross-mode averaging can produce poor actions and no mechanism addresses it, this would be a substantive methodological limitation. Conversely, a convincing resolution and evidence of effective planning across distinct branches would materially strengthen the contribution and could move my assessment toward acceptance.
Questions for authors
See Weaknesses.
2 comments
Thanks for the rebuttal and the clearification, I raised my score to 3.
Official Comment by Authors(1/1)
Many thanks to Reviewer MP6h for the careful assessment and for recognizing the gains on HumanoidBench and the usefulness of the value-estimation analysis. Below are our responses to your concerns.
1. Computational overhead (W1)
Thank you for raising this concern. GTP [1], a concurrent method that likewise trains a diffusion policy on planner-generated targets, is 3.3× slower than DPMPC at action selection under the same planning horizon and GPU. To avoid repetition, we refer to Table R1-T1 in our response to Reviewer 1p1N for the setup. Since GTP also runs this planner to generate policy targets, this overhead extends to training. In comparison, DPMPC achieves strong performance with substantially less computation.
2. MPPI aggregation (W2, W3)
Thank you for highlighting these missing details, which we have added to Section 2.1. The optimized mean in Eq. 1 is not the aggregated mean over the elite actions. Instead, after the last MPPI iteration, the planner picks one elite trajectory, sampled according to its MPPI weight. This trajectory serves as the mean, while the aggregated std is retained as the final std. Since the mean is an evaluated elite, it does not compromise between two modes.
We provide an example on Pendulum-Swingup. The result is provided in Appendix F. At a state where the prior places mass on both action bounds and both attain nearly the same return, the selected MPC actions split almost evenly between the two modes, with no compromise action between them. We also trace a single planning call at this state: the randomness of prior sampling and initial Gaussian action sampling can bias the optimization direction toward either mode.
3. Relation to TD-M(PC)2
Thank you for pointing us to TD-M(PC)2 [2], which also addresses value estimation error. It attributes this error to the mismatch between the planner and the policy prior, and constrains the Gaussian prior to stay close to the planner while maximizing the value. DPMPC follows a similar formulation and extends it to diffusion priors, so our prior-training components are complementary to TD-M(PC)2. We have added TD-M(PC)2 to Related Work (Section 5).
4. Conclusion
We hope the provided analysis addresses your concerns about our work. We would greatly appreciate your reconsideration of the evaluation in light of these revisions. We would be happy to address any further concerns.
References
[1] Balim et al. Model-Based Online Decision Making via Generative Trajectory Planning. NeurIPS, 2026.
[2] Lin et al. TD-M(PC)2: Improving Temporal Difference MPC Through Policy Constraint. arXiv:2502.03550, 2025.