- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper tries to establish a maximum entropy reinforcement learning framework for diffusion policies. Building upon the variational inference perspective of maximum-entropy RL, it establishes an upper bound on the KL divergence between the optimal diffusion path and the current policy-induced diffusion path. This leads to a tractable surrogate objective as in Equation 8. Based on which specific policy optimization objective is used, this paper proposes three variants, invoking PPO, REPPO, and WPO as the optimization objectives, respectively.
The proposed methods are first evaluated on several illustrative tasks, including a synthetic dual-mode distribution, a contact-rich multimodality stacking task, and finally PushT under different initial positions. Results demonstrated that the DA-MDP variants can retain higher task success rates while preserving solution diversity. Then, on MuJoCo Playground, which serves as the main testbed, DA-MDP methods seem to achieve better performance compared to the baseline algorithms.
Critical strengths
The motivation and instruction of the core idea is clear. The same idea is applied to different instantiations of policy optimization methods, including PPO, REPPO, and WPO. The experiments on MuJoCo Playground demonstrate improvement across all three variants, demonstrating the wide applicability of the proposed method.
Critical weaknesses
Despite the experimental improvements, I still consider the contribution of this paper quite limited. This remains my primary concern:
-
The variational formulation of the maximum entropy objective is first proposed by DIME. In fact, substituting Equation 15 in DIME into the maximum entropy RL objective exactly recovers the main theoretical framework of this paper.
-
The second effort of this paper seems to be using different policy optimization objectives to instantiate Equation 8. The proposed method uses local objectives, which bypass the BPTT issue in DIME; however, the local objective has already been extensively explored in both online and offline diffusion policy RL, for example, DPPO [1], BDPO [2]. The objectives used are all from existing published literature, which again limits the contribution of this paper to exercising the maximum entropy principles for different base algorithms.
I would also recommend that the authors include a detailed review of existing diffusion policy RL algorithms to better situate the contribution of this paper.
Questions for authors
1, As reported in previous methods that optimize diffusion policies across bi-level MDPs, using large diffusion steps (e.g., >= 20) may cause stability issues in temporal difference value learning due to delayed updates. Since the proposed method also uses temporal difference learning as described in Section D.2.1, could the author discuss the stability for larger diffusion steps?
- In the main locomotion experiment (Figure 3 (a)), the compared baseline algorithms include several non-diffusion methods and diffusion-based methods like FPO and DPPO. Given that more advanced methods have emerged in recent years, could the authors extend the comparison to recent algorithms, including but now limited to ReinFlow [3] and FPO++ [4]?
[1] Diffusion Policy Policy Optimization
[2] Behavior-Regularized Diffusion Policy Optimization for Offline Reinforcement Learning
[3] ReinFlow: Fine-tuning Flow Matching Policy with Online Reinforcement Learning.
[4] Flow Policy Gradients for Robot Control.
4 comments
Review Response 2/2
Coverage of related work
We have extended the related-work section (Section 4) to include BDPO and FPO++, suggested by reviewer uDBw, and QSM, SDAC/DPMD, DACER, and QVPO, suggested by reviewer JSkj. Table 2 in Appendix E.2 compares how these methods optimize diffusion policies, incorporate entropy, and differentiate through the denoising process, and identifies their training settings. We also clarify how DA-MDP differs from the already cited DIME, DPPO, ReinFlow, and TruDi in its objective and diffusion updates.
References
[1] Gao, Chen-Xiao, et al. [“Behavior-Regularized Diffusion Policy Optimization for Offline Reinforcement Learning.”] arXiv, 2025, arXiv:2502.04778v2.
Q1: Stability with longer diffusion chains
Does temporal-difference value learning remain stable with large diffusion depths (e.g., 20 or more steps), given delayed updates in the augmented MDP? Discuss the implications for the method in Section D.2.1.
A3: To address this question, we conducted a new experiment evaluating DA:REPPO with 8, 16, and 24 diffusion steps on four DMC tasks, using multiple training seeds and 50M environment interactions per run. Across all tested tasks, seeds, and diffusion depths, the learning curves show no training instabilities. Longer chains improve performance on AcrobotSwingupSparse, while performance on the other tasks remains broadly comparable. Figure 3(c) shows the learning curves averaged across these four tasks, and Figure 13 presents the learning curves for each task and diffusion-depth setting.
Q2: Recent diffusion and flow baselines
Extend the main locomotion comparison in Figure 3(a) to stronger recent methods, including ReinFlow and FPO++ (Flow Policy Gradients for Robot Control).
A4: Following your suggestion, we added ReinFlow and FPO++ to the twelve-task DMC comparison. We also evaluated on-policy adaptations of DIME and SDAC following reviewer JSkj’s suggestion. All four baselines use K = 8 and approximately 50M environment interactions per seed, matching the interaction budget of TruDi and our DA-MDP methods. For DIME and SDAC, we replace replay with fresh rollouts, following TruDi’s comparison setting for DIME. Figure 4(a) reports performance aggregated across the twelve tasks, and Figure 6 shows the learning curves for each task.
Final normalized interquartile mean (IQM) returns across the twelve tasks are reported below with 95% stratified-bootstrap confidence intervals:
| Method | IQM return ↑ | 95% confidence interval |
|---|---|---|
| DA:REPPO | 0.819 | [0.793, 0.836] |
| TruDi | 0.814 | [0.788, 0.835] |
| DA:WPO | 0.783 | [0.742, 0.812] |
| DIME (on-policy) | 0.727 | [0.709, 0.743] |
| SDAC (on-policy) | 0.549 | [0.506, 0.589] |
| DA:PPO | 0.548 | [0.514, 0.590] |
| ReinFlow | 0.477 | [0.438, 0.517] |
| FPO++ | 0.437 | [0.394, 0.483] |
| FPO | 0.360 | [0.329, 0.401] |
All three DA-MDP variants achieve higher aggregate IQM than ReinFlow and FPO++. DA:REPPO and DA:WPO also outperform the on-policy DIME and SDAC adaptations, while DA:PPO performs comparably to SDAC. TruDi’s stronger aggregate performance than the DIME adaptation is consistent with its published comparison.
We thank the reviewer again for their constructive feedback. Their comments have helped us clarify the attribution to prior work, expand the related-work discussion, and explain the contributions of the augmented Bellman construction, the ME-WPO and DA:WPO objectives, and the return-improvement bound. We have also strengthened the empirical evaluation with recent baselines and learning curves for each task and diffusion-depth setting.
All line references refer to the updated manuscript. All changes are highlighted in green text.
Follow-up on our response to your review
Dear Reviewer uDBw,
Thank you again for your feedback. We clarified the distinctions from DIME, DPPO, and BDPO and expanded the related-work discussion. We also added comparisons with ReinFlow, FPO++, and on-policy adaptations of DIME and SDAC, alongside experiments with up to 24 diffusion steps to investigate learning stability with longer chains.
As the discussion period nears its end, we would welcome your assessment of these additions and any remaining questions.
Thanks for providing further explanations and details. I have updated my evaluation accordingly.
Review Response 1/2
We thank the reviewer for their careful assessment and constructive suggestions. We appreciate their positive assessment of the motivation and formulation, the applicability to PPO, REPPO, and WPO, and the empirical improvements on MuJoCo Playground. The feedback has helped us clarify how DA-MDP builds on prior work and identify the specific contributions of our formulation. Below, we explain these distinctions and address the questions about longer diffusion chains and recent baselines.
W1: Novelty relative to DIME
The reviewer argues that substituting DIME Eq. (15) into the maximum-entropy RL objective exactly recovers the main theoretical framework, limiting the novelty of the variational formulation.
A1: We now explicitly attribute the shared diffusion-path maximum-entropy objective to DIME immediately after Eq. (8). Our contribution is the augmented Bellman construction and the resulting local actor objective, including the forward-kernel contribution when the forward process is learned, rather than local differentiation alone. We have extended the related-work section (Section 4) and added Table 2 in Appendix E.2 to distinguish these contributions from prior methods by their actor objectives, entropy treatment, differentiation, and task setting.
W2: Novelty of local updates and optimization objectives
Local diffusion-policy updates already appear in DPPO and BDPO, and the reviewer views the optimization objectives as existing methods. Clarify the additional contributions beyond combining these ideas and expand the related-work discussion.
A2: We agree that local updates for diffusion policies have been studied previously, including in DPPO and BDPO. Our contribution is not the use of local denoising decisions alone, but the derivation of their rewards, Bellman equations, and policy objectives for maximum-entropy RL, including learnable forward kernels.
For DPPO, we already describe the connection in Section 3.3 (p. 5, line 269): setting the temperature to zero in DA:PPO recovers a DPPO-style local PPO objective on the denoising-augmented MDP. At positive temperature, DA-MDP additionally includes the reverse–forward log-density correction derived in Eq. (8). This correction enters both the soft Bellman equations and the local actor objective, Eqs. (9)–(10), and is absent from DPPO’s reward-only objective.
We thank the reviewer for bringing BDPO to our attention. BDPO addresses offline RL by fitting a behavior diffusion model to a dataset and regularizing the learned policy toward that model. While BDPO also notes that a uniform action reference recovers maximum-entropy RL at the marginal-objective level, it does not specify the reference diffusion process needed to realize this setting. Constructing transition kernels whose final action marginal is uniform is an additional, nontrivial step. Its experiments evaluate policies regularized toward a learned behavior model; they do not evaluate the proposed uniform-reference extension to maximum-entropy RL. See BDPO [1], Sections 3–5.
Even with such a reference, BDPO’s local penalty would compare two denoising kernels, both directed from noise toward actions. Our correction instead compares the reverse policy with a forward noising kernel, directed from actions toward noise. Thus, choosing a uniform action marginal in BDPO would still leave a different local update construction; it would not by itself recover the DA-MDP algorithm. We explain this distinction in the extended related-work section (Section 4) and Appendix E.2.
Our contributions extend beyond instantiating existing policy optimization algorithms. We derive augmented soft Bellman equations and a local maximum-entropy actor objective, including the reverse–forward density correction and contributions from learnable forward kernels (Eqs. (9)–(10)). We also derive the maximum-entropy extension of WPO, ME-WPO (Appendix F.4), and its diffusion-augmented counterpart, DA:WPO (Eq. (12)). Finally, Proposition 3.1 extends TRPO’s return-improvement analysis to DA-MDPs, giving sufficient conditions for improved augmented return under joint changes to the reverse policy and forward kernels.