acceptodds
Under review as a conference paper at ICLR 2027

Diffusion-Augmented Markov Decision Processes for Maximum Entropy Reinforcement Learning

Abstract

Diffusion models provide an expressive framework for sampling from complex, unnormalized distributions. In this work, we extend Maximum Entropy Reinforcement Learning (ME-RL) to diffusion-based policies by introducing Diffusion-Augmented Markov Decision Processes (DA-MDPs). DA-MDPs interpret each reverse-diffusion transition as an individual reinforcement-learning decision, while only the final denoised action is executed in the environment. Our DA-MDPs follow from a principled derivation based on the variational-inference formulation of ME-RL. By augmenting policy and target trajectories with intermediate diffusion variables, we obtain a tractable reverse-KL upper bound via the data-processing inequality. This bound decomposes across denoising transitions, yielding diffusion-augmented variants of soft rewards, value functions, and local policy objectives. This provides a general framework for adapting ME-RL algorithms to diffusion policies while differentiating through only one diffusion transition at a time. We instantiate the framework with PPO, REPPO, and a maximum-entropy extension of WPO. Experiments demonstrate improved continuous-control performance, benefits from additional diffusion steps, and memory-efficient training. On the StackCube and PushT manipulation tasks, DA-MDP methods learn alternative successful strategies from the same initial state and achieve higher success rates and generally higher success-weighted mode entropy than the Gaussian ME-RL baseline. We also demonstrate successful training when using action chunking.

Primary area:
reinforcement learning
Keywords:
maximum entropy reinforcement learning, diffusion policies, policy optimization, Markov decision processes, multimodal policies, continuous control

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_uDBwOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper tries to establish a maximum entropy reinforcement learning framework for diffusion policies. Building upon the variational inference perspective of maximum-entropy RL, it establishes an upper bound on the KL divergence between the optimal diffusion path and the current policy-induced diffusion path. This leads to a tractable surrogate objective as in Equation 8. Based on which specific policy optimization objective is used, this paper proposes three variants, invoking PPO, REPPO, and WPO as the optimization objectives, respectively.

The proposed methods are first evaluated on several illustrative tasks, including a synthetic dual-mode distribution, a contact-rich multimodality stacking task, and finally PushT under different initial positions. Results demonstrated that the DA-MDP variants can retain higher task success rates while preserving solution diversity. Then, on MuJoCo Playground, which serves as the main testbed, DA-MDP methods seem to achieve better performance compared to the baseline algorithms.

Critical strengths

The motivation and instruction of the core idea is clear. The same idea is applied to different instantiations of policy optimization methods, including PPO, REPPO, and WPO. The experiments on MuJoCo Playground demonstrate improvement across all three variants, demonstrating the wide applicability of the proposed method.

Critical weaknesses

Despite the experimental improvements, I still consider the contribution of this paper quite limited. This remains my primary concern:

  1. The variational formulation of the maximum entropy objective is first proposed by DIME. In fact, substituting Equation 15 in DIME into the maximum entropy RL objective exactly recovers the main theoretical framework of this paper.

  2. The second effort of this paper seems to be using different policy optimization objectives to instantiate Equation 8. The proposed method uses local objectives, which bypass the BPTT issue in DIME; however, the local objective has already been extensively explored in both online and offline diffusion policy RL, for example, DPPO [1], BDPO [2]. The objectives used are all from existing published literature, which again limits the contribution of this paper to exercising the maximum entropy principles for different base algorithms.

I would also recommend that the authors include a detailed review of existing diffusion policy RL algorithms to better situate the contribution of this paper.

Questions for authors

1, As reported in previous methods that optimize diffusion policies across bi-level MDPs, using large diffusion steps (e.g., >= 20) may cause stability issues in temporal difference value learning due to delayed updates. Since the proposed method also uses temporal difference learning as described in Section D.2.1, could the author discuss the stability for larger diffusion steps?

  1. In the main locomotion experiment (Figure 3 (a)), the compared baseline algorithms include several non-diffusion methods and diffusion-based methods like FPO and DPPO. Given that more advanced methods have emerged in recent years, could the authors extend the comparison to recent algorithms, including but now limited to ReinFlow [3] and FPO++ [4]?

[1] Diffusion Policy Policy Optimization

[2] Behavior-Regularized Diffusion Policy Optimization for Offline Reinforcement Learning

[3] ReinFlow: Fine-tuning Flow Matching Policy with Online Reinforcement Learning.

[4] Flow Policy Gradients for Robot Control.

4 comments
AuthorsOpenReview ↗

Review Response 1/2

We thank the reviewer for their careful assessment and constructive suggestions. We appreciate their positive assessment of the motivation and formulation, the applicability to PPO, REPPO, and WPO, and the empirical improvements on MuJoCo Playground. The feedback has helped us clarify how DA-MDP builds on prior work and identify the specific contributions of our formulation. Below, we explain these distinctions and address the questions about longer diffusion chains and recent baselines.

W1: Novelty relative to DIME

The reviewer argues that substituting DIME Eq. (15) into the maximum-entropy RL objective exactly recovers the main theoretical framework, limiting the novelty of the variational formulation.

A1: We now explicitly attribute the shared diffusion-path maximum-entropy objective to DIME immediately after Eq. (8). Our contribution is the augmented Bellman construction and the resulting local actor objective, including the forward-kernel contribution when the forward process is learned, rather than local differentiation alone. We have extended the related-work section (Section 4) and added Table 2 in Appendix E.2 to distinguish these contributions from prior methods by their actor objectives, entropy treatment, differentiation, and task setting.

W2: Novelty of local updates and optimization objectives

Local diffusion-policy updates already appear in DPPO and BDPO, and the reviewer views the optimization objectives as existing methods. Clarify the additional contributions beyond combining these ideas and expand the related-work discussion.

A2: We agree that local updates for diffusion policies have been studied previously, including in DPPO and BDPO. Our contribution is not the use of local denoising decisions alone, but the derivation of their rewards, Bellman equations, and policy objectives for maximum-entropy RL, including learnable forward kernels.

For DPPO, we already describe the connection in Section 3.3 (p. 5, line 269): setting the temperature to zero in DA:PPO recovers a DPPO-style local PPO objective on the denoising-augmented MDP. At positive temperature, DA-MDP additionally includes the reverse–forward log-density correction derived in Eq. (8). This correction enters both the soft Bellman equations and the local actor objective, Eqs. (9)–(10), and is absent from DPPO’s reward-only objective.

We thank the reviewer for bringing BDPO to our attention. BDPO addresses offline RL by fitting a behavior diffusion model to a dataset and regularizing the learned policy toward that model. While BDPO also notes that a uniform action reference recovers maximum-entropy RL at the marginal-objective level, it does not specify the reference diffusion process needed to realize this setting. Constructing transition kernels whose final action marginal is uniform is an additional, nontrivial step. Its experiments evaluate policies regularized toward a learned behavior model; they do not evaluate the proposed uniform-reference extension to maximum-entropy RL. See BDPO [1], Sections 3–5.

Even with such a reference, BDPO’s local penalty would compare two denoising kernels, both directed from noise toward actions. Our correction instead compares the reverse policy with a forward noising kernel, directed from actions toward noise. Thus, choosing a uniform action marginal in BDPO would still leave a different local update construction; it would not by itself recover the DA-MDP algorithm. We explain this distinction in the extended related-work section (Section 4) and Appendix E.2.

Our contributions extend beyond instantiating existing policy optimization algorithms. We derive augmented soft Bellman equations and a local maximum-entropy actor objective, including the reverse–forward density correction and contributions from learnable forward kernels (Eqs. (9)–(10)). We also derive the maximum-entropy extension of WPO, ME-WPO (Appendix F.4), and its diffusion-augmented counterpart, DA:WPO (Eq. (12)). Finally, Proposition 3.1 extends TRPO’s return-improvement analysis to DA-MDPs, giving sufficient conditions for improved augmented return under joint changes to the reverse policy and forward kernels.

AuthorsOpenReview ↗

Review Response 2/2

Coverage of related work

We have extended the related-work section (Section 4) to include BDPO and FPO++, suggested by reviewer uDBw, and QSM, SDAC/DPMD, DACER, and QVPO, suggested by reviewer JSkj. Table 2 in Appendix E.2 compares how these methods optimize diffusion policies, incorporate entropy, and differentiate through the denoising process, and identifies their training settings. We also clarify how DA-MDP differs from the already cited DIME, DPPO, ReinFlow, and TruDi in its objective and diffusion updates.

References

[1] Gao, Chen-Xiao, et al. [“Behavior-Regularized Diffusion Policy Optimization for Offline Reinforcement Learning.”] arXiv, 2025, arXiv:2502.04778v2.

Q1: Stability with longer diffusion chains

Does temporal-difference value learning remain stable with large diffusion depths (e.g., 20 or more steps), given delayed updates in the augmented MDP? Discuss the implications for the method in Section D.2.1.

A3: To address this question, we conducted a new experiment evaluating DA:REPPO with 8, 16, and 24 diffusion steps on four DMC tasks, using multiple training seeds and 50M environment interactions per run. Across all tested tasks, seeds, and diffusion depths, the learning curves show no training instabilities. Longer chains improve performance on AcrobotSwingupSparse, while performance on the other tasks remains broadly comparable. Figure 3(c) shows the learning curves averaged across these four tasks, and Figure 13 presents the learning curves for each task and diffusion-depth setting.

Q2: Recent diffusion and flow baselines

Extend the main locomotion comparison in Figure 3(a) to stronger recent methods, including ReinFlow and FPO++ (Flow Policy Gradients for Robot Control).

A4: Following your suggestion, we added ReinFlow and FPO++ to the twelve-task DMC comparison. We also evaluated on-policy adaptations of DIME and SDAC following reviewer JSkj’s suggestion. All four baselines use K = 8 and approximately 50M environment interactions per seed, matching the interaction budget of TruDi and our DA-MDP methods. For DIME and SDAC, we replace replay with fresh rollouts, following TruDi’s comparison setting for DIME. Figure 4(a) reports performance aggregated across the twelve tasks, and Figure 6 shows the learning curves for each task.

Final normalized interquartile mean (IQM) returns across the twelve tasks are reported below with 95% stratified-bootstrap confidence intervals:

MethodIQM return ↑95% confidence interval
DA:REPPO0.819[0.793, 0.836]
TruDi0.814[0.788, 0.835]
DA:WPO0.783[0.742, 0.812]
DIME (on-policy)0.727[0.709, 0.743]
SDAC (on-policy)0.549[0.506, 0.589]
DA:PPO0.548[0.514, 0.590]
ReinFlow0.477[0.438, 0.517]
FPO++0.437[0.394, 0.483]
FPO0.360[0.329, 0.401]

All three DA-MDP variants achieve higher aggregate IQM than ReinFlow and FPO++. DA:REPPO and DA:WPO also outperform the on-policy DIME and SDAC adaptations, while DA:PPO performs comparably to SDAC. TruDi’s stronger aggregate performance than the DIME adaptation is consistent with its published comparison.

We thank the reviewer again for their constructive feedback. Their comments have helped us clarify the attribution to prior work, expand the related-work discussion, and explain the contributions of the augmented Bellman construction, the ME-WPO and DA:WPO objectives, and the return-improvement bound. We have also strengthened the empirical evaluation with recent baselines and learning curves for each task and diffusion-depth setting.

All line references refer to the updated manuscript. All changes are highlighted in green text.

AuthorsOpenReview ↗

Follow-up on our response to your review

Dear Reviewer uDBw,

Thank you again for your feedback. We clarified the distinctions from DIME, DPPO, and BDPO and expanded the related-work discussion. We also added comparisons with ReinFlow, FPO++, and on-policy adaptations of DIME and SDAC, alongside experiments with up to 24 diffusion steps to investigate learning stability with longer chains.

As the discussion period nears its end, we would welcome your assessment of these additions and any remaining questions.

Thanks for providing further explanations and details. I have updated my evaluation accordingly.

Reviewer_JSkjOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Similar to DPPO formulation, this paper makes Max-entropy RL work with diffusion policies by treating every denoising step as its own decision in an augmented MDP. The paper provides rigorous derivations on those; how the algorithm is derived from a tractable upper bound on the reverse KL. By such augmentation it is possible to only backprop through one diffusion step at a time (as TD do opposed to MC), so memory stays O(1) unlike TruDi. The same framework is instantiated into a number of algorithms, PPO, REPPO, and WPO. In the experiments, it is shown that DA:REPPO roughly matches TruDi while being faster under tight memory budgets.

Critical strengths
  • Principled and General framework, clean derivations of ME-RL version of DPPO (in RL as inference framework)
  • interesting points on forward kernel learning
Critical weaknesses
  • paper's contribution is based on combination of multiple algorithms (DPPO, DDPO, DIME, TruDi, REPPO, WPO, etc). Lacks comparisons to prior works on score-matching methods (QSM, SDAC/DPMD) and other diffusion ME-RL methods (DACER, QVPO).
  • most experiments except comparison to TruDi on figure 3 do not include comparisons to competitive diffusion baselines. showing the algorithm is learning multi-modal policy and thus giving better result than gaussian policy looks like a sanity-check at this point (with so many diffusion policy algorithms out there). The point is whether the proposed method can do better, learning better multimodal policy or getting higher return, compared to previous algorithms
  • In that sense, TruDi already performs on par (although with larger memory requirements than 2-3G, which is not a big deal nowadays with 80G vram being default), and humanoid task is also excluded. TruDi paper seems to say that humanoid task is where it is doing the best, which leaves the most informative comparison missing
Questions for authors
  • How does this compare to strong diffusion/flow baselines like DIME, SDAC, DACER, or more recent ones? I get that those are SAC-like instantiations so regime might be slightly different, but since off-policy algorithms can naturally applied to massively parallel settings I think the experiments regarding them and reason why the proposed method can excel compared to those needs to be addressed.
  • When does the memory advantage matter? Can you show a setting where TruDi runs out of memory or becomes impractical while the proposed methods work well?
  • Is there an ablation against a fixed forward kernels? some of the main content is devoted to this, so it makes me wonder how much difference it makes.
  • Can you add TruDi for all the experiments not only figure 3?
5 comments
AuthorsOpenReview ↗

Review Response 1/3

We thank the reviewer for their careful reading and constructive feedback. We appreciate their recognition of the principled and general framework, the clean derivation of maximum-entropy local policy updates, and the interesting points on learnable forward kernels. Their comments highlight the importance of distinguishing our contributions from prior work and establishing when local training offers a practical advantage. Below, we address these questions, the requested baseline comparisons, and the forward-kernel ablation.

W1a: Contribution as a combination of existing algorithms

The reviewer sees the paper’s contribution as a combination of existing algorithms (DPPO, DDPO, DIME, TruDi, REPPO, WPO, etc.).

A1: DA-MDP provides a general framework for local maximum-entropy optimization of diffusion policies with learnable forward kernels. Its central contribution is the construction of an augmented control problem whose rewards, soft Bellman equations, and actor objective preserve the reverse–forward density correction required by the maximum-entropy formulation. This construction enables optimization through individual diffusion transitions while retaining the contribution from learning the forward process (Eqs. (9)–(10)).

We further establish a TRPO-style return-improvement bound in Proposition 3.1. The bound accounts for changes in both the reverse policy and the learned forward kernels and, under its stated assumptions, gives sufficient conditions for improvement in the augmented return. The framework therefore provides both local optimization objectives and theoretical conditions connecting their optimization to return improvement.

These contributions clarify how DA-MDP extends prior approaches. DA:PPO extends a DPPO-style local objective to maximum-entropy RL with learnable forward kernels. DIME supplies a closely related diffusion-path maximum-entropy objective, but does not derive local actor objectives for individual denoising transitions. Its reparameterized actor update requires backpropagation through the complete denoising chain. DA-MDP instead derives augmented soft Bellman equations and local actor objectives. Applying this construction to REPPO yields DA:REPPO, whose intermediate diffusion critic enables local actor updates in place of the full-chain differentiation used by TruDi. For WPO, we derive its maximum-entropy extension, ME-WPO, and its diffusion-augmented counterpart, DA:WPO (Eq. (12); Appendices F.4–F.5). Building on PPO, REPPO, and a maximum-entropy extension of WPO, we derive their diffusion-policy counterparts by applying their policy-update rules to the augmented MDP, yielding local maximum-entropy updates at each denoising step.

We thank the reviewer for highlighting DDPO. We now explicitly cite and discuss DDPO in Section 4 and include it in Table 2 in Appendix E.2. DDPO shares the treatment of denoising as a sequence of decisions. At the task level, it can be viewed as a contextual-bandit counterpart to DPPO: a context conditions the generation of a sample, which receives a terminal reward, rather than an action influencing subsequent environment states. Its objective maximizes expected reward without the entropy term required by maximum-entropy RL. Thus, this shared denoising perspective does not provide the reverse–forward correction or augmented soft Bellman equations derived in our framework. See DDPO [1], Section 4.

References

[1] Black et al. Training Diffusion Models with Reinforcement Learning. 2023.

AuthorsOpenReview ↗

Review Response 2/3

W1b / W2 / Q1: Related work and competitive diffusion-policy baselines

Comparisons to QSM, SDAC/DPMD, DACER, and QVPO are missing. Extend the evaluation beyond Gaussian policies to strong diffusion and flow baselines, including DIME, SDAC, and DACER.

A2 / A3: We thank the reviewer for these suggestions. We now explicitly cite and discuss QSM, SDAC/DPMD, DACER, and QVPO in Section 4. Table 2 in Appendix E.2 compares their objectives, entropy treatment, differentiation, and training settings. Our original continuous-control evaluation already included the diffusion-policy methods TruDi and DPPO and the flow-policy method FPO. The Multimodal Agent benchmark also includes TruDi and DPPO (Appendix E.3.8, Figure 16).

Following your suggestion, we have additionally evaluated on-policy adaptations of DIME and SDAC. We also added ReinFlow and FPO++, as suggested by reviewer uDBw. These additional comparisons use the same twelve DMC tasks, K = 8, and approximately 50M environment interactions per seed. Figure 4(a) reports aggregate IQM, and Figure 6 shows the task-level learning curves.

Final normalized interquartile mean (IQM) returns across the twelve tasks are reported below with 95% stratified-bootstrap confidence intervals:

MethodIQM return ↑95% confidence interval
DA:REPPO0.819[0.793, 0.836]
TruDi0.814[0.788, 0.835]
DA:WPO0.783[0.742, 0.812]
DIME (on-policy)0.727[0.709, 0.743]
SDAC (on-policy)0.549[0.506, 0.589]
DA:PPO0.548[0.514, 0.590]
ReinFlow0.477[0.438, 0.517]
FPO++0.437[0.394, 0.483]
FPO0.360[0.329, 0.401]

All three DA-MDP variants achieve higher aggregate IQM than ReinFlow and FPO++. DA:REPPO and DA:WPO also outperform the on-policy DIME and SDAC adaptations, while DA:PPO performs comparably to SDAC. DA:REPPO remains competitive with TruDi.

DIME and SDAC originally use replay-based off-policy training. Following TruDi’s comparison setting for DIME, we evaluate explicitly labelled fresh-rollout adaptations in our massively parallel on-policy benchmark. Faithful replay-based comparisons require separate validation, since scaling off-policy learning to this regime requires careful algorithmic and implementation adjustments [1,2].

Our response to Q4 reports the completed StackCube comparison.

References

[1] Li et al. Parallel Q-Learning: Scaling Off-Policy Reinforcement Learning under Massively Parallel Simulation. ICML, 2023.

[2] Sabatini et al. Bridging the Gap: Enabling Soft Actor Critic for High Performance Legged Locomotion. 2026.

W3a / Q2: Practical memory and training-speed benefits

The reviewer questions whether the memory savings remain practically relevant on large-memory GPUs and asks for a setting where TruDi exceeds the available memory or becomes impractical while DA-MDP remains effective.

A4a: Our original memory benchmark uses small policy networks with approximately 0.55M parameters, which limits the absolute memory difference. This difference grows with larger networks. Increased policy capacity is also relevant for robust sim-to-real transfer and learning across many tasks, as demonstrated in [3], [4], and [5]. For example, RT-1 uses 35M parameters [4], while π₀ contains a 300M-parameter action expert within a 3.3B-parameter model [5]. These deployed robot policies are substantially larger than the approximately 0.55M-parameter networks used in our original memory benchmark.

We extended the study across four network sizes and 8, 16, and 24 diffusion steps on an A100 40GB GPU (Figure 14). With 16.9M policy parameters, DA:REPPO uses approximately 9.0 GiB at both 16 and 24 steps. At 16 steps, TruDi requires 33.5 GiB and DA:REPPO is 2.31× faster. At 24 steps, TruDi requires gradient accumulation to fit; preserving its effective batch, DA:REPPO remains 2.50× faster.

We also tested whether this computational advantage translates into faster learning on CheetahRun, using 16.9M-parameter policies and 16 diffusion steps. DA:REPPO reaches the first evaluated mean return above 700 2.02× sooner, above 900 1.95× sooner, and above 910 1.28× sooner in measured training time:

First evaluated mean return aboveDA:REPPOTruDi
7002.56 h5.18 h
9006.07 h11.85 h
9109.27 h11.85 h

Figures 3(a) and 15 show the learning curves against measured training time.

References

[3] OpenAI et al. Solving Rubik’s Cube with a Robot Hand. 2019.

[4] Brohan et al. RT-1: Robotics Transformer for Real-World Control at Scale. 2022.

[5] Black et al. π₀: A Vision-Language-Action Flow Model for General Robot Control. 2024.

AuthorsOpenReview ↗

Review Response 3/3

W3b: Humanoid locomotion

A4b: Following the reviewer’s suggestion, we added the Humanoid G1 benchmark (G1JoystickFlatTerrain) to Figure 3(d). After 50M environment interactions, DA:REPPO achieves 37.06 ± 0.48, compared with 36.12 ± 0.09 for TruDi across three training seeds and approximately 33.2 for the published REPPO reference. DA:REPPO achieves the highest mean final return in this comparison, extending the evaluation to the humanoid setting highlighted by the reviewer.

Q3: Fixed versus learned forward kernels

Provide an ablation against fixed forward kernels to quantify how much learning the forward process contributes to performance.

A5: Following your suggestion, we compared learned diffusion coefficients with a fixed cosine schedule. Learning improves mean return on three of the four evaluated tasks, with no improvement on AcrobotSwingupSparse. We evaluated DA:REPPO on AcrobotSwingup, FishSwim, HopperHop, and AcrobotSwingupSparse, using four paired training seeds and 50M environment interactions per run.

Mean final returns with fixed and learned diffusion coefficients are reported below as mean ± standard error across four training seeds:

TaskFixed cosine scheduleLearned coefficients
AcrobotSwingup372.4 ± 13.3555.8 ± 6.6
FishSwim575.9 ± 10.5666.7 ± 2.9
HopperHop310.8 ± 81.4472.7 ± 84.6
AcrobotSwingupSparse319.2 ± 17.3297.4 ± 10.8

Learning the coefficients improves mean return on three of the four tasks, although HopperHop exhibits substantial variation across seeds. AcrobotSwingupSparse shows no improvement. These results demonstrate a practical benefit in several settings and motivate allowing the diffusion schedule to adapt during training. Figure 3(b) reports the results with uncertainty estimates, and Figure 12 shows the learning curves for each task.

Q4: TruDi across the experimental suite

Add TruDi comparisons to the experiments beyond Figure 3.

A6: Following the reviewer’s suggestion, we extend the comparison with TruDi to StackCube and PushT. Within each task, all methods use the same training interaction budget and are evaluated from the same initial configurations. The table reports mean ± SD across three training seeds.

Task (interactions)MethodSuccess (%) ↑Success-weighted mode entropy (bits) ↑
StackCube (100M)DA:REPPO (ODE)96.4 ± 0.50.096 ± 0.008
TruDi (ODE)92.3 ± 1.50.089 ± 0.005
Gaussian REPPO86.5 ± 0.80.056 ± 0.005
PushT (250M)DA:REPPO (ODE)98.3 ± 0.40.0571 ± 0.0030
TruDi (ODE)95.1 ± 0.60.0504 ± 0.0113
Gaussian REPPO84.5 ± 1.60.0486 ± 0.0064

Among these methods, DA:REPPO achieves the highest mean success and success-weighted mode entropy on both tasks. Figure 2(c) additionally shows that the illustrated TruDi policy has only 0.4% success at 175° and a highest observed success-weighted mode entropy of 0.052 bits, versus 0.467 bits for DA:REPPO at 190°. Figure 10 shows how training success develops for each seed.

We also moved the TruDi results on the Multimodal Agent benchmark from the appendix into Figure 1.

We thank the reviewer again for their constructive suggestions. Their feedback has helped us clarify the contribution relative to prior work and strengthen the empirical assessment of local training and forward-kernel learning.

All line references refer to the updated manuscript. All changes are highlighted in green text.

Reviewer_FDvfOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces Diffusion-Augmented Markov Decision Processes (DA-MDPs) for maximum-entropy reinforcement learning with diffusion policies. By augmenting trajectories with intermediate diffusion variables, it derives a tractable path-space objective that decomposes into local denoising decisions. This enables policy updates that differentiate through one diffusion transition at a time. The framework is instantiated with PPO, REPPO, and a maximum-entropy extension of WPO. Experiments examine continuous-control performance, successful behavioral diversity, and memory–runtime trade-offs.

Critical strengths
  1. The local training formulation is technically substantive. The paper goes beyond treating denoising as an augmented decision process by deriving the reverse–forward log-density correction required by the maximum-entropy objective (pp. 4–5). It also accounts for learned forward kernels. The exact-enumeration experiment in Table 4 (p. 43) provides a useful check: the local and trajectory gradients agree to numerical precision, whereas omitting the forward-kernel gradient produces a substantial error.

  2. The experiments provide meaningful evidence of practical utility. The continuous-control results compare the DA variants with their corresponding base algorithms and relevant diffusion-policy baselines (Figure 3, pp. 8–9). The controlled comparison with TruDi uses matched network configurations and documents both the benefits and limitations of local training under different memory budgets (pp. 43–44). The manipulation evaluation also measures successful diversity within the same initial configuration, rather than pooling different states or training seeds (p. 35).

Critical weaknesses
  1. Unequal training budgets limit the interpretation of the StackCube comparison. The reported checkpoints use approximately 50M interactions for Gaussian REPPO, 100M for unchunked DA:REPPO, 250M for chunked DA:REPPO and DA:WPO, and 300M for DA:PPO (p. 36). Although these differences are disclosed, they make it difficult to separate the effects of the method from additional training in the success and diversity results in Figure 2b (p. 8). The comparison between chunked and unchunked DA:REPPO is similarly affected.

  2. The runtime study establishes a memory–throughput trade-off, but not yet time-to-performance efficiency. The measurements report the time required to process a fixed block of environment transitions (pp. 43–44). Because local-transition and full-chain training use different optimization procedures, faster processing does not necessarily imply reaching a target return sooner. A return-versus-wall-clock comparison under a matched memory budget would more directly establish the practical training benefit.

Questions for authors
  1. Could you report StackCube success and success-weighted mode entropy at a common interaction budget, using existing checkpoints where possible? In particular, how much of the improvement over Gaussian REPPO, and between chunked and unchunked DA:REPPO, remains at matched budgets (Figure 2b, p. 8; p. 36)?

  2. For the controlled TruDi comparison, do the lower-memory throughput advantages translate into less wall-clock time to reach a comparable return? A focused comparison on the existing CheetahRun setting would help clarify this point (pp. 43–44).

3 comments
AuthorsOpenReview ↗

Answer to review

We thank the reviewer for their positive assessment and constructive suggestions. We appreciate their recognition of the “technically substantive” local training formulation and the experiments’ “meaningful evidence of practical utility,” including the controlled memory–runtime comparison and the evaluation of successful behavioral diversity. Below, we address matched training budgets and training time to comparable performance.

W1 / Q1: StackCube comparisons at matched interaction budgets

The original StackCube comparison uses unequal training budgets. Report success and success-weighted mode entropy at a common interaction budget to assess how much of the gains over Gaussian REPPO and between chunked and unchunked DA:REPPO remains.

A1: Our aim was to evaluate each method near convergence using final checkpoints at prescribed training budgets. Training success approaches a plateau for most runs, as shown in Figure 10 in Appendix E.3.6. We have clarified this aim and added matched-budget comparisons to separate the effects of the method from additional training.

We have added a matched-budget comparison between unchunked DA:RePPO and Gaussian RePPO in Figure 2(b). After 100.14M environment interactions, stochastic DA:RePPO achieves 95.0% success and 0.149 bits of success-weighted mode entropy, compared with 86.5% and 0.056 bits for Gaussian RePPO. The advantage therefore persists at the same interaction budget.

We also added a matched-budget comparison of all six methods at 100M interactions (Appendix E.3.4, Table 5). Chunked DA:REPPO has not yet reached its final performance at this point: stochastic success increases from 56.2 ± 7.1% at 98.44M to 88.2 ± 3.0% after 250.09M interactions. Its success-weighted mode entropy is 0.2124 ± 0.0225 and 0.211 ± 0.009 bits, respectively. Thus, its diversity advantage is already present at the common budget, while task success improves substantially with further training. Figure 10 in Appendix E.3.6 shows the training learning curves for each method, allowing the reviewer to assess convergence and the effect of additional training beyond the common interaction budget.

Following another reviewer’s suggestion, we also added three-seed comparisons with TruDi on StackCube and G1. At 100.14M interactions on StackCube, stochastic DA:RePPO achieves 95.0% success and 0.149 bits of success-weighted mode entropy, versus 92.9% and 0.109 bits for TruDi (Figure 2(b)). On G1, after 50M interactions, DA:RePPO achieves a return of 37.06 ± 0.48, versus 36.12 ± 0.09 for TruDi (Figure 3(d)). DA:RePPO thus achieves higher mean performance in both comparisons.

On PushT, with three seeds per method and the same interaction budget, DA:REPPO achieves 98.3 ± 0.4% success and success-weighted mode entropy (pH) of 0.0571 ± 0.0030 bits, versus 95.1 ± 0.6% and 0.0504 ± 0.0113 for TruDi. In Figure 2(c), TruDi reaches only 0.4% success at 175°; its highest observed pH is 0.052 bits, versus 0.467 bits for DA:REPPO at 190°. Thus, DA:REPPO combines high task success near its direction transition with greater successful directional diversity.

W2 / Q2: Training time to comparable return against TruDi

Processing a fixed block of transitions faster does not establish faster learning. Do the lower-memory throughput advantages translate into less wall-clock time to reach a comparable return? Provide a focused, controlled comparison with TruDi on CheetahRun under a matched memory budget.

A2: Following your suggestion, we completed a CheetahRun learning comparison using matched policy networks on an A100 GPU. We increased the actor size to a 6-layer, width-2048 policy with approximately 16.9M parameters to assess the practical benefit when training a larger policy on an A100. Both methods use 16 diffusion steps and width-512 critics, while retaining their established training hyperparameters. In this configuration, DA:REPPO requires approximately 9.0 GiB of GPU memory, compared with 33.5 GiB for TruDi. The return-versus-training-time comparison therefore evaluates whether this memory advantage also translates into faster learning.

DA:REPPO reaches comparable returns sooner in measured training time:

First evaluated mean return aboveDA:REPPOTruDi
7002.56 h5.18 h
9006.07 h11.85 h
9109.27 h11.85 h

DA:REPPO therefore reaches a mean return above 900 approximately 1.95× sooner. This demonstrates that the throughput advantage translates into less training time to reach comparable performance in this setting.

Figures 3(a) and 15 show the CheetahRun learning curves against measured training time.

We thank the reviewer again for these useful suggestions. Their feedback has improved the experimental comparisons and helped us state more precisely what the success, diversity, and runtime results establish.

All line references refer to the updated manuscript. All changes are highlighted in green text.

AuthorsOpenReview ↗

Follow-up on our response to your review

Dear Reviewer FDvf,

Thank you again for your suggestions. We added StackCube evaluations at a common interaction budget, including Gaussian REPPO and both chunked and unchunked DA:REPPO. We also added the requested CheetahRun comparison against measured training time to assess whether the computational advantage translates into faster learning.

We would appreciate your feedback on these comparisons and whether any concerns remain that we could address before the discussion closes.

Thank you to the authors for the detailed rebuttal. I appreciate the additional clarifications, which have addressed several of my concerns. Based on the responses, I keep my score 3 (Weak Accept).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revisions and additional results

We thank the reviewers for their constructive feedback and follow-up assessments. Below, we summarize the main revisions and the conclusions supported by the additional results.

1. Contribution and relation to prior work. We expanded the related-work section and appendix to distinguish our contribution more clearly from previous approaches, explicitly attributing the shared diffusion-path maximum-entropy objective to DIME. Our contribution is the derivation of diffusion-augmented soft Bellman equations and local maximum-entropy actor objectives that retain the reverse–forward density correction and account for learning the forward kernels. This enables local optimization of individual denoising transitions while jointly learning the forward and reverse processes. We also establish a TRPO-style return-improvement bound that accounts for changes to both processes. Furthermore, we derive ME:WPO, a maximum-entropy extension of WPO based on Wasserstein gradient flows, and its diffusion-augmented counterpart, DA:WPO.

See response to JSkj A1, and response to uDBw A1–A2.

2. Broader baseline comparisons. We added ReinFlow, FPO++, DIME, and SDAC to Mujoco DMC evaluation. All three DA-MDP variants achieve higher aggregate IQM than ReinFlow and FPO++. DA:REPPO and DA:WPO also outperform DIME and SDAC, while DA:PPO performs comparably to SDAC. DA:REPPO remains competitive with TruDi.

We also extended the TruDi comparison to StackCube, PushT, and the G1JoystickFlatTerrain humanoid task. At matched training interaction budgets, DA:REPPO achieves higher mean success and success-weighted mode entropy on both manipulation tasks and higher mean return on G1. Together, these results demonstrate strong performance of DA-MDPs across continuous-control and manipulation benchmarks.

See response to JSkj A2/A3 and A4b/A6, and response to uDBw A4.

3. Memory savings and flexibility translate into faster learning. We extended the controlled comparison to larger policies and measured learning progress against training time. With 16.9M policy parameters and 16 diffusion steps on an A100, DA:REPPO uses approximately 9.0 GiB versus TruDi’s 33.5 GiB. On CheetahRun, DA:REPPO first exceeds a mean return of 900 after 6.07 hours, versus 11.85 hours for TruDi. Thus, the memory and throughput benefits translate into approximately 1.95× faster learning to this return threshold.

See response to JSkj A4a, and response to FDvf A2.

4. Learning forward kernels improves performance. A four-task ablation with four paired seeds shows that learning the diffusion coefficients improves mean return on three of the four tasks. These results demonstrate practical performance gains from learning the forward process.

See response to JSkj A5.

5. Stable learning with longer diffusion chains. Across four DMC tasks and multiple training seeds, DA:REPPO trains stably with 8, 16, and 24 diffusion steps. Longer chains improve performance on AcrobotSwingupSparse, while the other tasks remain broadly comparable. These results demonstrate stable local value learning over the tested diffusion depths.

See response to uDBw A3.

All revisions are incorporated into the updated manuscript and highlighted in green. We thank all reviewers for their careful assessments and constructive suggestions, which have substantially improved the paper’s clarity and strengthened its theoretical and experimental presentation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.