acceptodds
Under review as a conference paper at ICLR 2027

Diffusion Advantage Matching: Unified Diffusion Policy Optimization

Abstract

Diffusion policies can represent multimodal behaviors, but improving them online is challenging because their action likelihoods are intractable. Existing diffusion-RL methods address this difficulty through action gradients, likelihood approximations, or backpropagation through the denoising chain (BPTT), using different formulations with distinct limitations. Motivated by KL-regularized RL, we unify these seemingly different methods under Generalized Optimal Policy Matching (GOPM), a policy-agnostic framework that casts soft policy improvement as matching policy log-ratios to soft advantages under a general divergence. For diffusion policies, we instantiate GOPM as Diffusion Advantage Matching (DAM), a theoretically grounded method with a detached estimator that avoids BPTT while preserving the soft-optimal distribution. Under the Log-Variance divergence, DAM remains valid for samplers covering both policy and reference, enabling DAM to support online, offline, and hybrid learning under a single objective; to the best of our knowledge, it is the first diffusion-RL method to do so. Across MuJoCo and DeepMind Control, DAM achieves strong online and competitive offline performance, better preserves multimodal distributions, and delivers strong results in both single-stage and two-stage offline-to-online learning on D4RL.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, diffusion policies, diffusion models, KL-regularized reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_FH7KOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Generalized Optimal Policy Matching (GOPM), which formulates policy optimization as divergence minimization toward a soft-optimal policy, and develops Diffusion Advantage Matching (DAM) as its diffusion-policy instantiation. DAM uses a trajectory-level log-ratio objective with detached sampling and a leave-one-out advantage estimator to avoid backpropagation through the denoising chain. The method is further extended to offline and hybrid learning through a log-variance formulation and different sampling and critic-estimation strategies. Experiments evaluate online continuous control, offline D4RL tasks, hybrid learning, and a case study on distributional coverage on a Gaussian-mixture.

Critical strengths
  1. A well-motivated and useful approach to diffusion-policy optimization. The detached trajectory-level objective provides a practical way to update diffusion policies without backpropagating through the entire denoising process. Its integration into online actor–critic learning is appealing and has the potential to improve computational efficiency and training stability.

  2. Promising online performance on challenging tasks. The reported results show competitive performance against existing diffusion-policy methods, with encouraging improvements on challenging, high-dimensional control tasks. These findings suggest that the proposed approach has practical value for online reinforcement learning.

  3. Evaluation beyond average control return. The paper examines distributional coverage alongside policy performance and evaluates the method across online, offline, and hybrid learning settings. The coverage analysis and policy-parameterization ablation provide useful perspectives on the method’s behavior and the benefits of expressive diffusion policies.

Critical weaknesses
  1. The methodological contribution relative to VarGrad and RTB is insufficiently established. The central ingredients of the detached estimator—log-variance optimization, stopped sampling gradients, leave-one-out control variates, and the connection to reverse-KL gradients—are closely related to [1]. More directly, [2] introduce Relative Trajectory Balance (RTB), including trajectory-level log-ratio optimization without differentiating through sampling. The online extension may be useful, but the main text does not clearly isolate the additional algorithmic contribution. RTB is also absent from the reported offline and distribution-matching comparisons.

  2. The transition from trajectory-level optimization to action-level optimality requires additional justification. Section 4 uses the DPI to relate trajectory and action KL divergences, then asserts recovery of the original soft-optimal action policy. However, the trajectory KL contains an additional conditional-trajectory KL term. An upper bound alone therefore does not establish equivalence of the optima; appropriate tightness and realizability conditions are needed. It is unclear from the main text whether these conditions hold for the finite-step diffusion parameterization with a fixed initial noise distribution.

  3. The numerical experiments do not sufficiently isolate the source of the gains. Section 5.1.1 attributes the improvement over DIME to the estimator, citing the same architecture and reverse-KL objective. A controlled comparison within the same implementation, changing only the gradient estimator, would provide stronger evidence. This is especially relevant given the overlap with existing VarGrad/LOO estimators and the trajectory-based formulation of [3]. Matched environment interactions also do not establish computational efficiency such as runtime, peak memory, and the cost of multiple trajectories per state should be reported.

References

[1] Lorenz Richter et al. VarGrad: A Low-Variance Gradient Estimator for Variational Inference. NeurIPS, 2020.

[2] Siddarth Venkatraman et al. Amortizing Intractable Inference in Diffusion Models for Vision, Language, and Control. NeurIPS, 2024.

[3] Onur Celik, et al. DIME: Diffusion-Based Maximum Entropy Reinforcement Learning. ICML, 2025.

Questions for authors
  1. Could the authors provide a more detailed qualitative comparison with RTB’s offline VarGrad formulation? Which differences beyond the online critic updates are necessary for the reported improvements, and can their effects be isolated experimentally?

  2. Could the authors provide argument require the conditional-trajectory KL gap to vanish, and is this attainable under the implemented diffusion parameterization? Please also clarify whether the Section 4.1 critic estimates the action-regularized or trajectory-regularized value function.

  3. What exactly is invariant to the sampling distribution? Is the intended claim limited to an attainable zero-loss solution under sufficient support? How does the support-based explanation of the offline results account for the policy component in the mixture sampler and for errors in the IQL critic outside the dataset?

I would be willing to raise my score if the authors sufficiently address these concerns through clear qualitative justification and supporting empirical evidence.

5 comments

Part 1/3 — RTB and VarGrad comparison

We thank the reviewer for the detailed review and for the openness to raising the score. We particularly appreciate the recognition that DAM's “integration into online actor–critic learning is appealing and has the potential to improve computational efficiency and training stability”, with “encouraging improvements on challenging, high-dimensional control tasks”. DAM differs from VarGrad and RTB well beyond this online integration, in the problem it solves and its general design, as we make explicit below.

Problem 1: Contribution relative to VarGrad and RTB

[Weakness 1]: The methodological contribution relative to VarGrad and RTB is insufficiently established. [...] The online extension may be useful, but the main text does not clearly isolate the additional algorithmic contribution. RTB is also absent from the reported offline and distribution-matching comparisons.

[Question 1]: Could the authors provide a more detailed qualitative comparison with RTB's offline VarGrad formulation? Which differences beyond the online critic updates are necessary for the reported improvements, and can their effects be isolated experimentally?

[Answer]: We would like to clarify that DAM is not RTB with online critic updates: the two methods solve different problems. RTB [2] is limited to fine-tuning a pretrained diffusion model toward a fixed target via a predefined reward (RTB, Eq. 13). Its MDP is therefore not a general environment MDP but the denoising process of the diffusion model, and RTB is in this sense a variant of stochastic optimal control whose base process is a diffusion process, not an online RL method. VarGrad [1] is a variational-inference estimator toward a fixed, known density.

In contrast, DAM is designed for general RL: it learns a policy from scratch in an environment MDP, with a jointly learned critic, toward a target that moves with the policy (Eq. 3; Theorems 1–2). Our method applies to any policy class, with diffusion policies as one instance alongside Gaussian policies (App. E.1), whereas RTB is limited to diffusion models. As a result, a single DAM objective covers online learning (Table 1), offline learning (Table 2a), single-stage hybrid learning (Fig. 3; Sec. 5.1.3) and two-stage offline-to-online learning (App. E.2, Table 4), whereas RTB supports neither online nor hybrid learning. The distribution-matching comparison (Sec. 5.1.4) includes only methods that learn from scratch.

Offline DAM vs. RTB offline. Offline learning is a capability that DAM supports rather than its main contribution, and even here the two methods are designed differently. RTB fine-tunes on its own samples and makes little use of the data: its dataset trajectories carry no critic signal (App. B.1, Eq. 39), and adding them changes its scores by at most 1.2 points (RTB, Table G.1). DAM, in contrast, is built to learn from the data. Its objective is a divergence that remains valid under covering sampler (App. F), so its offline sampler is constructed around the dataset (Eq. 40–42) where the critic's ranking enters every update.

Empirically, trained on RTB-style dataset trajectories alone, the policy collapses on Hopper-Medium (17.8, vs. 89.2 for DAM). DAM also needs critic values only, whereas RTB additionally relies on the critic's action gradient ∇aQ\nabla_{a}Q (RTB, Eq. 11). With RTB added to Table 2a (RTB from its Table 4), DAM is ahead on 8 of the 9 datasets:

DatasetRTBDAM
HalfCheetah-ME74.987.9
HalfCheetah-M53.755.5
HalfCheetah-MR48.148.4
Hopper-ME96.7102.4
Hopper-M82.889.2
Hopper-MR100.4100.2
Walker2d-ME109.5110.0
Walker2d-M87.388.7
Walker2d-MR93.697.9

[1] VarGrad: A Low-Variance Gradient Estimator for Variational Inference, Richter et al., NeurIPS 2020.

[2] Amortizing Intractable Inference in Diffusion Models for Vision, Language, and Control, Venkatraman et al., NeurIPS 2024.

Part 2/3 — Trajectory KL and critic interpretation

Problem 2: From trajectory-level optimization to action-level optimality

[Weakness 2]: [...] Section 4 uses the DPI to relate trajectory and action KL divergences, then asserts recovery of the original soft-optimal action policy. However, the trajectory KL contains an additional conditional-trajectory KL term. An upper bound alone therefore does not establish equivalence of the optima; appropriate tightness and realizability conditions are needed. It is unclear from the main text whether these conditions hold for the finite-step diffusion parameterization with a fixed initial noise distribution.

[Question 2]: Could the authors provide argument require the conditional-trajectory KL gap to vanish, and is this attainable under the implemented diffusion parameterization? Please also clarify whether the Section 4.1 critic estimates the action-regularized or trajectory-regularized value function.

[Answer]: Optimizing a trajectory-level variational bound instead of the clean-action objective is standard: diffusion models are themselves trained by minimizing such a bound, which is not tight in general, and they still recover the data distribution. Trajectory-level bounds are likewise standard in diffusion RL, e.g. DIME [3].

The appendix already states when this bound is tight (App. A.6). Proposition 2 (Eq. 34) separates the trajectory KL into two KL terms: the clean-action KL, DKL(πθ(a0)∥π∗(a0))D_{\mathrm{KL}}(\pi_\theta(a_0)\Vert\pi^{\ast}(a_0)), and the conditional-trajectory gap, Ea0[DKL(πθ(a1:K∣a0)∥πref(a1:K∣a0))]\mathbb{E}_{a_0}[D_{\mathrm{KL}}(\pi_\theta(a_{1:K}\mid a_0)\Vert\pi_{\mathrm{ref}}(a_{1:K}\mid a_0))]. Together with Eq. 34, Corollary 1 then states that the bound is tight exactly when the gap vanishes, and that driving the trajectory KL to zero enforces the clean-action optimum. For readability, we move this result into the main text.

Both KL terms in Eq. 34 can be driven to zero simultaneously when the reference's noising path forgets the clean action by its final step, as a diffusion forward process does (Sec. 2.1). Any clean-action distribution can then be paired with this path while keeping the fixed Gaussian start of Eq. 16, so the two terms decouple and can be minimized independently. Both terms then reach zero, the gap vanishes, and Corollary 1 gives πθ(a0)∝πref(a0)exp⁡(A/β)\pi_{\theta}(a_{0})\propto\pi_{\text{ref}}(a_{0})\exp(A/\beta).

This is attainable under the implemented parameterization. Our sampler follows DIME's (App. G.2): the policy starts from a fixed Gaussian prior, and DIME designs its noising process to end in that same Gaussian (DIME [3], Sec. 3.2). We follow this design, so the reference ends where the policy starts. This holds for any variance of the Gaussian, so the choice of variance in Eq. 16 imposes no restriction. Moreover, Theorem 2 (App. A.4) applies with the whole chain as the action, so every DAM update is a soft policy improvement step for the trajectory-regularized objective, which lower-bounds the action-level one, as is standard for diffusion models.

The critic. The Sec. 4.1 critic estimates the trajectory-regularized value: its target (Eq. 20) penalizes the trajectory log-ratio, consistent with Theorems 1–2 applied with the whole chain as the action.

[3] DIME: Diffusion-Based Maximum Entropy Reinforcement Learning, Celik et al., ICML 2025.

Part 3/3 — Estimator, computational cost, and sampling

Problem 3: Isolating the source of the gains, and computational cost

[Weakness 3]: The numerical experiments do not sufficiently isolate the source of the gains. Section 5.1.1 attributes the improvement over DIME to the estimator, citing the same architecture and reverse-KL objective. A controlled comparison within the same implementation, changing only the gradient estimator, would provide stronger evidence. This is especially relevant given the overlap with existing VarGrad/LOO estimators and the trajectory-based formulation of [3]. Matched environment interactions also do not establish computational efficiency such as runtime, peak memory, and the cost of multiple trajectories per state should be reported.

[Answer]: The comparison with DIME is controlled (App. E.3, l.1555–1560): both share the implementation, architecture and training protocol (App. G.2, G.6.1; Table 7) and the same trajectory-level RL objective (Eq. 3; [3]), and DAM's actor gradients are two orders of magnitude smaller (Table 5) while its returns are higher (Fig. 2). To isolate the estimator, as suggested, we keep the DAM loss fixed and change only the gradient estimator: the detached estimator reaches 11849 ±\pm 684 on Humanoid, whereas backpropagating through the sampler fails on all five seeds (387 ±\pm 52).

Computational cost is already reported in App. E.3: DAM is faster than five of the seven baselines (Table 6b), and even with two trajectories per state it outperforms DIME with less wall-clock time (Table 6a). We add peak memory for each number of trajectories per state to App. E.3.

[3] DIME: Diffusion-Based Maximum Entropy Reinforcement Learning, Celik et al., ICML 2025.

Problem 4: What is invariant to the sampler, and the support explanation

[Question 3]: What exactly is invariant to the sampling distribution? Is the intended claim limited to an attainable zero-loss solution under sufficient support? How does the support-based explanation of the offline results account for the policy component in the mixture sampler and for errors in the IQL critic outside the dataset?

[Answer]: Yes, the claim is limited to the attainable zero-loss solution. “Independent of the sampling distribution” (Intro, l.065–067; Conclusion, l.484–485) is a slip in wording on our part. What is invariant is the zero-loss solution, which every sampling distribution covering the policy and the reference shares, provided it is attainable (Problem 2). What we meant is flexibility: the sampler can draw on the current policy, the offline data, or both. We revise both sentences accordingly.

We would like to clarify that offline learning is an additional capability supported by DAM, rather than its primary contribution, which is a stable method for online KL-regularized RL that supports both Gaussian and diffusion policies trained from scratch, with guaranteed soft policy improvement (Theorems 1–2).

The policy component adds samples, not support. The target is the reference tilted by the critic, so it has the reference's support, which offline is a behaviour clone of the data (App. B.1, l.1153–1156). The policy component can move beyond the recorded actions, but only within this support. Its role is to supply the policy's own trajectories, so that the critic's ranking enters every update (Problem 1). The coverage that matters is therefore the dataset's, which is broadest for Medium-Replay by the D4RL design (App. G.1, l.1746–1749).

Errors in the IQL critic. Relying on an approximate IQL-type critic is standard in critic-guided offline methods, e.g. IDQL [4] and QGPO [5], and works well in practice, as it does for DAM (Table 2a); the paper treats it as a surrogate (Sec. 4.2, l.322–323). DAM's guarantee also does not need an exact critic: an offset Qϕ=Qβ∗+c(s)Q_{\phi}=Q^{*}_{\beta}+c(s) shifts every residual of a state equally, leaving the loss and its minimizer unchanged (App. B.1, Eq. 36–37). As for errors outside the dataset, because the target stays within the behaviour-cloned reference's support, they matter only where that reference places mass, and a larger β\beta reduces their effect.

[4] IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies, Hansen-Estruch et al., 2023.

[5] Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline RL, Lu et al., ICML 2023.

Reviewer_vdXzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3(+1)
Soundness
2
Read review
Paper summary

The paper presents Generalized Optimal Policy Matching and its diffusion-policy instantiation, Diffusion Advantage Matching (DAM). DAM regresses denoising-trajectory log-policy ratios toward estimated soft advantages, using detached samples and a leave-one-out baseline to avoid backpropagation through the sampling chain. A log-variance formulation extends the actor objective to offline and hybrid data, with different critic-learning procedures. Experiments cover online continuous control, offline D4RL locomotion, hybrid learning, and a Gaussian-mixture distribution-matching task. The reported online results are strong, while offline results vary across dataset types.

Critical strengths
  1. The detached actor update is a useful practical contribution. The connection to log-variance estimation and leave-one-out control variates provides a principled way to optimize diffusion-policy probabilities without differentiating through the sampling chain. Applying this mechanism within an actor-critic framework is technically meaningful.

  2. The online results support practical utility. Table 1 reports improvements over several diffusion-RL baselines across six displayed tasks, with five-seed statistics. The gains over DIME on Dog Run and Dog Trot are particularly encouraging. The distribution-matching experiment additionally evaluates mode coverage rather than reporting return alone.

Critical weaknesses
  1. The trajectory-level surrogate does not automatically preserve the clean-action optimum. Section 4 correctly invokes an upper bound from trajectory KL to marginal action KL, but then claims convergence to the original soft-optimal action policy. The KL chain rule includes an additional conditional-trajectory divergence, which can change the optimum unless suitable realizability and tightness conditions hold. The fixed Gaussian starting distribution in Eq. 16 makes this substantive. As an illustrative one-step bandit, let independent z and epsilon be standard Gaussians, use reference action a=z+epsilon and policy a=z+c+epsilon, reward r(a)=a, and beta=1. The marginal-KL objective is c-c^2/4, whereas the trajectory-KL objective is c-c^2/2; their optimal shifts are 2 and 1, respectively. This example does not reproduce the reported experiments, but shows why the claimed equivalence requires additional assumptions. Clarifying those assumptions is central to assessing the distribution-preservation claim.

  2. The theoretical interpretation of the offline/hybrid extension is stronger than currently justified. A log-variance objective can share the same zero-loss solution across sufficiently covering sampling distributions, but its values, gradients, and finite-sample optimization still depend on that distribution. Thus the claim that the objective becomes sampling-distribution independent needs qualification. Moreover, Section 4.2 replaces the soft critic with an IQL-trained surrogate; IQL does not generally estimate the optimal soft Q-function for the stated KL-regularized objective. The paper acknowledges this approximation, but should separate a shared actor-loss form from a shared optimality guarantee. The explanation of Medium-Replay versus Medium-Expert performance in terms of support conditions also remains a hypothesis without a controlled coverage analysis.

Questions for authors
  1. What conditions make the trajectory-KL bound tight at the desired clean-action optimum, and how are they satisfied with the fixed initial noise distribution in Eq. 16?

  2. Does sampling-distribution independence refer only to a realizable zero-loss optimum? What theoretical claim remains when using the IQL critic?

  3. Does Appendix E.3 compare DAM and DIME under matched sampling and training budgets, including runtime, memory, and sensitivity to the number of trajectories used by the leave-one-out estimator? These results would affect my assessment of the estimator's practical advantage.

5 comments

Part 1/3 — Trajectory KL and clean-action optimum

We thank the reviewer for the careful review and for recognizing the detached actor update as “a useful practical contribution”, which provides “a principled way to optimize diffusion-policy probabilities without differentiating through the sampling chain”, and the gains over DIME on Dog Run and Dog Trot as “particularly encouraging”. We group the comments into three problems below.

Problem 1: Trajectory-level KL and the clean-action optimum

[Weakness 1]: The trajectory-level surrogate does not automatically preserve the clean-action optimum. Section 4 correctly invokes an upper bound from trajectory KL to marginal action KL, but then claims convergence to the original soft-optimal action policy. The KL chain rule includes an additional conditional-trajectory divergence, which can change the optimum unless suitable realizability and tightness conditions hold. The fixed Gaussian starting distribution in Eq. 16 makes this substantive. [...]

[Question 1]: What conditions make the trajectory-KL bound tight at the desired clean-action optimum, and how are they satisfied with the fixed initial noise distribution in Eq. 16?

[Answer]: We thank the reviewer for this observation. Optimizing a trajectory-level variational bound instead of the clean-action objective is standard: diffusion models are themselves trained by minimizing such a bound, which is not tight in general, and they still recover the data distribution. Trajectory-level bounds are likewise standard in diffusion RL, e.g. DIME [1].

The appendix already states when this bound is tight (App. A.6). Proposition 2 (Eq. 34) separates the trajectory KL into two KL terms: the clean-action KL, DKL(πθ(a0)∥π∗(a0))D_{\mathrm{KL}}(\pi_\theta(a_0)\Vert\pi^{\ast}(a_0)), and the expected conditional KL between the policy's and the reference's latent paths given a0a_{0}, Ea0[DKL(πθ(a1:K∣a0)∥πref(a1:K∣a0))]\mathbb{E}_{a_0}[D_{\mathrm{KL}}(\pi_\theta(a_{1:K}\mid a_0)\Vert\pi_{\mathrm{ref}}(a_{1:K}\mid a_0))]. Together with Eq. 34, Corollary 1 then states that the bound is tight exactly when the second term vanishes, and that driving the trajectory KL to zero enforces the clean-action optimum. For readability, we move this result into the main text.

Both KL terms in Eq. 34 can be driven to zero simultaneously when the reference's noising path forgets the clean action by its final step, as a diffusion forward process does (Sec. 2.1). Any clean-action distribution can then be paired with this path while keeping the fixed Gaussian start of Eq. 16, so the two terms decouple and can be minimized independently. Both terms then reach zero, and Corollary 1 gives πθ(a0)∝πref(a0)exp⁡(A/β)\pi_{\theta}(a_{0})\propto\pi_{\text{ref}}(a_{0})\exp(A/\beta). The reviewer's one-step example is exactly where this fails: its start zz still remembers the clean action, coupling the two terms so they cannot both reach zero and the optimum shifts. A forward process that fully noises the action removes this memory.

This is how the condition is satisfied with the fixed Gaussian of Eq. 16. Our sampler follows DIME's (App. G.2): the policy starts from a fixed Gaussian prior, and DIME designs its noising process to end in that same Gaussian (DIME [1], Sec. 3.2). We follow this design, so the reference ends where the policy starts. This holds for any variance of the Gaussian, so the choice of variance in Eq. 16 imposes no restriction. Moreover, Theorem 2 (App. A.4) applies with the whole chain as the action, so every DAM update is a soft policy improvement step for the trajectory-regularized objective, which lower-bounds the action-level one, as is standard for diffusion models.

[1] DIME: Diffusion-Based Maximum Entropy Reinforcement Learning, Celik et al., ICML 2025.

Part 2/3 — Offline objective and IQL critic

Problem 2: What the offline objective guarantees

[Weakness 2]: A log-variance objective can share the same zero-loss solution across sufficiently covering sampling distributions, but its values, gradients, and finite-sample optimization still depend on that distribution. Thus the claim that the objective becomes sampling-distribution independent needs qualification. Moreover, Section 4.2 replaces the soft critic with an IQL-trained surrogate; IQL does not generally estimate the optimal soft Q-function for the stated KL-regularized objective. The paper acknowledges this approximation, but should separate a shared actor-loss form from a shared optimality guarantee. The explanation of Medium-Replay versus Medium-Expert performance in terms of support conditions also remains a hypothesis without a controlled coverage analysis.

[Question 2]: Does sampling-distribution independence refer only to a realizable zero-loss optimum? What theoretical claim remains when using the IQL critic?

We would like to clarify that offline learning is an additional capability supported by DAM, rather than its primary contribution, which is a stable method for online KL-regularized RL that supports both Gaussian and diffusion policies trained from scratch, with guaranteed soft policy improvement (Theorems 1–2).

[Answer]: Yes, it refers only to the realizable zero-loss optimum. “Independent of the sampling distribution” (Intro, l.065–067) is a slip in wording on our part, and we thank the reviewer for catching it. What we meant is flexibility. Every sampling distribution that covers the policy and the reference shares the same zero-loss solution, so the sampler can draw on the current policy, the offline data, or both. We will revise this sentence accordingly.

The theoretical claim that remains with the IQL critic. The actor-loss form is shared, and the optimality guarantee holds relative to the critic. Relying on an approximate critic is standard in RL: actor-critic methods routinely optimize against biased value estimates, and many critic-guided offline methods, e.g. IDQL [2] and QGPO [3], pair an IQL-type critic with a policy tilted by it, and this works well in practice: DAM is competitive with the strongest offline methods (Table 2a). The paper accordingly treats the IQL critic as a surrogate (Sec. 4.2, l.322–323). What DAM adds is that its guarantee needs much less than an exact soft critic. If the critic is correct up to a state-dependent offset, Qϕ=Qβ∗+c(s)Q_\phi=Q^{\ast}_\beta+c(s), every residual log⁡πθ(a0:K∣s)πref(a0:K∣s)−Qϕ/β\log\frac{\pi_{\theta}(a_{0:K}|s)}{\pi_{\text{ref}}(a_{0:K}|s)}-Q_{\phi}/\beta of a state shifts by the same constant c(s)/βc(s)/\beta. The variance in the loss is unchanged, so the minimizer is still the soft-optimal policy (App. B.1, Eq. 36–37).

Medium-Replay vs. Medium-Expert. As described in App. G.1 (l.1746–1749), this pattern follows from the standard D4RL design [4]: Medium-Replay is the full replay buffer of a training run, whereas Medium-Expert concatenates two narrow policies. We verify it with the local action coverage of each dataset (the participation ratio of recorded actions over nearest-neighbour states, where 1 means an even spread):

TaskMediumMedium-ReplayMedium-Expert
HalfCheetah0.3850.4440.411
Hopper0.6520.6900.650
Walker2d0.4700.5060.481

Medium-Replay is the most covered dataset on all three tasks, which supports the explanation in Sec. 5.1.2 (l.405–410) and Sec. 5.1.3 (l.414–418). We add this analysis to the appendix.

[2] IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies, Hansen-Estruch et al., 2023.

[3] Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline RL, Lu et al., ICML 2023.

[4] D4RL: Datasets for Deep Data-Driven Reinforcement Learning, Fu et al., 2020.

Part 3/3 — Matched budgets and estimator costs

Problem 3: Matched budgets, runtime and sensitivity to NN

[Question 3]: Does Appendix E.3 compare DAM and DIME under matched sampling and training budgets, including runtime, memory, and sensitivity to the number of trajectories used by the leave-one-out estimator? These results would affect my assessment of the estimator's practical advantage.

[Answer]: Yes. DAM and DIME share the denoising network, the number of diffusion steps, the replay buffer, the critic and the update-to-data ratio, and use the same number of environment interactions (App. G.2, G.6; Table 7). The number of trajectories per state NN has no counterpart in DIME: it belongs to DAM's leave-one-out advantage estimator (l.270–272), while DIME's pathwise gradient uses no advantage. We therefore compare at matched compute (App. E.3). DAM is faster than five of the seven baselines (Table 6b), and even with N=2N=2, the minimum for a leave-one-out baseline, it outperforms DIME with less wall-clock time (Table 6a, N∈{2,…,32}N\in\{2,\dots,32\}). We will add peak memory for each NN to App. E.3.

Reviewer_j1FfOpenReview ↗
Rating
2
Confidence
2(−2)
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes an algorithm DAM for reinforcement learning of diffusion policies. It first formulate a framework called generalized optimal policy matching, that interprets the policy updates as minimizing discrepancies of the optimal policy and current policy under a specific distance measure. Building on this perspective, DAM trains a diffusion policy by matching the log-probability ratio between the current and reference policies to the estimated soft advantage (in the sense of entropy regularized RL). Finally, they provide experiments showing the effectiveness of the proposed method.

Critical strengths

The paper studies a relevant problem and provides a sound algorithm, showing consistent improvement over a good collection of baselines.

The paper also makes a good description of the method with an adequate amount of maths.

Critical weaknesses

This paper is not well-positioned. The paper seems to aim at unifying RL methods for diffusion models. However, it is overlooking a large body of prior methods. Besides, the only unification seems to happen on the objective level, which is very weak. More concretely, I do not see discussions about important MDP-based methods such as DDPO and DPPO, variational-inference-based methods such as FPO and DiPOD, stochastic-inference-based methods such as Adjoint Matching and QAM, residual policy learning such as DSRL. This also raises question about the true algorithmic novelty of DAM, and how general the proposed method is beyond setting learning a control policy.

Questions for authors
  1. What is the main algorithmic novelty of DAM
  2. How does DAM compare to other diffusion RL works algorithmically, and why is DAM advantageous. A non-exhaustive list includes DDPO, DPPO, FPO, DiPOD, QAM, DSRL.
  3. Section 5 seems to ascribe the mode coverage effect to the fact that the objective minimizes the policy's distance to πrefexp(A/β)\pi_{ref}exp(A/\beta). However a lot of other algorithms admit this property as well. Why do the authors think DAM is particularly good at preserving multi-modality?
2 comments

Part 1/2 — Problem setting and positioning

We thank the reviewer for recognizing DAM as “a sound algorithm, showing consistent improvement over a good collection of baselines”, with “an adequate amount of maths”, and for raising no concerns about its soundness. We group the comments into three problems: (1) positioning and algorithmic novelty, (2) why DAM preserves multimodality, and (3) generality beyond control.

Problem 1: Positioning and algorithmic novelty

[Weakness]: This paper is not well-positioned. The paper seems to aim at unifying RL methods for diffusion models. However, it is overlooking a large body of prior methods. Besides, the only unification seems to happen on the objective level, which is very weak. More concretely, I do not see discussions about important MDP-based methods such as DDPO and DPPO, variational-inference-based methods such as FPO and DiPOD, stochastic-inference-based methods such as Adjoint Matching and QAM, residual policy learning such as DSRL. This also raises question about the true algorithmic novelty of DAM [...]

[Question 1]: What is the main algorithmic novelty of DAM

[Question 2]: How does DAM compare to other diffusion RL works algorithmically, and why is DAM advantageous. A non-exhaustive list includes DDPO, DPPO, FPO, DiPOD, QAM, DSRL.

[Answer]: We believe there is a misunderstanding regarding the problem DAM addresses. DAM is not a method for fine-tuning pretrained diffusion models. As stated in Sec. 3 (l.165), “we focus on the online KL-regularized RL setting”, with a diffusion policy trained off-policy from scratch.

The paper does not aim to unify all diffusion-RL methods. As stated in the Intro (l.058–062) and Sec. 3 (l.161–165), GOPM unifies KL-regularized policy improvement: methods whose update moves the policy toward the soft-optimal target. The methods listed in the review address different problems:

  • Most fine-tune a pretrained model (DDPO, DPPO, Adjoint Matching, DSRL).

  • QAM steers a pretrained behaviour policy with the critic's action gradient ∇aQ\nabla_{a}Q.

  • FPO trains flow-based policies using on-policy PPO-style updates; DiPOD is on-policy and remains a non-peer-reviewed preprint (June 2026), for which ICLR's policy does not require comparisons.

These are relevant works, which we will discuss in the revision, but none targets online, off-policy KL-regularized RL with diffusion policies trained from scratch. For our problem, the established baselines are those of Table 1 (App. G.6.1), and DAM outperforms all of them, which is the consistent improvement the reviewer acknowledges.

We also do not believe it is accurate to judge DAM's novelty by grouping methods into “variational-inference-based” and “stochastic-optimal-control-based” classes. These are general perspectives from which almost every diffusion-RL method, including all of our baselines, can be described, and sharing a perspective does not make two methods algorithmically the same. What distinguishes methods is the problem, the objective and the gradient estimator.

On “the only unification seems to happen on the objective level, which is very weak”.

We respectfully disagree. Unification is one of the three contributions below, and objective-level unification is an established form of contribution, as in control-as-inference [2] and regularized MDPs [3]. In GOPM it is productive, not descriptive: it derives DAM and its offline and hybrid extensions.

Part 2/2 — Algorithmic novelty, multimodality, and generality

Problem 1: Positioning and algorithmic novelty (continued)

The main algorithmic novelty has three parts.

  • A theoretically grounded framework with a practical estimator. GOPM casts soft policy improvement as matching the soft-optimal target and recovers QSM, DIME and SDAC as special cases (Sec. 3; App. C, Props. 3–5). DAM, its diffusion instance, has the unique KL-regularized optimum as its fixed point (Theorem 1), and every update is a soft policy improvement step (Theorem 2). DAM performs this update with a detached estimator and a leave-one-out soft-advantage baseline, without action gradients or backpropagation through the sampler (Sec. 4.1; App. A.5).

  • To the best of our knowledge, DAM is the first diffusion-RL method to support online learning from scratch, offline learning, and hybrid learning (single-stage and two-stage offline-to-online) in a single framework.

  • Strong empirical results across all three regimes. Online, DAM is best on all six headline tasks against seven diffusion-RL baselines, including Dog Trot and Dog Run (Table 1; Figs. 2 and 6). Offline, it is competitive with the strongest offline methods (Table 2a). In hybrid learning, it is ahead of DUAL [1], a recent diffusion offline-to-online method, on 7 of 9 D4RL settings with two-stage training, and on 6 of 9 single-stage with a quarter of the training steps (App. E.2, Table 4). These results are highly significant.

[1] Efficient and Uncertainty-Aware Diffusion Framework for Offline-to-Online Reinforcement Learning (DUAL), Bui et al., ICML 2026.

[2] Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review, Levine, arXiv 2018.

[3] A Theory of Regularized Markov Decision Processes, Geist et al., ICML 2019.

Problem 2: Why DAM preserves multimodality

[Question 3]: Section 5 seems to ascribe the mode coverage effect to the fact that the objective minimizes the policy's distance to π∗\pi^{\ast}. However, a lot of other algorithms admit this property as well. Why do the authors think DAM is particularly good at preserving multi-modality?

[Answer]: The reason is that, among these methods, DAM is the only one that optimizes the KL-regularized objective both faithfully and stably, as we discuss in App. F. The alternatives fall into three groups.

  • No KL-regularized target. Action-gradient methods such as DIPO optimize toward individual high-value actions and rely on auxiliary exploration such as injected noise, which collapses the policy onto a subset of modes (Intro, l.040–048).

  • KL-regularized in name, approximate in practice. As discussed in App. F (l.1672–1690), QSM and DACER omit the entropy from the TD target, and DACER and SDAC replace the log-density with a heuristic proxy.

  • Unstable estimator. DIME backpropagates through the sampler, which destabilizes its actor gradients (Table 5; App. F).

In contrast, DAM optimizes the KL-regularized objective faithfully, with exploration controlled by β\beta (Sec. 5.2), the soft-optimal policy as its fixed point (Theorem 1), and a low-variance leave-one-out advantage (Sec. 4.1). Accordingly, DAM recovers all nine modes with the lowest error on every metric (Fig. 4, Table 2b).

Problem 3: Generality beyond control

[Weakness]: [...] and how general the proposed method is beyond setting learning a control policy.

[Answer]: DAM is not specific to control: it requires only a scalar value per generated sample and the sampler's per-step log-densities, with no gradients of that value and no backpropagation through the sampler (Sec. 4.1; App. A.5). It therefore applies to any KL-regularized reward objective, including non-differentiable rewards. The paper demonstrates this generality along two axes: across policy classes, GOPM also trains a competitive Gaussian policy (App. E.1); and beyond control, DAM solves the 9-GMM distribution-matching problem, recovering all nine modes with the lowest error on every metric (Sec. 5.1.4; Fig. 4, Table 2b). Like all of our baselines (DIME, QSM, DACER, DPMD, QVPO, SDAC, DIPO), we evaluate on continuous control, as our scope is online KL-regularized RL (Sec. 3, l.165).

Comments on the paper

1 comment

Our message to the Area Chair and all reviewers

We sincerely thank the Area Chair and all reviewers for their careful reading and constructive comments. We are encouraged that the reviewers recognized DAM as "a sound algorithm, showing consistent improvement over a good collection of baselines" (Reviewer j1Ff), its detached actor update as "a useful practical contribution" (Reviewer vdXz), and DAM as "a well-motivated and useful approach to diffusion-policy optimization" (Reviewer FH7K). Reviewer j1Ff reports "no substantive concerns about the paper's soundness".

Our main contribution is a theoretically sound framework for online KL-regularized RL with both Gaussian and diffusion policies trained from scratch (App. E.1), with guaranteed soft policy improvement (Theorems 1–2) and a detached estimator that needs no action gradients or backpropagation through the sampler. DAM is best on all six online tasks against seven diffusion-RL baselines (Table 1), and the same objective also supports offline (Table 2a) and hybrid learning, where DAM is ahead of DUAL (ICML 2026) on seven of nine datasets (Table 4).

The two main objections stem from a misunderstanding of the setting: DAM addresses online KL-regularized RL with a diffusion policy trained from scratch (Sec. 3, l.165), whereas the methods it was compared with, including RTB, fine-tune a pretrained model or learn on-policy. Most requested analyses are already in the paper: the tightness condition of the trajectory-level objective (App. A.6), the IQL critic and its offset invariance (Sec. 4.2; App. B.1), the critic's value (Eq. 20), runtime and sensitivity to the number of trajectories (App. E.3), the comparison with DIME under identical settings (Table 5), and multimodality and generality beyond control (Sec. 5.1.4; App. F).

The revisions are minor. The only correction is one wording slip, "independent of the sampling distribution", which App. F already states correctly as "a valid divergence under any covering sampler". We will also move Corollary 1 into the main text, add RTB and the requested analyses, and extend the related work. None of these changes the method, the theory or the reported results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.