acceptodds
Under review as a conference paper at ICLR 2027

Geometry Aware Diffusion Policies for Online Reinforcement Learning

Abstract

Diffusion models have emerged as a powerful class of expressive policies in Reinforcement Learning (RL). In this paper, we propose an improved diffusion policy method for online RL, where the diffusion process is driven entirely by a learned Q-function and adapts to the geometry of the action-value landscape. Unlike conventional diffusion policies that rely on separately trained score networks, our formulation approximates the score function by the gradient of the Q-function, motivated by the Boltzmann exploration. Furthermore, we introduce a state-dependent preconditioner, motivated by the geometry of Q function, using the covariance of Q-gradients across parallel sampling chains without additional network evaluations, which yields structured-anisotropic exploration grounded in information geometry. The algorithm is evaluated on the Gymnasium benchmarks and attains the highest aggregate normalized return among diffusion reinforcement learning baselines.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, diffusion models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_SQhWOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes an actor-free diffusion sampler for online RL, using critic gradients as an approximate score rather than training a separate score network. Its main component is a diagonal preconditioner built from Q-gradient covariance across parallel chains, which rescales the guidance and noise. The highest-Q chain endpoint is selected for execution. The authors motivate the preconditioner through local Hessian geometry and Fisher information, and evaluate the method on six MuJoCo tasks.

Critical strengths
  1. The actor-free formulation offers practical advantages. Generating actions directly from the learned critic removes the need for a separate actor and score-matching objective. Critic updates can immediately influence action generation without first fitting a separate policy network. The formulation retains iterative, stochastic action generation rather than restricting the policy to a Gaussian family. The preconditioner further reuses gradients from parallel chains to adapt sampling directions without Hessian evaluations or additional network calls. This provides a lightweight way to incorporate geometric information into online policy sampling.

  2. The method is competitive across six MuJoCo tasks and achieves the best aggregate normalized mean and IQM among eight baselines. Ten seeds and bootstrap confidence intervals strengthen the empirical evaluation. Figure 4 supports the benefit of adding the preconditioner within the same sampling framework, while the sensitivity studies and runtime analysis provide useful information about its practical behavior.

Critical weaknesses

1. The contribution beyond existing sampling and preconditioning methods is not sufficiently established.

Actor-free Q-gradient sampling has already been explored in LQL/NC-LQL, and the paper discusses related preconditioning approaches in pSGLD, Riemannian MALA and PDS. The main additional contribution appears to be the centered, state- and noise-level-specific covariance estimate across parallel chains, rather than gradient statistics accumulated over iterations as in pSGLD or Fisher-adaptive Langevin sampling [1]. The distinction is clear, but its significance relative to these approaches requires stronger justification. Figure 4 supports the benefit of preconditioning, but does not distinguish directional adaptation from overall scale adjustment (Question 2).

2. The preconditioner is only evaluated with an approximate noisy score.

Eq. (16) adopts the QSM-style approximation s^t(s,at)=∇atQ(s,at)/α\widehat{s}_t(s,a_t)=\nabla_{a_t}Q(s,a_t)/\alpha. This gradient is the exact score of the clean Boltzmann target p0p_0, but generally not of the noisy marginal ptp_t required during denoising, even with an exact critic. For the paper's DDPM kernel, the exact noisy score is

st∗(s,at)=αˉt,E[a0∣at,s]−at1−αˉt.s_t^{\ast}(s,a_t)=\frac{\sqrt{\bar\alpha_t}\\,\mathbb{E}[a_0\mid a_t,s]-a_t}{1-\bar\alpha_t}.

In contrast, iDEM [2] and DQS [3] estimate the noisy marginal score using Monte Carlo, rather than substituting the clean Q-gradient. RFM [4] also derives score/velocity targets from posterior expectations and develops estimators for them. The paper does not examine whether its covariance preconditioner can be used with these corrected score estimates and still improve sampling (Question 1).

Questions for authors

1. Can the preconditioner improve sampling with Monte Carlo estimates of the noisy score?

For the Boltzmann target, the posterior is

p(a0∣at,s)∝qt(at∣a0)exp⁡!(Q(s,a0)α),a0∈A.p(a_0\mid a_t,s)\propto q_t(a_t\mid a_0)\exp\\!\left(\frac{Q(s,a_0)}{\alpha}\right),\qquad a_0\in\mathcal A.

A Monte Carlo estimate of E[a0∣at,s]\mathbb{E}[a_0\mid a_t,s] can be substituted into Eq. (15) to estimate the noisy score. iDEM [2] and DQS [3] provide Monte Carlo noisy-score estimators. RFM [4, Sec. 4.2 and Appendix D] develops posterior-mean estimation and a score-matching formulation; Remark 4.13 explains how the estimates can also be used directly for training-free sampling. RDMC [5] implements direct posterior-based score estimation during sampling. Finite-sample Monte Carlo estimates need not be exact or unbiased.

Can Eq. (16) be replaced with such a score estimate, with the preconditioner computed from the estimated scores across chains? Would it still improve sampling compared with the same score estimator without preconditioning?

2. How much of the improvement comes from directional adaptation?

For example, consider the scalar preconditioner

Ptiso=(tr⁡(Σ^t)d+ϵ)−1/2I,P_t^{\mathrm{iso}}=\left(\frac{\operatorname{tr}(\widehat{\Sigma}_t)}{d}+\epsilon\right)^{-1/2}I,

where the covariance is defined in Eq. (21) and dd is the action dimension. This adapts the overall scale without distinguishing action directions. How does it compare with the proposed preconditioner under the same sampling setup? This comparison would clarify the contribution of directional information.

I am happy to raise my score if these concerns are addressed, particularly the contribution over existing preconditioners and the applicability to corrected-score sampling.

References

[1] FisherMALA. Titsias (2023). Optimal Preconditioning and Fisher Adaptive Langevin Sampling. NeurIPS 2023.

[2] iDEM. Akhound-Sadegh et al. (2024). Iterated Denoising Energy Matching for Sampling from Boltzmann Densities. arXiv:2402.06121.

[3] DQS (Diffusion Q-Sampling). Vineet Jain, Tara Akhound-Sadegh and Siamak Ravanbakhsh (2025). Sampling from Energy-based Policies using Diffusion. arXiv:2410.01312.

[4] RFM (Reverse Flow Matching). Zeyang Li, Sunbochen Tang and Navid Azizan (2026). Reverse Flow Matching: A Unified Framework for Online Reinforcement Learning with Diffusion and Flow Policies. arXiv:2601.08136v2.

[5] RDMC (Reverse Diffusion Monte Carlo). Huang et al. (2024). Reverse Diffusion Monte Carlo. arXiv:2307.02037.

4 comments
AuthorsOpenReview ↗

Response to the reviewer's comment

We thank the reviewer for the detailed review and suggestions for strengthening the contribution. We address each point below.

W1 and Q2. Contribution beyond existing preconditioners, and the role of directional adaptation. The main difference between our preconditioner and existing ones is where its information comes from. pSGLD and FisherMALA build their preconditioners from history information where pSGLD keeps a running average of squared gradients, and FisherMALA [1] estimates the Fisher matrix from the gradients along the sampling chain. PDS uses a fixed preconditioner built from data statistics, and Riemannian MALA requires derivatives of the metric. NC-LQL uses isotropic Langevin steps to obtain samples from the target.

Our preconditioner uses neither history nor fixed statistics. At every reverse step and for every state, we compute from the covariance of the critic gradients across the KK parallel chains at that step, and it reuses multi sampling chains that the sampler already computes.

To separate directional adaptation from overall scale adjustment, we ran the scalar preconditioner suggested by the reviewer in the same sampler.

PreconditionerMean [95% CI]IQM [95% CI]Optimality gap [95% CI]
Proposed91.6 [87.3, 95.9]91.0 [86.3, 95.7]12.4 [9.6, 15.3]
Scalar84.7 [76.4, 92.5]87.7 [79.8, 93.8]18.9 [12.4, 26.0]
Q-only84.2 [79.7, 88.6]83.7 [79.9, 87.3]17.5 [14.0, 21.2]

The scalar preconditioner gives a similar mean to the sampler without preconditioner. The proposed covariance preconditioner improves on the scalar one in all three metrics, so directional information contributes beyond the overall scale adjustment.

We revise the contributions, and add the comparison with existing preconditioners in the related works in Appendix B, and the scalar ablation in Section 5.3.

[1] Titsias. Optimal Preconditioning and Fisher Adaptive Langevin Sampling. NeurIPS 2023.

AuthorsOpenReview ↗

Response to reviewer's comment (follow up)

W2 and Q1. Can the preconditioner improve sampling with Monte Carlo noisy-score estimates?

Eq. 16 follows QSM [1] and is not the exact noisy score. Under the posterior p(a0∣s,at)∝exp⁡(Q(s,a0)/α)qt(at∣a0)p(a_0\mid s,a_t)\propto\exp(Q(s,a_0)/\alpha)q_t(a_t\mid a_0), Tweedie's formula (Eq. 15) gives st∗(at)=αˉtE[a0∣s,at]−at1−αˉt,s_t^*(a_t)=\frac{\sqrt{\bar\alpha_t}\mathbb E[a_0\mid s,a_t]-a_t}{1-\bar\alpha_t}, and, following iDEM [2] and rescaling ata_t by 1/αˉt1/\sqrt{\bar\alpha_t}, it can also be written through the critic gradient: st∗(at)=1ααˉtE[∇aQ(s,a0)∣s,at].s_t^*(a_t)=\frac{1}{\alpha\sqrt{\bar\alpha_t}}\mathbb E\big[\nabla_aQ(s,a_0)\mid s,a_t\big]. We replaced Eq. 16 by Monte Carlo estimates of these two forms of the exact score. At every reverse step and for every chain, we draw JJ clean actions a(j)∼N(at/αˉt,(1−αˉt)/αˉtI)a^{(j)}\sim\mathcal N\big(a_t/\sqrt{\bar\alpha_t},(1-\bar\alpha_t)/\bar\alpha_tI\big) within the action space and weight them by wj∝exp⁡(Q(s,a(j))/α)w_j\propto\exp(Q(s,a^{(j)})/\alpha), with ∑jwj=1\sum_jw_j=1. The weighted samples approximate the posterior without an actor. Following the reference suggested by the reviewer, we consider the following two variants:

  • Gradient estimate (variance preserving; iDEM [2]; DQS [3]): st∗(at)≈1ααˉt∑jwj∇aQ(s,a(j))s_t^*(a_t)\approx\frac{1}{\alpha\sqrt{\bar\alpha_t}}\sum_jw_j\nabla_aQ(s,a^{(j)}). which uses critic gradients.
  • Posterior-mean estimate (RFM [4]; RDMC [5]): E[a0∣s,at]≈∑jwja(j)\mathbb E[a_0\mid s,a_t]\approx\sum_jw_ja^{(j)}, substituted into Eq. 15, which uses critic values.

The preconditioner is computed from the estimated scores across chains, and all other settings remain unchanged.

ScorePreconditionerMean [95% CI]IQM [95% CI]Optimality gap [95% CI]
OursYes91.6 [87.3, 95.9]91.0 [86.3, 95.7]12.4 [9.6, 15.3]
Gradient estimateYes92.2 [87.1, 97.2]92.9 [86.7, 98.3]11.4 [7.6, 15.4]
Gradient estimateNo68.5 [61.0, 76.1]70.3 [60.2, 81.6]33.0 [26.0, 40.0]
Posterior-mean estimateYes80.3 [70.5, 90.0]82.5 [73.0, 91.0]23.5 [16.1, 31.4]
Posterior-mean estimateNo77.8 [68.2, 87.9]76.7 [67.2, 87.7]26.2 [18.9, 33.5]

The preconditioner improves the gradient estimate in all three metrics, where the normalized mean increases from 68.5 to 92.2, which is comparable to ours. With the posterior-mean estimate, we observe a slight improvement with overlap in the confidence intervals. Both estimators evaluate the critic at J=4J=4 clean actions per chain, so they require more computational overhead.

We add these estimators and their results in Table 8 in Appendix L.

[1] M. Psenka, A. Escontrela, P. Abbeel, Y. Ma. Learning a diffusion model policy from rewards via Q-score matching. International Conference on Machine Learning (ICML), 2024.

[2] T. Akhound-Sadegh, J. Rector-Brooks, A. J. Bose, et al. Iterated denoising energy matching for sampling from Boltzmann densities. International Conference on Machine Learning (ICML), 2024.

[3] V. Jain, T. Akhound-Sadegh, S. Ravanbakhsh. Sampling from energy-based policies using diffusion. arXiv:2410.01312, 2025.

[4] Z. Li, S. Tang, N. Azizan. Reverse flow matching: a unified framework for online reinforcement learning with diffusion and flow policies. arXiv:2601.08136, 2026.

[5] X. Huang, H. Dong, Y. Hao, Y. Ma, T. Zhang. Reverse diffusion Monte Carlo. International Conference on Learning Representations (ICLR), 2024.

Thank you for the detailed response and the additional experiments. The Monte Carlo score results and the scalar-preconditioner comparison answer my questions, and I have raised my score.

AuthorsOpenReview ↗

Response to the reviewer's comment

We sincerely thank the reviewer for their positive and encouraging feedback. We appreciate the reviewer's valuable comments and constructive suggestions, which have helped us improve the quality of our paper.

Reviewer_gwuCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The Paper tackles the problem of using diffusion models as a policy class in reinforcement learning (RL). More specifically, the authors propose a novel sampling method for diffusion policies in RL by leveraging the landscape, or the geometry, of the Q function. The main idea behind it to use a state conditioned preconditioner by using the covariance of Q gradients from multiple samples from the diffusion policy.

The paper experimentally validate the method on the gym locomotion control environments and achieve the strongest aggregate normalized performance across different environments and when compared to other methods.

Critical strengths

The strengths of the paper can be summarized as follows:

  1. The paper is clear and easy to follow.
  2. The problem is well motivated and the proposed geometry-aware approach is interesting and practical.
  3. The aggregate normalized performance is strong compared to prior work.
  4. The paper clearly states the assumptions and the theoretical limitations of the work.
Critical weaknesses

The weaknesses of the paper can be summarized as follows:

  1. The approach of the paper is motivated from the entropy regularized RL perspective. However, in algorithm 1, the Q function is updated using the standard Bellman update and there is no entropy maximization part. This creates a conflict between the theory and the implementation.
  2. in Assumption 1, the paper approximates the noisy score using the clean Q-gradient, the paper assumes a bound on the error and the authors acknowledge that this approximation is not exact, which I appreciate. However, this problem has been partially solved in prior work [1] which proposes defining a value function in the noisy space using the expected clean action reward. Although the theory in [1] was primarily targeted towards offline RL, I think the paper can make use of some of the methodology there or at least discuss the connection explicitly.
  3. The proposed method requires making many forward passes (sampling many samples) from the diffusion policy which is computationally expensive. However, this was highlighted in the paper as well.
  4. Finally, it seems that the preconditioned isn't necessarily accurate for a local hessian geometry. The current geometry uses linear approximations and doesn't account for the sampling chain in the preconditioned.

[1] Selim, M., et al. "Noisy-Space Policy Gradient for Diffusion Policies in Offline Reinforcement Learning", ICML'26

Questions for authors
  1. Regarding Weakness #1, is that a missing detail in the algorithm writing or is it an implementation choice ?
2 comments
AuthorsOpenReview ↗

Response to the reviewer's comment

We thank the reviewer for the positive assessment of the clarity of the paper. We address each concern below.

Q and W1. Standard Bellman update: missing detail or implementation choice? It is an implementation choice. We do not solve the MaxEnt RL problem with a soft Bellman backup. Following NC-LQL [1], and in line with QSM [2], we use the entropy in the policy-improvement step and train the critic with a standard Bellman update.

For a fixed state ss and a given critic QQ, the policy-improvement step solves the entropy-regularized problem max⁡π(⋅∣s) Ea∼π[Q(s,a)]+αH(π(⋅∣s))subject to∫π(a∣s)da=1.\max_{\pi(\cdot\mid s)}\ \mathbb E_{a\sim\pi}\big[Q(s,a)\big]+\alpha\mathcal H\big(\pi(\cdot\mid s)\big)\quad\text{subject to}\quad\int\pi(a\mid s)da=1. Its closed-form solution is π(a∣s)=exp⁡(Q(s,a)/α)Z(s).\pi(a\mid s)=\frac{\exp\big(Q(s,a)/\alpha\big)}{Z(s)}. This holds for any critic, whether it is trained by a standard or a soft Bellman update. With a standard update, QQ estimates the value of the current Boltzmann policy.

Our technical contributions concern sampling from this Boltzmann distribution for a given critic, so they do not depend on how the critic is trained. The score (Eq. 14) follows from the Gibbs form. The Hessian–Riemannian construction (Eq. 17) and the covariance surrogate (Eqs. 19–21) use only the local curvature of the given critic. The target-invariance argument (Appendix C) holds for whichever distribution exp⁡(Q/α)/Z\exp(Q/\alpha)/Z defines.

Exact MaxEnt optimality would require the soft Q-function. A soft Bellman update needs the entropy, and estimating it is beyond the scope of this paper.

We revise Section 3 to state this distinction clearly.

[1] D. Ki, H.-J. Ahn, K. Kim, B.-J. Lee. Direct soft-policy sampling via Langevin dynamics. arXiv:2602.07873, 2026.

[2] M. Psenka, A. Escontrela, P. Abbeel, Y. Ma. Learning a diffusion model policy from rewards via Q-score matching. International Conference on Machine Learning (ICML), 2024.

W2. Noisy-Space Policy Gradient method (Selim et al., ICML'26).

Thank you for the reference, which helps us improve the paper.

NSPG defines a noisy-space Q-function as the expectation of the critic under the posterior of the clean action given the noisy action, Qnoisy(s,at)=Ea∼p(⋅∣s,at)[Q(s,a)]Q_{\rm noisy}(s,a_t)=\mathbb E_{a\sim p(\cdot\mid s,a_t)}[Q(s,a)]. The posterior is defined by the forward kernel qtq_t and the action distribution of the diffusion actor. The gradient of QnoisyQ_{\rm noisy} is written as ∇atQnoisy(s,at)=Ea∼p(⋅∣s,at)[(Q(s,a)−Qnoisy(s,at))∇atlog⁡qt(at∣a)].\nabla_{a_t}Q_{\rm noisy}(s,a_t)=\mathbb E_{a\sim p(\cdot\mid s,a_t)}\Big[\big(Q(s,a)-Q_{\rm noisy}(s,a_t)\big)\nabla_{a_t}\log q_t(a_t\mid a)\Big]. NSPG estimates this gradient by Monte Carlo and uses it to improve the policy. Its posterior sampling needs a complete run of the diffusion actor from ata_t to step zero, which is not available in our setting since our sampler does not train an actor.

For our target p0(a)∝eQ(s,a)/αp_0(a)\propto e^{Q(s,a)/\alpha}, following iDEM [1] and taking the gradient of log⁡pt(at)∝log⁡∫eQ(s,a)/αqt(at∣a)da\log p_t(a_t)\propto\log\int e^{Q(s,a)/\alpha}q_t(a_t\mid a)da gives st∗(at)=Ea∼p(⋅∣s,at)[∇atlog⁡qt(at∣a)]=1ααˉtEa∼p(⋅∣s,at)[∇aQ(s,a)],s_t^{\ast}(a_t)=\mathbb E_{a\sim p(\cdot\mid s,a_t)}\big[\nabla_{a_t}\log q_t(a_t\mid a)\big]=\frac{1}{\alpha\sqrt{\bar\alpha_t}}\mathbb E_{a\sim p(\cdot\mid s,a_t)}\big[\nabla_aQ(s,a)\big], where p(a∣s,at)∝eQ(s,a)/αqt(at∣a)p(a\mid s,a_t)\propto e^{Q(s,a)/\alpha}q_t(a_t\mid a). Both NSPG and our score are posterior averages under the forward kernel, but NSPG uses the diffusion actor as the prior, while we use the Boltzmann target to define the score. Our Eq. 16 approximates the posterior average of the critic gradient by the critic gradient at the noisy action. It needs no posterior samples.

The error of our approximation to the exact score is bounded in our response to Reviewer DZDT (W1). (E∥s^t−st∗∥2)1/2≤αˉt−1/2−1α(Lm2+d+G)+Lααˉtd(1−αˉt)αˉt,\Big(\mathbb E\|\widehat s_t-s_t^*\|^2\Big)^{1/2}\le \frac{\bar\alpha_t^{-1/2}-1}{\alpha}\big(L\sqrt{m_2+d}+G\big)+\frac{L}{\alpha\sqrt{\bar\alpha_t}}\sqrt{\frac{d(1-\bar\alpha_t)}{\bar\alpha_t}}, and it goes to zero as the noise goes to zero.

Since our sampler is actor-free, we follow the similar idea as NSPG in [1] and [2]. At every reverse step and for every chain, we sample a few clean actions from N(at/αˉt,(1−αˉt)/αˉtI)\mathcal N\big(a_t/\sqrt{\bar\alpha_t},(1-\bar\alpha_t)/\bar\alpha_tI\big) and weight them by exp⁡(Q(s,a)/α)\exp(Q(s,a)/\alpha), which gives an estimate of the posterior average above. We conduct the related experiments with the same settings in the paper. These results are reported in Table 8 in Appendix L.

We also add this discussion in Appendix L, and cite NSPG in the related works.

[1] T. Akhound-Sadegh, J. Rector-Brooks, A. J. Bose, et al. Iterated denoising energy matching for sampling from Boltzmann densities. International Conference on Machine Learning (ICML), 2024.

[2] Z. Li, S. Tang, N. Azizan. Reverse flow matching: a unified framework for online reinforcement learning with diffusion and flow policies. arXiv:2601.08136, 2026.

AuthorsOpenReview ↗

Response to reviewer's comment (follow up)

W3. Computational cost of multi-chain sampling. We agree that sampling several chains per state requires more forward passes y. Multi-chain sampling is an established way to improve the returns of diffusion policies. It was proposed in QVPO [1], and several baselines in our comparison also draw multiple candidate actions per state and select among them based on the critic.

Our aim is to further leverage these parallel chains for RL without introducing much additional computation. The preconditioner is computed from the Q-gradients that the chains already evaluate. It therefore needs no extra network evaluations and adds only O(Kd)O(Kd) element-wise operations per reverse step.

[1] S. Ding, et al. Diffusion-based reinforcement learning via Q-weighted variational policy optimization. Advances in Neural Information Processing Systems (NeurIPS), 2024.

W4. Accuracy of the Hessian surrogate and the sampling chain. Eq. 18-19 relies on an approximation of the critic gradient. We now bound the error of this approximation to show the accuracy and how it depends on the spread of the sampling chains.

Following Sec. 4.3, let the chains be at=a0+σtϵa_t=a_0+\sigma_t\epsilon with ϵ∼N(0,I)\epsilon\sim\mathcal N(0,I), and H=∇a2Q(s,a0)H=\nabla_a^2Q(s,a_0). Assume that the Hessian is LHL_H-Lipschitz near a0a_0. Then based on the Taylor's theorem, we bound the remainder of the linearization: ∇aQ(s,at)=∇aQ(s,a0)+σtHϵ+R,∥R∥≤LH2σt2∥ϵ∥2.\nabla_aQ(s,a_t)=\nabla_aQ(s,a_0)+\sigma_tH\epsilon+R,\qquad\|R\|\le\frac{L_H}{2}\sigma_t^2\|\epsilon\|^2. The gradient covariance can be written as Cov[∇aQ(s,at)]=σt2HH⊤+σtHCov(ϵ,R)+σtCov(R,ϵ)H⊤+Cov(R).\mathrm{Cov}\big[\nabla_aQ(s,a_t)\big]=\sigma_t^2HH^\top+\sigma_tH\mathrm{Cov}(\epsilon,R)+\sigma_t\mathrm{Cov}(R,\epsilon)H^\top+\mathrm{Cov}(R). By the Cauchy–Schwarz inequality, ∥Cov(ϵ,R)∥≤(E∥ϵ∥2E∥R∥2)1/2\|\mathrm{Cov}(\epsilon,R)\|\le(\mathbb E\|\epsilon\|^2\mathbb E\|R\|^2)^{1/2} and ∥Cov(R)∥≤E∥R∥2\|\mathrm{Cov}(R)\|\le\mathbb E\|R\|^2. Using E∥ϵ∥2=d\mathbb E\|\epsilon\|^2=d and E∥ϵ∥4=d(d+2)\mathbb E\|\epsilon\|^4=d(d+2), we obtain ∥Cov[∇aQ(s,at)]−σt2HH⊤∥≤LH∥H∥dd+2σt3+LH2d(d+2)4σt4.\big\|\mathrm{Cov}\big[\nabla_aQ(s,a_t)\big]-\sigma_t^2HH^\top\big\|\le L_H\|H\|d\sqrt{d+2}\sigma_t^3+\frac{L_H^2d(d+2)}{4}\sigma_t^4. We denote the right-hand side by EtE_t. The term σt2HH⊤\sigma_t^2HH^\top is of order σt2\sigma_t^2, while the error EtE_t is of order σt3\sigma_t^3, and the relative error is therefore of order LHσt/∥H∥L_H\sigma_t/\|H\|. The approximation is accurate when σt\sigma_t is small at the late, low-noise steps.

We also compared our covariance preconditioner PtcovP^{\rm cov}_t with the preconditioner PtHessP^{\rm Hess}_t built from the exact Hessian, along the reverse process. Both cosine similarities stay high and increase towards the final sample, which is consistent with the bound.

Reverse step tt0123456789
cos⁡(Ptcov,PtHess)\cos\big(P^{\rm cov}_t,P^{\rm Hess}_t\big)0.970.960.970.960.950.940.940.930.920.92
cos⁡(Ptcov∇aQ,PtHess∇aQ)\cos\big(P^{\rm cov}_t\nabla_aQ,P^{\rm Hess}_t\nabla_aQ\big)0.980.980.980.970.970.950.950.940.930.93

We state this bound in Appendix K.2, summarize it in Section 4.3.

Reviewer_DZDTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper learns a Q-function through temporal-difference (TD) updates and uses its action gradient to approximate the noise-dependent score required for reverse diffusion. At each sampling step, it estimates gradient covariance across parallel candidate actions and constructs a diagonal preconditioner to rescale both the guidance and the noise.

Although the method reports strong aggregate results, its per-task performance is mixed, and the aggregate comparisons should be interpreted cautiously. Reporting each method’s mean rank across tasks, alongside returns, could provide a complementary perspective.

Overall, the major concern is the justification for the score approximation. The Q-gradient provides the exact score of the corresponding clean Boltzmann distribution, up to temperature scaling, but not generally of its noisy intermediate marginals. The paper does not establish that this approximation error is sufficiently small throughout the reverse-diffusion trajectory.

Critical strengths
  • The method reuses Q-gradients across parallel candidate actions to construct a diagonal covariance-based preconditioner, avoiding explicit Hessian computation and requiring no additional network evaluations for this step. Rescaling both the guidance and the noise provides a practical, low-overhead way to incorporate estimated curvature information into reverse-diffusion sampling for online RL.

  • Across six MuJoCo tasks, the method achieves the highest reported aggregate mean and interquartile mean of normalized returns among the compared diffusion-based methods. The comparison against the same parallel Q-gradient sampler without preconditioning provides evidence that the preconditioner improves performance beyond the benefits of generating multiple candidates.

Critical weaknesses
  • Eq. 16 substitutes the learned critic’s action gradient for the noise-dependent score, although the exact Q-gradient identity applies only to the clean Boltzmann target. TD value fitting does not directly ensure gradient accuracy, particularly at noisy actions. Assumption 4.1 allows arbitrarily large finite errors and therefore does not establish that the approximation is accurate enough to justify the reverse-diffusion interpretation.

  • Per-task results are mixed. The return comparisons omit standard online-RL baselines such as SAC, while equal environment-interaction budgets do not ensure comparable computational costs. Thus, the results support competitiveness within the selected diffusion baselines, but broader claims require stronger baseline coverage and compute-aware comparisons. Mean ranks across tasks could provide a complementary performance summary.

Questions for authors

Clarification of Assumption 4.1: Can you quantify the magnitude and tightness of the score-error bound ϵt\epsilon_t across diffusion noise levels, particularly at high noise? As written, the assumption permits arbitrarily large finite errors, so it is unclear whether it provides meaningful support for using the approximation in Eq. 16 throughout the reverse-diffusion process.

2 comments
AuthorsOpenReview ↗

Response to the reviewer's comment

We thank the reviewer for the valuable suggestions on the paper. We address each point below.

W1 and Q. Score approximation and Assumption 4.1.

We agree that Assumption 4.1 does not guarantee a small error and we further derive an explicit bound on εt\varepsilon_t theoretically.

Firstly, we consider a state ss. Let QπQ^\pi be the ideal critic, stπs_t^\pi the exact score of the noisy marginal$\exp(Q^\pi/\alpha)$, and εQ\varepsilon_Q the critic error.

Firstly, following iDEM [1], we can write the exact score with the critic gradient. Let σt2=(1−αˉt)/αˉt\sigma_t^2=(1-\bar\alpha_t)/\bar\alpha_t, then the noisy marginal can be written as pt(at)∝Ea∼N(a^0,σt2I)[p0(a)].p_t(a_t)\propto\mathbb E_{a\sim\mathcal N(\hat a_0,\sigma_t^2I)}\big[p_0(a)\big]. Since ata_t only appears in the mean a^0=at/αˉt\hat a_0=a_t/\sqrt{\bar\alpha_t} and with ∇log⁡p0=∇Q/α\nabla\log p_0=\nabla Q/\alpha, we obtain s_t^*(a_t)=\frac{1}{\alpha\sqrt{\bar\alpha_t}}\mathbb E\big[\nabla Q(a_0)\mid a_t\big].\tag{R1}

Then we bound the error of Eq. 16. By subtracting (R1) from Eq. 16, it gives s^t−st∗=1α∇Q(at)−1ααˉtE[∇Q(a0)∣at].\widehat s_t-s_t^*=\frac{1}{\alpha}\nabla Q(a_t)-\frac{1}{\alpha\sqrt{\bar\alpha_t}}\mathbb E\big[\nabla Q(a_0)\mid a_t\big]. We compare both with the gradient at a^0=at/αˉt\hat a_0=a_t/\sqrt{\bar\alpha_t}, the point at which the Gaussian likelihood qt(at∣a0)q_t(a_t\mid a_0) is centred in a0a_0. We add and subtract 1α∇Q(a^0)\frac{1}{\alpha}\nabla Q(\hat a_0) and 1ααˉt∇Q(a^0)\frac{1}{\alpha\sqrt{\bar\alpha_t}}\nabla Q(\hat a_0). Since a^0\hat a_0 is fixed given ata_t, we obtain s^t−st∗=1α[∇Q(at)−∇Q(a^0)]+1α(1−αˉt−1/2)∇Q(a^0)+1ααˉtE[∇Q(a^0)−∇Q(a0)∣at].\widehat s_t-s_t^*=\frac{1}{\alpha}\big[\nabla Q(a_t)-\nabla Q(\hat a_0)\big]+\frac{1}{\alpha}\big(1-\bar\alpha_t^{-1/2}\big)\nabla Q(\hat a_0)+\frac{1}{\alpha\sqrt{\bar\alpha_t}}\mathbb E\big[\nabla Q(\hat a_0)-\nabla Q(a_0)\mid a_t\big].

Then we derive the bound for each term. We use the forward diffusion process at=αˉta0+1−αˉtϵa_t=\sqrt{\bar\alpha_t}a_0+\sqrt{1-\bar\alpha_t}\epsilon, where ϵ∼N(0,I)\epsilon\sim\mathcal N(0,I), and dd is the action dimension.

  • For the first term, since ∇Q\nabla Q is LL-Lipschitz (Assumption 4.2) and ∥at−a^0∥=(αˉt−1/2−1)∥at∥\|a_t-\hat a_0\|=(\bar\alpha_t^{-1/2}-1)\|a_t\|, its norm is at most Lα(αˉt−1/2−1)∥at∥\frac{L}{\alpha}(\bar\alpha_t^{-1/2}-1)\|a_t\|. If p0p_0 has a finite second moment m2=E∥a0∥2m_2=\mathbb E\|a_0\|^2, then we have E∥at∥2=αˉtm2+(1−αˉt)d≤m2+d\mathbb E\|a_t\|^2=\bar\alpha_tm_2+(1-\bar\alpha_t)d\le m_2+d. Its root mean square is therefore at most Lα(αˉt−1/2−1)m2+d\frac{L}{\alpha}(\bar\alpha_t^{-1/2}-1)\sqrt{m_2+d}.
  • For the second term, if the critic gradient is bounded by GG, its norm is at most Gα(αˉt−1/2−1)\frac{G}{\alpha}(\bar\alpha_t^{-1/2}-1).
  • For the third term, based on the same Lipschitz property, its norm is at most LααˉtE[∥a0−a^0∥∣at]\frac{L}{\alpha\sqrt{\bar\alpha_t}}\mathbb E[\|a_0-\hat a_0\|\mid a_t]. Dividing the forward process by αˉt\sqrt{\bar\alpha_t} gives a0−a^0=−(1−αˉt)/αˉtϵa_0-\hat a_0=-\sqrt{(1-\bar\alpha_t)/\bar\alpha_t}\epsilon. Since E∥ϵ∥2=d\mathbb E\|\epsilon\|^2=d, we have E∥a0−a^0∥2=d(1−αˉt)/αˉt\mathbb E\|a_0-\hat a_0\|^2=d(1-\bar\alpha_t)/\bar\alpha_t. By Jensen's inequality, its root mean square is at most Lααˉtd(1−αˉt)/αˉt\frac{L}{\alpha\sqrt{\bar\alpha_t}}\sqrt{d(1-\bar\alpha_t)/\bar\alpha_t}.

By Minkowski's inequality and adding the three bounds, we obtain (E∥s^t−st∗∥2)1/2≤αˉt−1/2−1α(Lm2+d+G)+Lααˉtd(1−αˉt)αˉt.\Big(\mathbb E\|\widehat s_t-s_t^*\|^2\Big)^{1/2}\le\frac{\bar\alpha_t^{-1/2}-1}{\alpha}\big(L\sqrt{m_2+d}+G\big)+\frac{L}{\alpha\sqrt{\bar\alpha_t}}\sqrt{\frac{d(1-\bar\alpha_t)}{\bar\alpha_t}}. Therefore, Assumption 4.1 holds with εt\varepsilon_t equal to the right-hand side, and it goes to zero as the noise goes to zero (αˉt→1\bar\alpha_t\to1).

Our target is defined by the learned critic, so this bound is the error of Eq. 16 for the distribution that our sampler actually samples from.

The critic error εQ\varepsilon_Q also changes this target. Since the gradient of Qϕ−QπQ_\phi-Q^\pi is MM-Lipschitz, based on the descent lemma, for any actions aa and bb we have ∣(Qϕ−Qπ)(b)−(Qϕ−Qπ)(a)−⟨∇Qϕ(a)−∇Qπ(a),b−a⟩∣≤M2∥b−a∥2.\Big|(Q_\phi-Q^\pi)(b)-(Q_\phi-Q^\pi)(a)-\big\langle\nabla Q_\phi(a)-\nabla Q^\pi(a),b-a\big\rangle\Big|\le\frac{M}{2}\|b-a\|^2. When we apply this inequality at b=at+rub=a_t+ru and at b=at−rub=a_t-ru for any r>0r>0, where uu is the unit vector along ∇Qϕ(at)−∇Qπ(at)\nabla Q_\phi(a_t)-\nabla Q^\pi(a_t), and subtract the two inequalities, we obtain 2r∥∇Qϕ(at)−∇Qπ(at)∥≤(Qϕ−Qπ)(at+ru)−(Qϕ−Qπ)(at−ru)+Mr2≤2εQ+Mr2.2r\|\nabla Q_\phi(a_t)-\nabla Q^\pi(a_t)\|\le(Q_\phi-Q^\pi)(a_t+ru)-(Q_\phi-Q^\pi)(a_t-ru)+Mr^2\le2\varepsilon_Q+Mr^2. Dividing by 2r2r and taking r=2εQ/Mr=\sqrt{2\varepsilon_Q/M}, we obtain ∥∇Qϕ(at)−∇Qπ(at)∥≤2εQM\|\nabla Q_\phi(a_t)-\nabla Q^\pi(a_t)\|\le\sqrt{2\varepsilon_QM}.

We state this bound as Theorem K.1 in Appendix K.1 of the revised paper, and summarize it in Section 4.1. The additional critic-gradient error bound above illustrates how critic approximation error may affect the target distribution.

[1] T. Akhound-Sadegh, J. Rector-Brooks, A. J. Bose, et al. Iterated denoising energy matching for sampling from Boltzmann densities. International Conference on Machine Learning (ICML), 2024.

AuthorsOpenReview ↗

Response to reviewer's comment (follow up)

W2 (1). Mixed per-task results and mean rank. We add the mean rank over the six tasks. The ranking covers 11 methods, including two additional off-policy RL baselines, SAC and TD3.

MethodOursSDACNC-LQLDIMETD3DACERDPMDSACDIPOQSMQVPO
Mean rank3.334.174.835.175.505.676.506.836.836.8310.33

Our method has the best mean rank, although the per-task results are mixed. We add the mean rank as a column of Table 1.

W2 (2). Broader RL baselines and compute-aware comparison.

For a broader comparison, we add two off-policy algorithms, SAC and TD3, to the results section. They use the same environment settings as our method. The results are also shown below:

Mean ± SD [95% CI]IQM [95% CI]Optimality gap [95% CI]
Ours91.6 ± 18.7 [87.3, 95.9]91.0 [86.3, 95.7]12.4 [9.6, 15.3]
TD376.9 ± 27.0 [69.2, 84.7]78.6 [70.3, 84.9]25.7 [19.6, 32.3]
SAC68.4 ± 30.5 [59.4, 76.8]67.6 [55.8, 78.2]33.6 [26.2, 41.8]

For the compute-aware comparison, we train every method for 10610^6 steps on one NVIDIA L40S GPU, using the settings in the paper. The results are shown below.

MethodTrainable paramsTraining FLOPs (10610^6 steps, TFLOP)Inference FLOPs (per action, MFLOP)Training wall-clock time (min)
SAC415,2472290.36.5
QSM415,24010,442196.016.8
DIPO423,4323,39329.527.7
DPMD420,40911,009207.230.7
SDAC420,40926,302207.231.3
DIME420,4161,3916.034.8
Ours275,97019,585373.046.2
DACER420,92361,8676.046.9
NC-LQL286,30640,297767.390.1
QVPO420,40941,178207.2180.8

Our method has the fewest trainable parameters, because it does not train an actor. Its training cost lies within the range of the diffusion-based baselines. It needs fewer training FLOPs than SDAC, DACER, NC-LQL and QVPO, and less wall-clock time than DACER, NC-LQL and QVPO.

We add SAC and TD3 to the baselines to Table 1 in Section 5.2, and report the compute-aware comparison in Table 9 in Appendix M.

Comments on the paper

1 comment

Dear reviewers and AC,

We thank all reviewers for their careful reviews and valuable suggestions, and for finding our proposed diffusion RL method practical where the sampling is driven by a learned Q function and adapted to the geometry of the action-value landscape efficiently, and the paper clear and easy to follow. We have revised the paper accordingly, and all changes are highlighted in blue. The main revisions are summarized below.

1. Theoretical guarantees for the approximations

  • Score approximation error. We bound the error of the Q-gradient score approximation in Eq. 16 (Appendix K.1), and summarize it in Section 4.1. The bound goes to zero during the low-noise steps that determine the final sample. (DZDT W1 and Q; gwuC W2)
  • Linearization error for the covariance surrogate. We bound the error of the local linear approximation for the covariance surrogate (Appendix K.2), and summarize it in Section 4.3. The approximation is accurate at the late, low-noise steps. (gwuC W4)

2. Broader evaluation

  • More baselines and mean rank. We add SAC and TD3 as baselines, and the mean rank over the six tasks (Section 5.2 and Table 1). Our method has the highest aggregate normalized return and the best mean rank. (DZDT W2)
  • Compute-aware comparison. We compare all methods in trainable parameters, training and inference FLOPs, and wall-clock time (Appendix M). (DZDT W2)
  • Other estimation methods with preconditioner. We conduct additional experiments where we replace Eq. 16 with Monte Carlo estimates of the noisy score and evaluate them with and without the proposed preconditioner (Appendix L). The results show that our preconditioner improves the normalized mean, particularly with the gradient-based estimator. (SQhW W2 and Q1; gwuC W2)
  • Directional information matters. We compare the covariance preconditioner with a scalar preconditioner that only adjusts the overall step size (Section 5.3). Our covariance preconditioner improves on it in returns. (SQhW W1 and Q2)

3. Clarifications

  • Contribution clarified. We revise the contributions to state that our preconditioner is state-and-noise-level-dependent and to distinguish it from the existing methods. (SQhW W1)
  • Critic update clarified. We revise Section 3 to explicitly state that the entropy enters the policy-improvement step and that the critic is trained with a standard Bellman update. (gwuC W1 and Q)
  • Connection to noisy-space value functions. We discuss NSPG in Appendix L, and add NSPG, iDEM, DQS, RDMC and RFM to the related works (Appendix B). (gwuC W2; SQhW W2)

We hope these revisions address the reviewers' concerns.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.