- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes an actor-free diffusion sampler for online RL, using critic gradients as an approximate score rather than training a separate score network. Its main component is a diagonal preconditioner built from Q-gradient covariance across parallel chains, which rescales the guidance and noise. The highest-Q chain endpoint is selected for execution. The authors motivate the preconditioner through local Hessian geometry and Fisher information, and evaluate the method on six MuJoCo tasks.
Critical strengths
-
The actor-free formulation offers practical advantages. Generating actions directly from the learned critic removes the need for a separate actor and score-matching objective. Critic updates can immediately influence action generation without first fitting a separate policy network. The formulation retains iterative, stochastic action generation rather than restricting the policy to a Gaussian family. The preconditioner further reuses gradients from parallel chains to adapt sampling directions without Hessian evaluations or additional network calls. This provides a lightweight way to incorporate geometric information into online policy sampling.
-
The method is competitive across six MuJoCo tasks and achieves the best aggregate normalized mean and IQM among eight baselines. Ten seeds and bootstrap confidence intervals strengthen the empirical evaluation. Figure 4 supports the benefit of adding the preconditioner within the same sampling framework, while the sensitivity studies and runtime analysis provide useful information about its practical behavior.
Critical weaknesses
1. The contribution beyond existing sampling and preconditioning methods is not sufficiently established.
Actor-free Q-gradient sampling has already been explored in LQL/NC-LQL, and the paper discusses related preconditioning approaches in pSGLD, Riemannian MALA and PDS. The main additional contribution appears to be the centered, state- and noise-level-specific covariance estimate across parallel chains, rather than gradient statistics accumulated over iterations as in pSGLD or Fisher-adaptive Langevin sampling [1]. The distinction is clear, but its significance relative to these approaches requires stronger justification. Figure 4 supports the benefit of preconditioning, but does not distinguish directional adaptation from overall scale adjustment (Question 2).
2. The preconditioner is only evaluated with an approximate noisy score.
Eq. (16) adopts the QSM-style approximation . This gradient is the exact score of the clean Boltzmann target , but generally not of the noisy marginal required during denoising, even with an exact critic. For the paper's DDPM kernel, the exact noisy score is
In contrast, iDEM [2] and DQS [3] estimate the noisy marginal score using Monte Carlo, rather than substituting the clean Q-gradient. RFM [4] also derives score/velocity targets from posterior expectations and develops estimators for them. The paper does not examine whether its covariance preconditioner can be used with these corrected score estimates and still improve sampling (Question 1).
Questions for authors
1. Can the preconditioner improve sampling with Monte Carlo estimates of the noisy score?
For the Boltzmann target, the posterior is
A Monte Carlo estimate of can be substituted into Eq. (15) to estimate the noisy score. iDEM [2] and DQS [3] provide Monte Carlo noisy-score estimators. RFM [4, Sec. 4.2 and Appendix D] develops posterior-mean estimation and a score-matching formulation; Remark 4.13 explains how the estimates can also be used directly for training-free sampling. RDMC [5] implements direct posterior-based score estimation during sampling. Finite-sample Monte Carlo estimates need not be exact or unbiased.
Can Eq. (16) be replaced with such a score estimate, with the preconditioner computed from the estimated scores across chains? Would it still improve sampling compared with the same score estimator without preconditioning?
2. How much of the improvement comes from directional adaptation?
For example, consider the scalar preconditioner
where the covariance is defined in Eq. (21) and is the action dimension. This adapts the overall scale without distinguishing action directions. How does it compare with the proposed preconditioner under the same sampling setup? This comparison would clarify the contribution of directional information.
I am happy to raise my score if these concerns are addressed, particularly the contribution over existing preconditioners and the applicability to corrected-score sampling.
References
[1] FisherMALA. Titsias (2023). Optimal Preconditioning and Fisher Adaptive Langevin Sampling. NeurIPS 2023.
[2] iDEM. Akhound-Sadegh et al. (2024). Iterated Denoising Energy Matching for Sampling from Boltzmann Densities. arXiv:2402.06121.
[3] DQS (Diffusion Q-Sampling). Vineet Jain, Tara Akhound-Sadegh and Siamak Ravanbakhsh (2025). Sampling from Energy-based Policies using Diffusion. arXiv:2410.01312.
[4] RFM (Reverse Flow Matching). Zeyang Li, Sunbochen Tang and Navid Azizan (2026). Reverse Flow Matching: A Unified Framework for Online Reinforcement Learning with Diffusion and Flow Policies. arXiv:2601.08136v2.
[5] RDMC (Reverse Diffusion Monte Carlo). Huang et al. (2024). Reverse Diffusion Monte Carlo. arXiv:2307.02037.
4 comments
Response to reviewer's comment (follow up)
W2 and Q1. Can the preconditioner improve sampling with Monte Carlo noisy-score estimates?
Eq. 16 follows QSM [1] and is not the exact noisy score. Under the posterior , Tweedie's formula (Eq. 15) gives and, following iDEM [2] and rescaling by , it can also be written through the critic gradient: We replaced Eq. 16 by Monte Carlo estimates of these two forms of the exact score. At every reverse step and for every chain, we draw clean actions within the action space and weight them by , with . The weighted samples approximate the posterior without an actor. Following the reference suggested by the reviewer, we consider the following two variants:
- Gradient estimate (variance preserving; iDEM [2]; DQS [3]): . which uses critic gradients.
- Posterior-mean estimate (RFM [4]; RDMC [5]): , substituted into Eq. 15, which uses critic values.
The preconditioner is computed from the estimated scores across chains, and all other settings remain unchanged.
| Score | Preconditioner | Mean [95% CI] | IQM [95% CI] | Optimality gap [95% CI] |
|---|---|---|---|---|
| Ours | Yes | 91.6 [87.3, 95.9] | 91.0 [86.3, 95.7] | 12.4 [9.6, 15.3] |
| Gradient estimate | Yes | 92.2 [87.1, 97.2] | 92.9 [86.7, 98.3] | 11.4 [7.6, 15.4] |
| Gradient estimate | No | 68.5 [61.0, 76.1] | 70.3 [60.2, 81.6] | 33.0 [26.0, 40.0] |
| Posterior-mean estimate | Yes | 80.3 [70.5, 90.0] | 82.5 [73.0, 91.0] | 23.5 [16.1, 31.4] |
| Posterior-mean estimate | No | 77.8 [68.2, 87.9] | 76.7 [67.2, 87.7] | 26.2 [18.9, 33.5] |
The preconditioner improves the gradient estimate in all three metrics, where the normalized mean increases from 68.5 to 92.2, which is comparable to ours. With the posterior-mean estimate, we observe a slight improvement with overlap in the confidence intervals. Both estimators evaluate the critic at clean actions per chain, so they require more computational overhead.
We add these estimators and their results in Table 8 in Appendix L.
[1] M. Psenka, A. Escontrela, P. Abbeel, Y. Ma. Learning a diffusion model policy from rewards via Q-score matching. International Conference on Machine Learning (ICML), 2024.
[2] T. Akhound-Sadegh, J. Rector-Brooks, A. J. Bose, et al. Iterated denoising energy matching for sampling from Boltzmann densities. International Conference on Machine Learning (ICML), 2024.
[3] V. Jain, T. Akhound-Sadegh, S. Ravanbakhsh. Sampling from energy-based policies using diffusion. arXiv:2410.01312, 2025.
[4] Z. Li, S. Tang, N. Azizan. Reverse flow matching: a unified framework for online reinforcement learning with diffusion and flow policies. arXiv:2601.08136, 2026.
[5] X. Huang, H. Dong, Y. Hao, Y. Ma, T. Zhang. Reverse diffusion Monte Carlo. International Conference on Learning Representations (ICLR), 2024.
Thank you for the detailed response and the additional experiments. The Monte Carlo score results and the scalar-preconditioner comparison answer my questions, and I have raised my score.
Response to the reviewer's comment
We sincerely thank the reviewer for their positive and encouraging feedback. We appreciate the reviewer's valuable comments and constructive suggestions, which have helped us improve the quality of our paper.
Response to the reviewer's comment
We thank the reviewer for the detailed review and suggestions for strengthening the contribution. We address each point below.
W1 and Q2. Contribution beyond existing preconditioners, and the role of directional adaptation. The main difference between our preconditioner and existing ones is where its information comes from. pSGLD and FisherMALA build their preconditioners from history information where pSGLD keeps a running average of squared gradients, and FisherMALA [1] estimates the Fisher matrix from the gradients along the sampling chain. PDS uses a fixed preconditioner built from data statistics, and Riemannian MALA requires derivatives of the metric. NC-LQL uses isotropic Langevin steps to obtain samples from the target.
Our preconditioner uses neither history nor fixed statistics. At every reverse step and for every state, we compute from the covariance of the critic gradients across the K parallel chains at that step, and it reuses multi sampling chains that the sampler already computes.
To separate directional adaptation from overall scale adjustment, we ran the scalar preconditioner suggested by the reviewer in the same sampler.
The scalar preconditioner gives a similar mean to the sampler without preconditioner. The proposed covariance preconditioner improves on the scalar one in all three metrics, so directional information contributes beyond the overall scale adjustment.
We revise the contributions, and add the comparison with existing preconditioners in the related works in Appendix B, and the scalar ablation in Section 5.3.
[1] Titsias. Optimal Preconditioning and Fisher Adaptive Langevin Sampling. NeurIPS 2023.