- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes Soft Deterministic Policy Gradient (Soft-DPG) to address the instability of deterministic policy gradient methods caused by unreliable critic action-gradients, particularly in environments with discrete or non-smooth rewards. The authors introduce a Gaussian-smoothed Bellman equation and derive a policy gradient that relies on critic value evaluations rather than action-gradients. They further develop Soft DDPG and provide theoretical bounds on the approximation bias introduced by smoothing. Experiments on continuous control benchmarks show that the proposed method improves performance in most discretized-reward environments, while its performance on standard continuous-reward tasks is mixed.
Critical strengths
The paper is well-motivated and clearly identifies the limitations of relying on critic action-gradients in non-smooth reward environments. The proposed framework provides a systematic connection between Gaussian smoothing, Bellman equations, and policy gradient methods, supported by theoretical analysis. The experiments cover a variety of continuous control tasks and demonstrate promising improvements in most discretized-reward settings.
Critical weaknesses
W1. Limited theoretical novelty. The proposed smoothed Bellman operator is exactly the standard policy-evaluation operator for a fixed-variance Gaussian policy, and Theorem 5.7 follows directly from the classical stochastic policy gradient theorem, as acknowledged in Appendix B.8. The paper should more clearly establish its incremental contributions relative to existing Gaussian policy optimization and zeroth-order actor–critic methods.
W2. Inconsistency between the theoretical gradient and the proposed algorithm. Algorithm 1 clips Gaussian-perturbed actions but uses the clipped action differences in the Gaussian score estimator, which generally introduces bias and is not justified by Theorem 5.7. The pseudocode also does not specify the necessary stop-gradient operations. The authors should clarify the actual implementation and demonstrate that the actor update correctly estimates the intended gradient.
Questions for authors
Questions:
-
As shown in Appendix B.8, Theorem 5.7 can be derived directly from the standard stochastic policy gradient theorem. What is the main theoretical novelty of Soft-DPG beyond applying a fixed-variance Gaussian policy?
-
How does the proposed method differ from existing zeroth-order DPG methods and Gaussian policy optimization approaches? In particular, what additional benefit does the proposed Bellman-level smoothing provide?
-
In Algorithm 1, the Gaussian-perturbed actions are clipped to the action bounds. Since this changes the sampling distribution, does the resulting gradient estimator still satisfy Theorem 5.7? If not, how significant is the introduced bias in practice?
2 comments
Thank you for the detailed response. The clarification regarding action clipping partially addresses my concern about the consistency between theory and implementation.
However, my main concern regarding novelty remains. As shown in Appendix B.8, Theorem 5.7 follows directly from the standard stochastic policy gradient theorem, and the proposed smoothed Bellman operator corresponds to standard Gaussian policy evaluation. Closely related ideas have also been explored in prior work [1–3]. While the authors explain the differences from these methods, I am still not convinced that the proposed formulation provides sufficient additional theoretical novelty.
Therefore, I maintain my original rating of 2.
References
[1] Nachum, O., Norouzi, M., Tucker, G., & Schuurmans, D. (2018). Smoothed Action Value Functions for Learning Gaussian Policies. ICML 2018. https://proceedings.mlr.press/v80/nachum18a.html
[2] Kumar, H., Kalogerias, D. S., Pappas, G. J., & Ribeiro, A. (2020). Zeroth-order Deterministic Policy Gradient. arXiv:2006.07314. https://arxiv.org/abs/2006.07314
[3] Saglam, B., & Kalogerias, D. (2024). Compatible Gradient Approximations for Actor-Critic Algorithms. arXiv:2409.01477. https://arxiv.org/abs/2409.01477
Thank you. We address W1/Q1/Q2 and W2/Q3 below (revisions in blue in the PDF).
W1, Q1 & Q2. Role of Soft-DPG and comparison with Gaussian SPG and ZO-DPG:
Motivation & contribution (Q1): DPG relies directly on ∇aQ(s,a), which can be undefined at jumps or zero on flat plateaus. Instead, we update πθ using critic values at Gaussian-perturbed actions. For fixed σ>0, Tσπ is the evaluation operator of νθ,σ(⋅∣s)=N(πθ(s),σ2I), giving Theorem 5.7 (Appendix B.8). Our contribution is not this Gaussian score identity itself, but using it to build a Bellman-consistent, ∇aQ-free alternative while deploying πθ deterministically:
Comparison with Gaussian SPG & ZO-DPG (Q2):
W2 & Q3. Stop-gradient, action clipping vs. Theorem 5.7, and practical bias:
Stop-gradient (Algorithm 1): Section 5.3 and Algorithm 1 specify sg[⋅] on yi, ai=clip(sg[πθ(s)]+σwi,amin,amax), and sg[Qw(s,ai)], so differentiating 2σ21∥ai−πθ(s)∥22sg[Qw(s,ai)] differentiates only πθ(s).
Policy gradient under action clipping (Appendix E.1):
Effect on trained checkpoints (Appendix E.2, Table 6): Evaluating g^clip vs. g^ref across all checkpoints (512 states, 500 draws/state) shows: (1) In 13 of 14 settings (7/7 discrete, 6/7 continuous), clipping is rare (0.00%–0.34%; 1.65%–2.59% in Humanoid), giving high directional agreement (cosine similarity 0.938–1.000, mean 0.973) alongside relative norm differences of 0.004–0.436. (2) In Continuous HalfCheetah (40.05% clipping), 4/5 seeds experience tanh saturation where ∇θπθ(s)=0 makes both estimators vanish, while the unsaturated seed has 0.25% clipping (cosine similarity 0.981, relative difference 0.196); This saturation stems from single-critic in DDPG rather than the smoothed actor update itself, as
Soft-TD3 (Actor-only)avoids saturation and outperforms TD3 (4426.5 vs. 3293.5, Table 4). These measurements compare the two estimators at fixed checkpoints and do not show how clipping affects final returns.