acceptodds
Under review as a conference paper at ICLR 2027

Soft Deterministic Policy Gradient with Gaussian Smoothing

Abstract

Deterministic policy gradient (DPG) is widely utilized for continuous control; however, it inherently relies on the differentiability of the critic with respect to the action during policy updates. This assumption is violated in practical control problems involving sparse or discrete rewards, leading to ill-defined policy gradients and unstable learning. To address these challenges, we propose a principled alternative based on a smoothed Bellman equation formulated via Gaussian smoothing. Specifically, we define a novel action-value function based on a smoothed Bellman equation and derive the soft deterministic policy gradient (Soft-DPG). Our formulation eliminates explicit dependence on critic action-gradients and ensures that the gradient remains well-defined even for non-smooth Q-functions. We instantiate this framework into a deep reinforcement learning algorithm, which we call soft deep deterministic policy gradient (Soft DDPG). Empirical evaluations on standard continuous control benchmarks and their discretized-reward variants show that Soft DDPG remains competitive in dense-reward settings and provides clear gains in most discretized-reward environments, where standard DDPG is more sensitive to irregular critic landscapes.

Primary area:
reinforcement learning
Keywords:
Deterministic policy gradient, Gaussian smoothing, Actor-Critic, Continuous control, Reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_cajSOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Soft Deterministic Policy Gradient (Soft-DPG) to address the instability of deterministic policy gradient methods caused by unreliable critic action-gradients, particularly in environments with discrete or non-smooth rewards. The authors introduce a Gaussian-smoothed Bellman equation and derive a policy gradient that relies on critic value evaluations rather than action-gradients. They further develop Soft DDPG and provide theoretical bounds on the approximation bias introduced by smoothing. Experiments on continuous control benchmarks show that the proposed method improves performance in most discretized-reward environments, while its performance on standard continuous-reward tasks is mixed.

Critical strengths

The paper is well-motivated and clearly identifies the limitations of relying on critic action-gradients in non-smooth reward environments. The proposed framework provides a systematic connection between Gaussian smoothing, Bellman equations, and policy gradient methods, supported by theoretical analysis. The experiments cover a variety of continuous control tasks and demonstrate promising improvements in most discretized-reward settings.

Critical weaknesses

W1. Limited theoretical novelty. The proposed smoothed Bellman operator is exactly the standard policy-evaluation operator for a fixed-variance Gaussian policy, and Theorem 5.7 follows directly from the classical stochastic policy gradient theorem, as acknowledged in Appendix B.8. The paper should more clearly establish its incremental contributions relative to existing Gaussian policy optimization and zeroth-order actor–critic methods.

W2. Inconsistency between the theoretical gradient and the proposed algorithm. Algorithm 1 clips Gaussian-perturbed actions but uses the clipped action differences in the Gaussian score estimator, which generally introduces bias and is not justified by Theorem 5.7. The pseudocode also does not specify the necessary stop-gradient operations. The authors should clarify the actual implementation and demonstrate that the actor update correctly estimates the intended gradient.

Questions for authors

Questions:

  1. As shown in Appendix B.8, Theorem 5.7 can be derived directly from the standard stochastic policy gradient theorem. What is the main theoretical novelty of Soft-DPG beyond applying a fixed-variance Gaussian policy?

  2. How does the proposed method differ from existing zeroth-order DPG methods and Gaussian policy optimization approaches? In particular, what additional benefit does the proposed Bellman-level smoothing provide?

  3. In Algorithm 1, the Gaussian-perturbed actions are clipped to the action bounds. Since this changes the sampling distribution, does the resulting gradient estimator still satisfy Theorem 5.7? If not, how significant is the introduced bias in practice?

2 comments

Thank you. We address W1/Q1/Q2 and W2/Q3 below (revisions in blue in the PDF).

W1, Q1 & Q2. Role of Soft-DPG and comparison with Gaussian SPG and ZO-DPG:

  • Motivation & contribution (Q1): DPG relies directly on ∇aQ(s,a)\nabla _ a Q(s,a), which can be undefined at jumps or zero on flat plateaus. Instead, we update πθ\pi _ \theta using critic values at Gaussian-perturbed actions. For fixed σ>0\sigma \gt 0, TσπT _ \sigma^\pi is the evaluation operator of νθ,σ(⋅∣s)=N(πθ(s),σ2I)\nu _ {\theta,\sigma}(\cdot\mid s)=\mathcal{N}(\pi _ \theta(s),\sigma^2 I), giving Theorem 5.7 (Appendix B.8). Our contribution is not this Gaussian score identity itself, but using it to build a Bellman-consistent, ∇aQ\nabla _ a Q-free alternative while deploying πθ\pi _ \theta deterministically:

    1. Fixed σ>0\sigma \gt 0: Under bounded measurable rewards, TσπT _ \sigma^\pi is a contraction and Theorem 5.7 yields a well-defined gradient without ∇aQ\nabla _ a Q. When RR and PP are Lipschitz, Lemmas 5.3–5.5 and A.2 also give Lipschitz QσπQ _ \sigma^\pi and O(σ)\mathcal{O}(\sigma) bounds on value gaps and occupancy drift relative to πθ\pi _ \theta.
    2. Limit σ↓0\sigma\downarrow 0: Recovery of Vπ,Qπ,V^\pi, Q^\pi, and DPG under Assumption B.1 (Lemma 3.4, Appendix B.1).
  • Comparison with Gaussian SPG & ZO-DPG (Q2):

    1. Gaussian SPG: Soft DDPG optimizes Jσ(θ)J _ \sigma(\theta) off-policy using NN perturbations per state and deploys πθ(s)\pi _ \theta(s) deterministically.
    2. ZO-DPG: Kumar et al. (2020) use critic-free rollouts, while Saglam & Kalogerias (2024) approximate DPG via two-point critic evaluations (with TD3-style target smoothing). We instead analyze the Gaussian-perturbed policy's Bellman fixed point (QσπθQ _ \sigma^{\pi _ \theta}) and its relation to deterministic policy evaluation.
    3. Bellman smoothing (Appendix D, Table 4): Smoothing both critic and actor improves DDPG on discrete rewards. In TD3, also smoothing the critic update ('Soft-TD3') often degraded performance, whereas 'Soft-TD3 (Actor-only)' beat TD3 on most discrete-reward tasks.

W2 & Q3. Stop-gradient, action clipping vs. Theorem 5.7, and practical bias:

  • Stop-gradient (Algorithm 1): Section 5.3 and Algorithm 1 specify sg[⋅]\mathrm{sg}[\cdot] on yiy _ i, ai=clip(sg[πθ(s)]+σwi,amin⁡,amax⁡)a _ i = \mathrm{clip}(\mathrm{sg}[\pi _ \theta(s)] + \sigma w _ i, a _ {\min}, a _ {\max}), and sg[Qw(s,ai)]\mathrm{sg}[Q _ w(s,a _ i)], so differentiating 12σ2∥ai−πθ(s)∥22sg[Qw(s,ai)]\frac{1}{2\sigma^2}\lVert a _ i - \pi _ \theta(s) \rVert _ 2^2 \mathrm{sg}[Q _ w(s,a _ i)] differentiates only πθ(s)\pi _ \theta(s).

  • Policy gradient under action clipping (Appendix E.1):

    1. Clipping the executed action: Sampling z=πθ(s)+σwz = \pi _ \theta(s) + \sigma w and clipping a=c(z):=clip(z,amin⁡,amax⁡)a = c(z) := \mathrm{clip}(z, a _ {\min}, a _ {\max}) defines a bounded-action policy νˉθ,σ\bar{\nu} _ {\theta,\sigma} whose Bellman operator Tσ,cπT _ {\sigma,c}^\pi (Line 4) has unique fixed point Qσ,cπθQ _ {\sigma,c}^{\pi _ \theta}. The exact gradient of Jσ,c(θ)J _ {\sigma,c}(\theta) uses Qσ,cπθQ _ {\sigma,c}^{\pi _ \theta} and ρνˉθ,σ\rho^{\bar{\nu} _ {\theta,\sigma}} with zz in the score; replacing Qσ,cπθQ _ {\sigma,c}^{\pi _ \theta} with QwQ _ w yields the reference estimator g^ref(s)=1σ2∇θπθ(s)(z−πθ(s))Qw(s,c(z))\hat{g} _ {\text{ref}}(s) = \frac{1}{\sigma^2}\nabla _ \theta \pi _ \theta(s)(z - \pi _ \theta(s))Q _ w(s, c(z)).
    2. Clipping in the actor update: Line 8 also uses c(z)c(z) in ∥a−πθ(s)∥22\lVert a - \pi _ \theta(s)\rVert _ 2^2, giving g^clip(s)=1σ2∇θπθ(s)(c(z)−πθ(s))Qw(s,c(z))\hat{g} _ {\text{clip}}(s) = \frac{1}{\sigma^2}\nabla _ \theta \pi _ \theta(s)(c(z) - \pi _ \theta(s))Q _ w(s, c(z)). This biased approximation to Theorem 5.7 acts (for πθ(s)∈[amin⁡,amax⁡]\pi _ \theta(s) \in [a _ {\min}, a _ {\max}]) as a magnitude-limited surrogate restricting weights on out-of-bound samples, with expected boundary bias Ew[g^clip(s)−g^ref(s)]=1σ2∇θπθ(s)Ez[(c(z)−z)Qw(s,c(z))]\mathbb{E} _ w[\hat{g} _ {\text{clip}}(s) - \hat{g} _ {\text{ref}}(s)] = \frac{1}{\sigma^2}\nabla _ \theta \pi _ \theta(s)\mathbb{E} _ z[(c(z) - z)Q _ w(s, c(z))], which vanishes on interior samples (c(z)=zc(z)=z) and arises only when samples cross action bound.
  • Effect on trained checkpoints (Appendix E.2, Table 6): Evaluating g^clip\hat{g} _ {\text{clip}} vs. g^ref\hat{g} _ {\text{ref}} across all checkpoints (512512 states, 500500 draws/state) shows: (1) In 13 of 14 settings (7/7 discrete, 6/7 continuous), clipping is rare (0.00%0.00\%–0.34%0.34\%; 1.65%1.65\%–2.59%2.59\% in Humanoid), giving high directional agreement (cosine similarity 0.9380.938–1.0001.000, mean 0.9730.973) alongside relative norm differences of 0.0040.004–0.4360.436. (2) In Continuous HalfCheetah (40.05%40.05\% clipping), 4/5 seeds experience tanh⁡\tanh saturation where ∇θπθ(s)=0\nabla _ \theta \pi _ \theta(s) = 0 makes both estimators vanish, while the unsaturated seed has 0.25%0.25\% clipping (cosine similarity 0.9810.981, relative difference 0.1960.196); This saturation stems from single-critic in DDPG rather than the smoothed actor update itself, as Soft-TD3 (Actor-only) avoids saturation and outperforms TD3 (4426.54426.5 vs. 3293.53293.5, Table 4). These measurements compare the two estimators at fixed checkpoints and do not show how clipping affects final returns.

Thank you for the detailed response. The clarification regarding action clipping partially addresses my concern about the consistency between theory and implementation.

However, my main concern regarding novelty remains. As shown in Appendix B.8, Theorem 5.7 follows directly from the standard stochastic policy gradient theorem, and the proposed smoothed Bellman operator corresponds to standard Gaussian policy evaluation. Closely related ideas have also been explored in prior work [1–3]. While the authors explain the differences from these methods, I am still not convinced that the proposed formulation provides sufficient additional theoretical novelty.

Therefore, I maintain my original rating of 2.

References

[1] Nachum, O., Norouzi, M., Tucker, G., & Schuurmans, D. (2018). Smoothed Action Value Functions for Learning Gaussian Policies. ICML 2018. https://proceedings.mlr.press/v80/nachum18a.html

[2] Kumar, H., Kalogerias, D. S., Pappas, G. J., & Ribeiro, A. (2020). Zeroth-order Deterministic Policy Gradient. arXiv:2006.07314. https://arxiv.org/abs/2006.07314

[3] Saglam, B., & Kalogerias, D. (2024). Compatible Gradient Approximations for Actor-Critic Algorithms. arXiv:2409.01477. https://arxiv.org/abs/2409.01477

Reviewer_ZyYoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the dependence of deterministic policy gradient methods on the critic’s action gradients. It proposes a policy update based on Gaussian smoothing and critic function evaluations. The authors introduce Gaussian action perturbations into the Bellman update, derive Soft-DPG, establish value approximation error bounds, and instantiate the approach as Soft-DDPG. Experiments cover standard continuous-control benchmarks and their discrete-reward variants, with additional TD3 extensions presented in the appendix. The results show promising improvements on some discrete-reward tasks, alongside substantial degradation on several continuous-reward tasks and in some TD3 experiments.

Critical strengths
  1. The paper addresses a practically relevant problem. Accurate critic value predictions do not necessarily imply reliable action gradients for policy improvement. Exploring policy updates based on function evaluations is therefore well motivated and potentially useful.
  2. Some discrete-reward results are promising. Table 1 reports favorable results for Soft-DDPG on discrete-reward Ant, Walker2d, and InvertedDoublePendulum. The TD3 extensions in the appendix also provide useful evidence for assessing the approach beyond the DDPG framework.
Critical weaknesses
  1. The substantive contribution and the scope of the theoretical guarantees are not sufficiently clear. Equation (11) in Section 5.1 can be interpreted as the Bellman expectation equation of a fixed-variance Gaussian policy. Accordingly, Theorem 5.7 in Section 5.2 directly corresponds to the standard stochastic policy gradient for that policy, as acknowledged in Appendix B.8. The authors should distinguish results that reformulate existing theory from genuinely new contributions. Section 2 should also explain more precisely what new properties or advantages the proposed formulation offers over smoothed value functions, TD3 target policy smoothing, and existing zeroth-order methods. Furthermore, Lemmas 5.3–5.5 assume that the reward is Lipschitz continuous with respect to the action, whereas the discontinuous rewards emphasized in Section 6 and Appendix B.10 generally do not satisfy this condition. Equation (9) also retains the unsmoothed immediate reward, so reward-induced discontinuities may remain in the resulting action-value function. The authors should clearly delimit the applicability of the guarantees and distinguish a well-defined policy update from a smooth action-value function. These issues directly affect the assessment of the theoretical contribution and its practical significance.
  2. The correspondence between theory and implementation, together with the empirical support, remains insufficient. Theorem 5.7 is derived for unclipped Gaussian actions, whereas Steps 7–8 of Algorithm 1 construct the actor update using clipped actions. Clipping changes the action distribution, so directly applying the original Gaussian gradient formula may introduce additional bias. This requires an appropriate derivation or correction. Section 5.3 and Algorithm 1 also do not specify whether gradients are stopped through the sampled actions and the critic outputs used as weights. The authors should provide the relevant implementation details and verify that the actual update matches the stated estimator. Regarding empirical validation, Section 5.3 modifies both the critic target and the actor update, but the main experiments in Section 6 do not adequately isolate their respective contributions. Although Appendix D includes an actor-only TD3 variant, a complete DDPG ablation comparing actor-only, critic-only, and joint modifications is still needed. Since Appendices B.9 and B.10 use 50 perturbation samples per update, training time or comparisons under matched computational budgets should also be reported. Finally, Tables 1 and 3 show substantial degradation on some continuous-reward tasks. In Table 4, the full Soft-TD3 variant has lower mean performance than standard TD3 on five of the seven discrete-reward tasks. The authors should explain these failures and the differences between the actor-only and full variants, and appropriately qualify the performance and applicability claims in Sections 6 and 7.
Questions for authors
  1. Given Equation (11), Theorem 5.7, and Appendix B.8, what is the substantive theoretical contribution beyond the standard fixed-variance Gaussian policy gradient? What specific new properties arise from the different placement of smoothing emphasized in Section 2?
  2. How do the continuity assumptions in Lemmas 5.3–5.5 apply to the discontinuous rewards described in Appendix B.10? Which guarantees remain valid when these assumptions are violated? Does Equation (9) necessarily produce a smooth action-value function?
  3. How do Steps 7–8 of Algorithm 1 account for the distributional change introduced by action clipping? Are gradients stopped through the sampled actions and the critic outputs used as weights? Please provide the corresponding derivation and implementation details.
  4. Within the DDPG framework, how do actor-only, critic-only, and joint modifications compare? What is the computational overhead of using 50 perturbation samples per update, and how does performance compare under matched computational budgets?
  5. How do the authors explain the substantial degradation on some continuous-reward tasks in Tables 1 and 3, and the weaker performance of the full Soft-TD3 variant on most discrete-reward tasks in Table 4? What do these results imply about the method’s scope of applicability?
  6. The proposed Soft-DPG framework is evaluated only against deterministic policy gradient methods, such as DDPG and TD3. How does its performance compare with representative stochastic off-policy algorithms, such as SAC and its distributional variants (e.g., DSAC), particularly in environments with discontinuous rewards? Additional comparisons would help clarify whether the proposed Gaussian smoothing provides advantages beyond conventional stochastic policy optimization.
2 comments

Thank you. We address Q1–Q6 below (revisions in blue in the PDF).

Q1 (W1). Contribution and related work:

  • Contribution: DPG relies on ∇aQ(s,a)\nabla _ a Q(s,a), failing at jumps or plateaus. For σ>0\sigma \gt 0, TσπT _ \sigma^\pi evaluates νθ,σ(⋅∣s)=N(πθ(s),σ2I)\nu _ {\theta,\sigma}(\cdot\mid s)=\mathcal{N}(\pi _ \theta(s),\sigma^2 I) with score gradient in Theorem 5.7 (Appendix B.8). We do not claim this Gaussian identity is new; we use it as a Bellman-consistent ∇aQ\nabla _ a Q-free surrogate while deploying πθ\pi _ \theta deterministically.
  • Section 2: Smoothing only Vσπθ(s′)V _ \sigma^{\pi _ \theta}(s') in Eq. (8) leaves jumps in R(s,a)R(s,a) intact (Section 4), so ∇aQw\nabla _ a Q _ w actors (including TD3 target smoothing) can lack signal. Unlike critic-free rollouts (Kumar et al., 2020) or two-point critic differences with TD3 target smoothing (Saglam & Kalogerias, 2024), we analyze QσπθQ _ \sigma^{\pi _ \theta} vs. QπQ^\pi.

Q2 (W1). Guarantee scope (Section 5.1): Lemmas 5.3–5.5 and A.2 assume Lipschitz R,PR,P and do not cover discontinuous rewards: (1) For bounded measurable RR and γ<1\gamma \lt 1, contraction and fixed point QσπθQ _ \sigma^{\pi _ \theta} hold (Lemma 5.2), and Theorem 5.7 (σ>0\sigma \gt 0) holds under policy regularity/integrability without ∇aQσπθ\nabla _ a Q _ \sigma^{\pi _ \theta}. (2) Eq. (9) need not smooth Qσπ(s,a)Q _ \sigma^\pi(s,a) since R(s,a)R(s,a) is unsmoothed, though Jσ(θ)J _ \sigma(\theta) is smooth in θ\theta. (3) Lemmas 5.3–5.5 and A.2 give Lipschitz QσπQ _ \sigma^\pi and O(σ)\mathcal{O}(\sigma) bounds for Lipschitz R,PR, P (recovering DPG as σ↓0\sigma \downarrow 0; Lemma 3.4).

Q3 (W2). Clipping & stop-gradient (Alg. 1, App. E):

  • Stop-gradient: Alg. 1 applies sg[⋅]\mathrm{sg}[\cdot] to yiy _ i, ai=c(sg[πθ(s)]+σwi)a _ i = c(\mathrm{sg}[\pi _ \theta(s)] + \sigma w _ i), and sg[Qw(s,ai)]\mathrm{sg}[Q _ w(s,a _ i)] (c=clipc=\mathrm{clip} on [amin⁡,amax⁡][a _ {\min},a _ {\max}]), differentiating only πθ(s)\pi _ \theta(s) in Line 8.
  • Clipping: For z=πθ(s)+σwz = \pi _ \theta(s) + \sigma w, a=c(z)a = c(z) defines policy νˉθ,σ\bar{\nu} _ {\theta,\sigma} with fixed point Qσ,cπθQ _ {\sigma,c}^{\pi _ \theta}. Exact ∇θJσ,c\nabla _ \theta J _ {\sigma,c} uses Qσ,cπθ,ρνˉθ,σ,Q _ {\sigma,c}^{\pi _ \theta}, \rho^{\bar{\nu} _ {\theta,\sigma}}, and score z−πθz - \pi _ \theta (giving g^ref\hat{g} _ {\text{ref}} with QwQ _ w), whereas Line 8 uses c(z)−πθc(z) - \pi _ \theta (g^clip\hat{g} _ {\text{clip}}), adding bias 1σ2∇θπθ(s)Ez[(c(z)−z)Qw(s,c(z))]\frac{1}{\sigma^2}\nabla _ \theta \pi _ \theta(s)\mathbb{E} _ z[(c(z) - z)Q _ w(s,c(z))] (00 when c(z)=zc(z)=z). Across 13/14 tasks (Table 6), clipping is ≤0.34%\le 0.34\% (1.65%1.65\%–2.59%2.59\% Humanoid) with cosine sim. 0.9380.938–1.0001.000 (mean 0.9730.973, rel. diff. 0.0040.004–0.4360.436); in Cont. HalfCheetah, 4/5 seeds saturate (∇θπθ=0\nabla _ \theta \pi _ \theta=0) and 1/5 has 0.25%0.25\% clipping (cos. sim. 0.9810.981). This isolates checkpoint score-clipping bias without claiming return effects.

Q4 (W2). Ablations & overhead (Appendix D & F):

  • Ablations: Section 4 shows critic-only smoothing leaves jumps in Qσπ(s,a)Q _ \sigma^\pi(s,a) intact, and TD3 ablations across all 7 tasks (Soft-TD3 (Actor-only) vs. Soft-TD3, Appendix D, Table 4) show actor smoothing drives discrete gains. Five-seed Actor-only and Critic-only ablations for single-critic DDPG on representative Discrete Ant are finishing now and will be added before the discussion deadline.
  • Overhead & budgets (Appendix F, Table 7): Batched N=50N=50 GPU passes without ∇aQw\nabla _ a Q _ w backprop add +16%–26% 1M-step runtime for Soft-DDPG vs. DDPG on 1–8D (9999–113113 vs. 8282–8989m; +72%+72\% Humanoid) and +7%–9% for Soft-TD3 (Actor-only) vs. TD3 (6969–8989 vs. 6464–8383m), beating SAC (120120–145145m). At DDPG/TD3 1M-step runtimes, Soft-DDPG (788k–863k steps, 1–8D) beats DDPG on 6/7 discrete tasks (HalfCheetah: 1660.81660.8 vs. 1599.41599.4), and Soft-TD3 (Actor-only) (925k–933k steps) beats TD3 on 5/7 discrete tasks.

Q5 (W2). Continuous tasks & Soft-TD3 (Secs. 6–7): (1) On smooth rewards where ∇aQw\nabla _ a Q _ w is reliable, N=50N=50 sampling can hurt via smoothing bias, high-mm variance (Humanoid, m=17m=17), or tanh⁡\tanh saturation (Cont. HalfCheetah); Soft-DPG targets discontinuous/flat rewards. (2) Since TD3 already smooths critic targets, extra critic smoothing (Soft-TD3) over-smooths critics, whereas applying our ∇aQ\nabla _ a Q-free update only to the actor (Soft-TD3 (Actor-only)) beats TD3 on 5/7 discrete tasks.

Q6. SAC comparison (Appendix F, Table 8): Evaluating SAC on all 14 tasks shows two regimes: where fixed perturbations suffice (Disc. Ant: Soft-DDPG 271.9271.9 vs. SAC 218.4218.4; Disc. Inv. Pend.: 10383.110383.1 vs. 10375.110375.1; Cont. Inv. Double Pend.: 9343.29343.2 vs. 8803.58803.5), Soft-DPG matches/beats SAC with lower variance and runtime. On multi-joint locomotion, SAC beats fixed-noise baselines via learned variance and adaptive entropy regularization (while still using critic action-gradients); Soft-DPG instead removes critic action-gradients within deterministic policies.

Follow-up on Q4 (W2): Completed 5-seed DDPG component ablation (Appendix G, Algorithms 4–5, Table 9)

We have completed the 5-seed DDPG component ablation on Discrete Ant and updated the revised PDF (Appendix G, Table 9):

MethodCritic SmoothingActor SmoothingReturn (Mean ±\pm Std)
DDPGNoNo190.56±60.23190.56 \pm 60.23
Soft-DDPG (Critic-only)YesNo201.26±27.01201.26 \pm 27.01
Soft-DDPG (Actor-only)NoYes201.77±116.84201.77 \pm 116.84
Soft-DDPG (Full)YesYes271.93±5.54271.93 \pm 5.54

Critic-only reduces variance but remains limited by critic action-gradients, whereas Actor-only removes action-gradients but suffers from high variance without critic smoothing. Combining both (Soft-DDPG (Full)) achieves the highest return and lowest variance, while TD3's built-in target smoothing makes Soft-TD3 (Actor-only) sufficient.

Reviewer_45UPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Maximizing the policy objective of deterministic policy gradient (DPG) relies on calculating the gradient of critic with respect to the action. The paper argues that the gradient of a learned Q function with respect to the action can be unstable, especially in discrete-reward environments. The paper proposes soft DDPG based on zeroth-order approximation of the gradient of the Q function, which eliminates the need for explicitly computing the gradient of the Q function w.r.t. the action in the policy objective.

Critical strengths

I appreciate the presented theoretical justifications for adding the smoothing operator to the Bellman equation and deriving soft DPG.

Critical weaknesses

The motivation for the paper and its contributions can be significantly improved. The main section motivating the need for soft DPG is section 4. However, section 4 does not adequately explain exactly why DPG fails, in what scenarios soft DPG can be helpful and why. In my opinion, the problem motivation section should be expanded significantly and explain the nuances of the problem thoroughly and in detail. What does “smoothness” of the action gradient mean? How should Figures 1a and 1b be interpreted? How were the lines in Figure 1a plotted? Why does DDPG not have a spike at a=0.5 in Figure 1a? What’s the difference between DDPG and soft DDPG in Figure 1b? Is soft DDPG gradient smooth or non-smooth? Accurate or inaccurate? If it is accurate and smooth, can it be used in the policy objective without changing the policy objective to the soft version in Algorithm 1?


  • “This assumption is violated in practical control problems involving sparse or discrete rewards, leading to ill-defined policy gradients and unstable learning” - Why would the policy gradient be ill-defined when the reward is sparse or discrete?
  • “sparse or discrete reward structures are commonly used in robotics and autonomous driving (Kiran et al., 2021; Vecerik et al., 2017), leading to non-smooth or even non-differentiable Q-functions” - What does a non-smooth Q function mean? Why would the Q-function become non-differentiable if the reward is changed from dense to sparse?
  • “standard DDPG suffers from degraded performance due to the non-differentiable nature of the discrete rewards” - What does “non-differentiable nature of the discrete rewards” mean?
Questions for authors

No questions.

1 comment

Thank you for your constructive feedback. We revised Section 4, Figure 1, and Section 6 (revisions in blue in the PDF).

W1. Clarifying the quoted statements and sparse/discrete rewards:

  • "Ill-defined policy gradients" & "non-smooth/non-differentiable QQ-functions": These refer to regularity in continuous action aa. At jump discontinuities of Qπ(s,a)Q^\pi(s,a) in aa, ∇aQπ(s,a)\nabla _ a Q^\pi(s,a) does not exist, making ∇θπθ(s)∇aQπ(s,a)∣a=πθ(s)\nabla _ \theta \pi _ \theta(s)\nabla _ a Q^\pi(s,a)\rvert _ {a=\pi _ \theta(s)} undefined. On flat plateaus, ∇aQπ(s,a)=0\nabla _ a Q^\pi(s,a) = 0 gives no directional signal.
  • Dense vs. sparse/discrete rewards: Sec. 4 analyzes a single-state MDP (a∈[−1,1]a \in [-1,1], P(s∣s,a)=1P(s\mid s,a)=1, π(s)=0.5\pi(s)=0.5, γ=0.99\gamma=0.99) comparing Rdense(s,a)=1−(a−0.5)2/1.52R _ {\text{dense}}(s,a) = 1 - (a-0.5)^2 / 1.5^2 with Rsparse(s,a)=I(∣a−0.5∣<ϵ)R _ {\text{sparse}}(s,a) = \mathbb{I}(\lvert a-0.5 \rvert \lt \epsilon) (ϵ=0.005\epsilon=0.005). Since Qπ(s,a)=R(s,a)+γ1−γR(s,π(s))Q^\pi(s,a) = R(s,a) + \frac{\gamma}{1-\gamma} R(s, \pi(s)) has an aa-independent second term, Qsparseπ(s,a)Q _ {\text{sparse}}^\pi(s,a) is a step function in aa: ∂aQsparseπ(s,a)\partial _ a Q _ {\text{sparse}}^\pi(s,a) is undefined at a=0.5±ϵa = 0.5 \pm \epsilon and 00 elsewhere. Because TD learning minimizes value error without constraining ∂aQw(s,a)\partial _ a Q _ w(s,a), the derivative of a smooth critic Qw(s,a)Q _ w(s,a) on plateaus or near jumps reflects approximation error rather than an improvement direction.
  • "Non-differentiable nature of the discrete rewards" (Sec. 6): This wording was imprecise; it is Qπ(s,a)Q^\pi(s,a) that can be non-differentiable in aa at jumps, so we revised Sec. 6.

W2. Revision of Section 4 and Figures 1a–1b: Originally, Fig. 1a plotted Qw(s,a)Q _ w(s,a) and Fig. 1b plotted ∣∂aQw(s,a)∣\lvert \partial _ a Q _ w(s,a) \rvert on a∈[−1,1]a \in [-1,1] from trained DDPG and Soft-DDPG critics. The DDPG checkpoint missed the narrow rewarding interval, but that plot alone could not separate insufficient exploration from critic error; also, plotting ∣∂aQw∣\lvert \partial _ a Q _ w \rvert instead of the signed derivative hid gradient direction, and the Soft-DDPG curve in Fig. 1b plotted the critic derivative rather than our value-based actor estimator.

To isolate the analytical issue, revised Fig. 1 plots exact Qπ(s,a)Q^\pi(s,a) and signed ∂aQπ(s,a)\partial _ a Q^\pi(s,a) at π(s)=0.5\pi(s)=0.5. In Fig. 1a, Qπ(s,a)Q^\pi(s,a) is a parabola under RdenseR _ {\text{dense}} and a step function with unit jumps at a=0.5±ϵa = 0.5 \pm \epsilon under RsparseR _ {\text{sparse}}. In Fig. 1b, ∂aQdenseπ(s,a)=−2(a−0.5)1.52\partial _ a Q _ {\text{dense}}^\pi(s,a) = -\frac{2(a-0.5)}{1.5^2} points toward a=0.5a=0.5, whereas ∂aQsparseπ(s,a)\partial _ a Q _ {\text{sparse}}^\pi(s,a) is 00 off the boundaries and undefined at a=0.5±ϵa = 0.5 \pm \epsilon (open circles).

W3. Soft-DDPG gradient properties and why the smoothed objective is needed: In Sec. 5, Jσ(θ)=Es0∼p0[Vσπθ(s0)]J _ \sigma(\theta) = \mathbb{E} _ {s _ 0 \sim p _ 0}[V _ \sigma^{\pi _ \theta}(s _ 0)] with Vσπθ(s)=Ew∼N(0,I)[Qσπθ(s,πθ(s)+σw)]V _ \sigma^{\pi _ \theta}(s) = \mathbb{E} _ {w \sim \mathcal{N}(0,I)}[Q _ \sigma^{\pi _ \theta}(s, \pi _ \theta(s) + \sigma w)]. By Thm. 5.7, ∇θJσ(θ)\nabla _ \theta J _ \sigma(\theta) is weighted by ρνθ,σ\rho^{\nu _ {\theta,\sigma}} and uses critic values Qσπθ(s,πθ(s)+σw)Q _ \sigma^{\pi _ \theta}(s, \pi _ \theta(s) + \sigma w) rather than ∇aQσπθ\nabla _ a Q _ \sigma^{\pi _ \theta}. For any bounded critic, this Gaussian expectation is smooth in πθ(s)\pi _ \theta(s) even when the critic has jumps in aa. On accuracy, Thm. 5.7 is exact for Jσ(θ)J _ \sigma(\theta) (with controlled O(σ)\mathcal{O}(\sigma) bias vs. J(θ)J(\theta) under Lipschitz conditions; Lemmas 3.4, 5.5), and Alg. 1 approximates it via QwQ _ w, NN samples, and clipping. Moreover, we cannot simply plug this zeroth-order update into unsmoothed J(θ)J(\theta) without the σ\sigma-smoothed Bellman backup: Differentiating VσπθV _ \sigma^{\pi _ \theta} unrolls through Eq. (9), so Bellman consistency requires training the critic toward QσπθQ _ \sigma^{\pi _ \theta} rather than QπθQ^{\pi _ \theta}.

Conversely, smoothing only the critic target while keeping the DPG actor update ∇θπθ(s)∇aQw(s,a)∣a=πθ(s)\nabla _ \theta \pi _ \theta(s)\nabla _ a Q _ w(s,a)\rvert _ {a=\pi _ \theta(s)} does not yield a useful action-gradient. In Qσπθ(s,a)=R(s,a)+γEs′∣s,a[Vσπθ(s′)]Q _ \sigma^{\pi _ \theta}(s,a) = R(s,a) + \gamma \mathbb{E} _ {s'\mid s,a}[V _ \sigma^{\pi _ \theta}(s')], smoothing enters only via Vσπθ(s′)V _ \sigma^{\pi _ \theta}(s'). In Sec. 4, s′s' is aa-independent, so Vσπθ(s′)V _ \sigma^{\pi _ \theta}(s') adds an aa-independent constant, leaving jumps at a=0.5±ϵa = 0.5 \pm \epsilon and zero derivative elsewhere. Thus, ∇aQw(s,a)∣a=πθ(s)\nabla _ a Q _ w(s,a)\rvert _ {a=\pi _ \theta(s)} can still lack directional signal without Alg. 1's actor update.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.