acceptodds
Under review as a conference paper at ICLR 2027

An analysis of Mirror-Descent Soft Actor-Critic

Abstract

Soft Actor-Critic (SAC) is widely used for entropy-regularised reinforcement learning with continuous action spaces, and practical implementations perform only a few actor steps towards an evolving target. In this work, we prove convergence guarantees when the target policy arises from policy mirror descent and compare it with the classical Gibbs target. We derive sufficient conditions for the strong convexity and smoothness of the actor objective, characterised by the curvature of the -function estimate, and establish an best-iterate finite-time convergence rate up to actor and critic approximation errors. Moreover, the mirror-descent step size directly controls the target drift and hence actor tracking error, whereas the analogous Gibbs bound contains a non-vanishing tracking term.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Markov Decision Process, Entropy Regularization, Non-convex optimization, Mirror descent, Global convergence, Function approximation, Actor-Critic, Soft Actor-Critic, Continuous Action Spaces, RL Theory

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_G2j8OpenReview ↗
Rating
3(+2)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper analyzes a population-level formalization of Soft Actor-Critic (SAC) that matches common implementations: a fixed-covariance tanh⁡\tanh-squashed Gaussian actor with linear mean, one projected reverse-KL actor step per outer iteration, and a critic oracle with uniform error ϵcrit\epsilon_{\mathrm{crit}}. It compares the classical Gibbs target in Eq. (4) with a policy mirror-descent (MD) target with proximal step size λ\lambda in Eq. (5), where the MD target is updated from the previous target rather than from the current actor. Under optimal-occupancy concentrability and a Legendre-operator curvature condition on the critic, RQ2<τ\frac{R_Q}{2}<\tau, the reverse-KL actor objective is shown to be strongly convex and smooth on a parameter ball. The analysis separates target drift from actor tracking error: the MD target drift scales with λ\lambda, while Theorem 1 also contains 1λ\frac{1}{\lambda} improvement terms; Corollary 1 shows that the Gibbs tracking bound contains a non-vanishing 1τ\frac{1}{\tau} term. Theorem 3 gives a best-iterate regularized value gap of order O(N−15)O\left(N^{-\frac{1}{5}}\right) plus approximation error under a prescribed λ\lambda scaling and sufficiently small actor step size hh. The experiments in Appendix K are restricted to a small tabular MDP with exact or perturbed critics, illustrating curvature effects and MD-versus-Gibbs tracking; there is no deep-RL or empirical rate validation.

Critical strengths
  1. Many MD/convergence papers update in policy space or assume exact convex projections; this paper's actor model matches tanh-squashed Gaussian SAC with a single reverse-KL step, which is how SAC is often run in code.
  2. Corollary 1's persistent 1τ\frac{1}{\tau} term vs λ\lambda-controlled MD drift gives a precise sense in which a fresh softmax target can be harder to track under partial actor updates; Figure 2 illustrates the phenomenon under exact critics.
  3. Theorem 3's O(N−15)O\left(N^{-\frac{1}{5}}\right) best-iterate gap with λ=Θ(N−45)\lambda=\Theta\left(N^{-\frac{4}{5}}\right) is a concrete timescale prescription, rare in SAC analyses that include evolving targets.
  4. Theorem 2 explains when the actor KL subproblem is strongly convex/smooth via RQ2<τ\frac{R_Q}{2}<\tau; Appendix K shows matched ϵcrit\epsilon_{\mathrm{crit}} but different curvature can yield qualitatively different learning (Figure 1).
  5. The authors acknowledge population-level abstraction and that experiments are mainly qualitative support for theory rather than benchmark validation.
Critical weaknesses
  1. A uniform ϵcrit\epsilon_{\mathrm{crit}} does not model TD bootstrapping, target networks, double Q-learning, nonstationary replay, or stochastic gradients—so the main theorems are not guarantees for standard training loops.
  2. The condition RQ2<τ\frac{R_Q}{2}<\tau is strong and unverifiable in practice. Constructive discussion is largely for Fourier linear critics (Remark 1). There is no proposal for measuring or enforcing this during neural critic learning.
  3. Many implementations take multiple actor updates per critic step; the paper does not analyze how K>1K>1 steps interact with Theorem 3 or tracking bounds.
  4. Eq. (5) defines πn\pi^n from the previous target, not necessarily from {wn}\{w^n\} or slow Q-target networks alone; the incremental novelty over “proximal targets move slowly” should be sharpened given the built-in 1λ\frac{1}{\lambda} KL term.
  5. Even accepting the theory, readers cannot see log-log validation of the exponent in Appendix K; constants (e.g., the strong convexity constant) may be tiny in reported regimes.
Questions for authors
  1. Can you provide checkable sufficient conditions (architectural constraints, regularization, Lipschitz/Legendre bounds, or online estimators) ensuring RQ2<τ\frac{R_Q}{2}<\tau for non-Fourier critics with high probability during training?
  2. How do Corollary 1 and Theorem 3 change if the actor performs K>1K>1 projected gradient steps per critic update? Is there a recommended scaling of KK with NN analogous to N45N^{\frac{4}{5}}?
  3. What is a plausible roadmap to replace the oracle critic with TD sample complexity—even in a linear function approximation setting with replay?
  4. If the Gibbs target is slowed (EMA on QQ, Q target network) to mimic small effective drift of MD at small λ\lambda, does Figure 2's MD advantage persist? Please quantify.
  5. Can you add log-log plots of best-iterate value gap vs NN on Appendix K and report estimated slopes vs the theoretical −15-\frac{1}{5} exponent?
  6. How sensitive is Theorem 3 to the projection radius RR? When minimizers lie on the boundary of B(R)\mathfrak{B}(R), do rates degrade in practice?
Reviewer_RC8zOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper provides a theoretical analysis of Soft Actor-Critic (SAC) with a parameterized actor that performs a single gradient step toward an evolving target policy. The authors consider a fixed-covariance squashed Gaussian policy class and investigate two policy-improvement targets: the classical Gibbs target and a target induced by policy mirror descent. The main technical contribution is a characterization of the curvature of the reverse-KL actor objective. Using the Legendre differential operator, the authors establish sufficient conditions under which this objective is strongly convex and smooth, despite its general non-convexity. This allows them to control the error incurred by the actor when tracking a sequence of evolving target policies. A particularly interesting result is that the mirror-descent step size directly controls target drift, leading to a tracking-error bound that can vanish as this parameter decreases. In contrast, the corresponding bound for the classical Gibbs target contains a non-vanishing contribution. Combining these results, the authors establish an O(N−1/5)\mathcal{O}(N^{-1/5}) best-iterate convergence rate, up to critic and actor approximation errors.

Critical strengths
  • Despite the widespread practical success of SAC, understanding the convergence of its parameterized actor updates remains challenging. Existing policy-space analyses typically assume that policy-improvement subproblems can be solved exactly, whereas practical implementations perform only a few gradient steps. This paper addresses precisely this discrepancy between the theoretical idealization and practical algorithms. I find this research direction particularly relevant and timely.

  • The characterization of the reverse-KL actor objective through the Legendre differential operator is, in my opinion, the most interesting contribution of the paper. The authors exploit the structure of squashed Gaussian policies to obtain explicit conditions guaranteeing strong convexity and smoothness of an objective that is generally non-convex. Theorem 2 is particularly appealing because it connects the curvature of the critic, the entropy regularization parameter, and the geometry of the actor parameterization. Rather than simply assuming strong convexity of the actor objective, the authors provide verifiable sufficient conditions under which it holds. This substantially strengthens the theoretical justification of the subsequent convergence analysis.

  • The comparison between the two target constructions is especially compelling. The authors demonstrate that the mirror-descent formulation introduces an explicit mechanism for controlling target drift through the parameter λ\lambda. This provides a theoretical explanation for why proximal policy updates may be easier to track using a limited number of gradient steps. The distinction between Lemma 2 and Corollary 1 is particularly informative: the mirror-descent tracking bound can be made arbitrarily small through the choice of λ\lambda, whereas the Gibbs-target bound contains a term that does not vanish in the same limit. I appreciate that the paper identifies a concrete advantage of mirror descent beyond its usual interpretation as a stabilizing regularization mechanism.

  • Theorem 3 establishes an explicit O(N−1/5)\mathcal{O}(N^{-1/5}) best-iterate convergence rate, up to approximation errors. Obtaining such a guarantee while allowing the actor to perform only one gradient step per iteration is a substantial contribution. I particularly appreciate the decomposition of the convergence bound into policy-improvement, actor-tracking, and approximation errors. This decomposition makes the different sources of error transparent and highlights the trade-off between aggressive policy improvement and the actor's ability to track its target.

  • The paper does not simply assume a Gaussian policy over an unconstrained action space, but explicitly handles the tanh transformation used in practical SAC implementations. The use of push-forward measures and the invariance of KL divergence under bijective transformations provides a clean mathematical framework for this analysis. This attention to the actual policy parameterization makes the contribution more compelling than an analysis restricted to abstract policy-space updates.

  • The paper is generally well written, with a logical progression from the policy-space formulation to the parameterized actor, curvature analysis, tracking bounds, and final convergence theorem. The numerical illustrations are also useful. In particular, Figure 2 provides an intuitive illustration of the relationship between the mirror-descent parameter and the actor tracking error, complementing the theoretical findings.

Critical weaknesses

I have only a few relatively minor reservations.

  • The analysis assumes access to a sufficiently accurate critic oracle rather than explicitly studying the coupled actor-critic dynamics. Although this is a reasonable first step and allows the authors to isolate the actor optimization problem, extending the results to a learned critic would further strengthen the practical significance of the work.

  • The analysis focuses on fixed-covariance squashed Gaussian actors with a linear mean parameterization. It would be interesting to understand whether similar curvature conditions can be derived for more expressive neural-network actors.

Questions for authors
  • Theorem 2 provides an interesting sufficient condition involving the curvature of the critic and the entropy regularization parameter. Can the authors comment on how restrictive this condition is in practice, particularly when the critic is represented by a neural network?

  • Theorem 3 establishes an O(N−1/5)\mathcal{O}(N^{-1/5}) rate. Do the authors believe that this rate is intrinsic to the single-step actor-tracking problem, or could a refined analysis yield faster convergence under the same assumptions?

  • Could the authors elaborate on which aspects of their analysis might extend to jointly learning the actor and critic, rather than relying on a critic oracle?

Reviewer_vY5DOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents a population-level theoretical analysis of Soft Actor-Critic for entropy-regularized reinforcement learning with continuous action spaces. It considers tanh-squashed Gaussian policies with fixed covariance and linearly parameterized means, an approximate critic oracle, and an actor that tracks an evolving target policy using a single projected gradient update per iteration. The authors establish an O(N(−1/5))O(N^(-1/5)) best-iterate finite-time convergence guarantee up to actor and critic approximation errors.Furthermore, the paper shows that the mirror-descent step size lambda directly controls target drift and actor tracking error, providing a trade-off between policy improvement and the actor’s ability to track the target..

Critical strengths
  • The paper addresses a well-motivated, clearly formulated, and meaningful problem: how an actor can track an evolving target policy with only a single gradient update per iteration.
  • The theoretical results are valuable, providing explicit conditions for strong convexity of the actor objective and clarifying how the mirror-descent step size controls target drift and actor tracking error.
  • The paper is well organized and easy to follow
Critical weaknesses

The upper bound of the tracking error for Gibbs' target in Corollary 1 includes a non-vanishing term, but this does not necessarily mean that the actual tracking error cannot approach zero. I understand that the paper does not make this stronger assertion. Could the authors further clarify to what extent the difference in the upper bound reflects the actual behavior of the two targets rather than the looseness of the analysis? In particular, when the critic stabilizes over training, could the tracking error for Gibbs' target also approach zero?

Questions for authors

see weakness part

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.