- Rating
- 3(+2)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper analyzes a population-level formalization of Soft Actor-Critic (SAC) that matches common implementations: a fixed-covariance -squashed Gaussian actor with linear mean, one projected reverse-KL actor step per outer iteration, and a critic oracle with uniform error . It compares the classical Gibbs target in Eq. (4) with a policy mirror-descent (MD) target with proximal step size in Eq. (5), where the MD target is updated from the previous target rather than from the current actor. Under optimal-occupancy concentrability and a Legendre-operator curvature condition on the critic, , the reverse-KL actor objective is shown to be strongly convex and smooth on a parameter ball. The analysis separates target drift from actor tracking error: the MD target drift scales with , while Theorem 1 also contains improvement terms; Corollary 1 shows that the Gibbs tracking bound contains a non-vanishing term. Theorem 3 gives a best-iterate regularized value gap of order plus approximation error under a prescribed scaling and sufficiently small actor step size . The experiments in Appendix K are restricted to a small tabular MDP with exact or perturbed critics, illustrating curvature effects and MD-versus-Gibbs tracking; there is no deep-RL or empirical rate validation.
Critical strengths
- Many MD/convergence papers update in policy space or assume exact convex projections; this paper's actor model matches tanh-squashed Gaussian SAC with a single reverse-KL step, which is how SAC is often run in code.
- Corollary 1's persistent term vs -controlled MD drift gives a precise sense in which a fresh softmax target can be harder to track under partial actor updates; Figure 2 illustrates the phenomenon under exact critics.
- Theorem 3's best-iterate gap with is a concrete timescale prescription, rare in SAC analyses that include evolving targets.
- Theorem 2 explains when the actor KL subproblem is strongly convex/smooth via ; Appendix K shows matched but different curvature can yield qualitatively different learning (Figure 1).
- The authors acknowledge population-level abstraction and that experiments are mainly qualitative support for theory rather than benchmark validation.
Critical weaknesses
- A uniform does not model TD bootstrapping, target networks, double Q-learning, nonstationary replay, or stochastic gradients—so the main theorems are not guarantees for standard training loops.
- The condition is strong and unverifiable in practice. Constructive discussion is largely for Fourier linear critics (Remark 1). There is no proposal for measuring or enforcing this during neural critic learning.
- Many implementations take multiple actor updates per critic step; the paper does not analyze how steps interact with Theorem 3 or tracking bounds.
- Eq. (5) defines from the previous target, not necessarily from or slow Q-target networks alone; the incremental novelty over “proximal targets move slowly” should be sharpened given the built-in KL term.
- Even accepting the theory, readers cannot see log-log validation of the exponent in Appendix K; constants (e.g., the strong convexity constant) may be tiny in reported regimes.
Questions for authors
- Can you provide checkable sufficient conditions (architectural constraints, regularization, Lipschitz/Legendre bounds, or online estimators) ensuring for non-Fourier critics with high probability during training?
- How do Corollary 1 and Theorem 3 change if the actor performs projected gradient steps per critic update? Is there a recommended scaling of with analogous to ?
- What is a plausible roadmap to replace the oracle critic with TD sample complexity—even in a linear function approximation setting with replay?
- If the Gibbs target is slowed (EMA on , Q target network) to mimic small effective drift of MD at small , does Figure 2's MD advantage persist? Please quantify.
- Can you add log-log plots of best-iterate value gap vs on Appendix K and report estimated slopes vs the theoretical exponent?
- How sensitive is Theorem 3 to the projection radius ? When minimizers lie on the boundary of , do rates degrade in practice?