-
My main concern is Assumption 5, which requires the performance objective L(θ) to be globally m-strongly concave over $ \mathbb{R}^d$. This assumption is not satisfied by standard policy optimization problems. Even in the simpler discounted setting, the policy optimization objective is generally non-concave with respect to the policy parameters. More importantly, the assumption appears to be mathematically inconsistent with the setting considered in the paper. Since rewards are bounded by Ur, the average-reward objective satisfies
∣L(θ)∣≤Ur,∀θ∈Rd.
However, global strong concavity implies
L(θ)≤L(θ0)+⟨∇L(θ0),θ−θ0⟩−2m∥θ−θ0∥2.
The right-hand side tends to −∞ as ∥θ∥→∞, contradicting the boundedness of L. Thus, Assumptions 5 and the bounded-reward setting cannot simultaneously hold as stated. This is not a minor technical restriction: the main convergence result critically relies on this assumption. Consequently, I do not see a nontrivial class of MDPs satisfying the complete set of assumptions under which the theorem can be applied.
-
Without the strong concavity assumption, the main technical ingredients do not appear to substantially advance the existing finite-time analysis of actor-critic algorithms, particularly the work of |1]. The proof relies on a decomposition of the actor error into martingale noise, Markovian sampling error, critic estimation error, and average-reward estimation error. Such decompositions and the control of coupled actor-critic recursions are already well established in the literature. The principal distinction is the conversion of these error estimates into an all-time parameter-distance guarantee. However, this step relies crucially on the problematic strong concavity assumption. The paper should identify more precisely which technical arguments are genuinely new relative to existing analyses.
-
The presentation is particularly difficult to follow. Several constants and technical conditions appear immediately before the main theorem without sufficient explanation of their roles. The proof sketch contains lengthy expressions involving multiple error terms and auxiliary martingale sequences, but provides little intuition about the key analytical difficulties or how they are resolved. The relationship between the different lemmas, the choice of learning rates, and the resulting concentration bound is not explained clearly. The main text is dominated by technical manipulations rather than an accessible account of the proof strategy. In its current form, I found it unnecessarily difficult to identify the actual mathematical contribution.
-
Theorem 1 contains residual terms depending on the critic approximation error and the mixing parameters. However, the simplified rate emphasized in the abstract and Corollary 1 omits these contributions. The authors should clarify precisely under which additional conditions these terms can be neglected. In particular, the assumption b≤ϵapp is unusual, since it connects the mixing constant to the approximation error without a clear justification. It is also unclear how the exact-representation case is intended to fit this assumption when b>0.
-
The discussion overlooks several important recent developments in actor-critic theory. In particular, the papers [2] and [3] should be discussed. Although these works study different settings, they provide important context for assessing the assumptions, convergence guarantees, and novelty of the present paper.
[1] Chen, Xuyang and Zhao, Finite-Time Analysis of Single-Timescale Actor-Critic, NeurIPS 2023
[2] Kumar et al, On the Convergence of Single-Timescale Actor-Critic, NeurIPS 2025
[3] Labbi et al. Refined Analysis of Entropy-Regularized Actor-Critic, ICML 2026