- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces a policy family connecting a base policy to its soft-greedy update. It derives Advanced Actor–Critic (AAC) by optimizing independent policy and Q coordinates, recovering soft policy gradient and a centered Q-learning-like update at the endpoints. Evaluation includes exact-MDP diagnostics and discrete and continuous control tasks.
Critical strengths
The joint policy/Q formulation provides an interesting perspective on actor–critic learning. The explanation of action-relative Q updates helps clarify the endpoint behavior. The exact-MDP diagnostics distinguish improvement over the base policy from monotonic improvement along the path. Failure cases and seed variation are also reported clearly.
Critical weaknesses
The policy construction shares the form of the KL-and-entropy-regularized update in Leverage the Average: an Analysis of KL Regularization in RL by Vieillard et al. (2020), which is not cited. Although AAC uses different parameter updates, the additional insight from independently optimizing policy and Q coordinates needs stronger justification. The experiments mainly compare AAC variants, with different direct learning rates across variants. Without matched standard baselines and controlled ablations, it is difficult to determine the benefit of the additional components.
Questions for authors
-
What specific new insight or optimization property does AAC provide beyond the policy construction in Leverage the Average: an Analysis of KL Regularization in RL by Vieillard et al. (2020) ?
-
Can you provide a matched SAC comparison and a chasing ablation with fixed direct learning rates? These would help assess the practical contribution.
Summary of Revisions Addressing Your Concerns
Thank you again for your constructive feedback. We would like to briefly summarize how the revision addresses your main concerns.
We clarified how the policy-gradient and action-centered soft Q-learning-like endpoints arise from a single advanced-policy objective. We also expanded the comparisons with standard baselines, added learning curves across all seven environments, and clarified the roles and limitations of Direct, Chasing, and Combined AAC, without claiming a universal performance advantage.
We hope these additions help address your concerns, and we would welcome any further questions.