acceptodds
Under review as a conference paper at ICLR 2027

Advanced Policies: A First-Principles Path from Policy Gradient to Q-Learning

Abstract

We apply entropy to reinforcement learning along two axes: entropy-augmented reward defines the soft objective, while relative-entropy regularization controls the size of a policy-improvement step without changing that objective. Together they yield a one-parameter family of advanced policies connecting a base policy to its soft-greedy policy. Under exact evaluation, every nontrivial member improves upon the common base, although the improvement need not be monotone along the family. With a consistent policy and action-value function, this family forms an entropic mirror-descent path. We then relax this consistency, treating the policy and action-value function as independent coordinates of the advanced policy. Differentiating the objective of the advanced policy yields Advanced Actor–Critic (AAC), whose endpoint gradients recover soft policy gradient and an action-centered Q-learning-like update. We develop implementations for discrete and continuous actions. Experiments validate both endpoints and demonstrate effective learning at intermediate parameter values.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Policy Optimization, Entropy Regularization, Policy Gradient, Q-Learning, Actor-Critic Methods, Mirror Descent, Advanced Policy

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_qqaoOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a policy family connecting a base policy to its soft-greedy update. It derives Advanced Actor–Critic (AAC) by optimizing independent policy and Q coordinates, recovering soft policy gradient and a centered Q-learning-like update at the endpoints. Evaluation includes exact-MDP diagnostics and discrete and continuous control tasks.

Critical strengths

The joint policy/Q formulation provides an interesting perspective on actor–critic learning. The explanation of action-relative Q updates helps clarify the endpoint behavior. The exact-MDP diagnostics distinguish improvement over the base policy from monotonic improvement along the path. Failure cases and seed variation are also reported clearly.

Critical weaknesses

The policy construction shares the form of the KL-and-entropy-regularized update in Leverage the Average: an Analysis of KL Regularization in RL by Vieillard et al. (2020), which is not cited. Although AAC uses different parameter updates, the additional insight from independently optimizing policy and Q coordinates needs stronger justification. The experiments mainly compare AAC variants, with different direct learning rates across variants. Without matched standard baselines and controlled ablations, it is difficult to determine the benefit of the additional components.

Questions for authors
  1. What specific new insight or optimization property does AAC provide beyond the policy construction in Leverage the Average: an Analysis of KL Regularization in RL by Vieillard et al. (2020) ?

  2. Can you provide a matched SAC comparison and a chasing ablation with fixed direct learning rates? These would help assess the practical contribution.

Reviewer_9tuNOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper derives a family of entropy-regularized policies connecting a base policy to its soft-greedy update. AAC then optimizes a composite policy through independent policy and action-value parameters, using a separate executable actor for continuous actions. The paper studies Direct, Chasing, and Combined updates through finite-state diagnostics and seven control environments.

Critical strengths
  1. The paper gives a coherent account of reward entropy, step-level KL regularization, and the policy-gradient and Q-learning-like endpoints. It carefully distinguishes improvement over a common base from monotonic improvement along the policy family. These distinctions clarify what the construction guarantees and avoid overstating its relationship to standard Q-learning.

  2. The exact finite-state diagnostic provides useful evidence about representation. Across paired initializations, stronger chasing substantially reduces the number of modes in the implicit policy. Reporting this alongside the small change in executable return makes the distinction between implicit expressivity and realized control performance clear.

Critical weaknesses
  1. The connection between the exact Direct-AAC gradients and their practical implementation is not sufficiently validated. Equations 26-27 use expectations under the implicit advanced policy and its on-policy state distribution, whereas the continuous implementation uses a squashed-Gaussian approximation, a critic evaluating that approximation, and replay states. Appendix A.8 explains these substitutions, but does not quantify their effect on the update direction. This leaves unclear how closely the practical updates follow the derived gradients.

  2. The experiments do not establish when the additional parameterization is useful in practice. There are no matched SAC, TD3, or PPO comparisons, and learning rates differ across the AAC update modes. The multimodality diagnostic also shows little change in executable return. The results demonstrate that AAC can learn, but do not yet separate the benefit of independent policy–value coordinates from the effects of optimization settings and policy projection.

Questions for authors

Please address the concerns listed in the Weaknesses section above.

Reviewer_nL8aOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates two distinct uses of entropy in reinforcement learning, entropy augmentation at the reward level to define the soft objective and relative-entropy regularization to control individual policy-improvement steps. The authors introduce an advanced-policy family that interpolates between the current policy and its soft-greedy counterpart through a parameterized formulation. The paper also presents a theoretical analysis and empirical evaluations comparing different variations of the proposed approach.

Critical strengths
  1. The distinction between reward-level entropy augmentation and step-level relative-entropy regularization provides an interesting perspective on entropy-based policy optimization.
  2. The theoretical formulation of the advanced-policy family offers a potentially useful framework for understanding policy-improvement behavior under different regularization settings.
Critical weaknesses
  1. The empirical evaluation is limited and does not sufficiently establish the advantages of the proposed approach over existing reinforcement learning algorithms. The experiments primarily compare variations of the proposed method, without clear comparisons against established baselines such as PPO, SAC, or TD3. Consequently, it is difficult to determine whether the theoretical formulation translates into meaningful practical improvements or supports the broader motivation of the paper.
  2. The novelty and practical implications of the proposed framework are not sufficiently clear. Although the paper provides theoretical analysis, its significance relative to existing policy optimization approaches remains unclear. Furthermore, the different experimental variations are not adequately explained in terms of their contributions to the main objective. Table 1 does not clearly establish which configuration performs better or should be preferred, while the absence of learning curves makes it difficult to assess training behavior and overall learning efficiency.
Questions for authors
  1. What are the main theoretical contributions beyond existing entropy-regularized and mirror-descent policy optimization formulations, and how do they translate into practical advantages?
  2. How does the proposed approach compare empirically with established algorithms such as PPO, SAC, and TD3 under comparable evaluation settings?
  3. What are the differences between the evaluated variations in Table 1, and which configuration is recommended based on the reported results?
  4. Could the authors provide learning curves to clarify the training dynamics, convergence behavior, and relative performance of the different configurations?
1 comment
AuthorsOpenReview ↗

Summary of Revisions Addressing Your Concerns

Thank you again for your constructive feedback. We would like to briefly summarize how the revision addresses your main concerns.

We clarified how the policy-gradient and action-centered soft Q-learning-like endpoints arise from a single advanced-policy objective. We also expanded the comparisons with standard baselines, added learning curves across all seven environments, and clarified the roles and limitations of Direct, Chasing, and Combined AAC, without claiming a universal performance advantage.

We hope these additions help address your concerns, and we would welcome any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.