acceptodds
Under review as a conference paper at ICLR 2027

GAE: Gradient-Aligned Generalized Advantage Estimation for Adaptive Selection

Abstract

Generalized Advantage Estimation (GAE) is a widely used advantage estimation method in reinforcement learning, with its performance substantially influenced by the trace parameter . Since the effective choice of can vary across tasks and stages of training, existing policy optimization methods typically rely on manually fixed values. Prior works generally motivates selection through the bias–variance trade-off of advantage estimation, which does not directly characterize the effectiveness of the resulting policy updates. In this work, we revisit selection from the perspective of policy optimization and introduce Gradient-Aligned Generalized Advantage Estimation (GAE), a novel framework that adaptively selects by maximizing the alignment of candidate policy gradients with a reference gradient that approximates the true policy gradient direction. We establish a theoretical connection between gradient alignment and policy improvement, and develop a parameter-free algorithm that can be seamlessly integrated into existing policy optimization methods such as PPO. Extensive experiments on MinAtar and MuJoCo benchmarks demonstrate that GAE consistently achieves strong and robust performance across diverse environments while reducing the dependence on manually tuned fixed values. Code is available at https://anonymous.4open.science/r/PGAE-B4EB.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Generalized Advantage Estimation

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_3SPMOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Overall, this paper mainly investigates the problem of parameter selection in GAE calculations within PPO methods. Briefly, the authors observe that previous studies have largely discussed parameter selection from the perspective of the bias-variance trade-off, without directly characterizing its impact on policy updates. This paper addresses this problems and proposes GA$^2$E; extensive experiments appear to validate/support the proposed method.

Critical strengths

I believe the problem this paper propose to address is significant (and might be interested by RL community); understanding and improving GAE within PPO is indeed a common challenge in the practical application of reinforcement learning. Furthermore, the paper presents a substantial number of experiments and results.

Critical weaknesses

This paper appears to contain numerous errors, including incorrect, fabricated, or incomplete citations, which severely undermine its credibility.

For instance, the authors of many cited works are listed simply as "Unknown"—examples include "Unknown. A closer look at deep policy gradients. In International Conference on Learning Representations, 2020." and "Unknown. Fast proximal policy optimization. In International Conference on Intelligent Computing, Lecture Notes in Networks and Systems." These appear to be AI-generated or improperly used citations.

Questions for authors

I recommended that the author should carefully check the citations in the manuscript before submission to avoid similar errors (see "Critical Weaknesses"), or alternatively, provide an explanation regarding this issues.

Reviewer_uLPZOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the issue of manually fixing the trace parameter λ\lambda in GAEs by defining the optimal λ\lambda from a policy optimization perspective. A gradient alignment criterion is obtained through a first-order expansion, and a grefg_{\text{ref}} is constructed on the disjoint reference set with a fixed λref\lambda_{\text{ref}}. Then, a two-stage grid search is used to select λ∗\lambda^*.

Critical strengths

This paper includes a wide range of testing environments, including Atari for discrete actions and MuJoCo for continuous actions, and provides metrics such as Mean, Median, and IQM. The experimental scope and metrics in this paper are very comprehensive.

Critical weaknesses
  • The paper neither discusses nor compares against the meta-gradient[1], in which the trace parameter λ\lambda is already optimised as a learnable meta-parameter.
  • The paper provides no empirical comparison with the closest adaptive-λ\lambda methods mentioned (AAE, Chen et al., 2021); the baselines are limited to DAE, RVL, EAPO, and fixed-λ\lambda PPO.
  • The gradient alignment objective is an agreement with a fixed reference estimator. Its population optimum is λref\lambda_\text{ref} by construction, and its connection to ∇J\nabla J is never measured. g_ref is the GAE gradient at a specific fixed λ\lambda, but it is still affected by the value function error, making it difficult to interpret as an estimate of ∇J\nabla J.
  • Five bibliography entries carry "Unknown" as the author, producing invalid in-text citations.

[1] Xu Z, van Hasselt H P, Silver D. Meta-gradient reinforcement learning[J]. Advances in neural information processing systems, 2018, 31.

Questions for authors
  • How close is grefg_{\text{ref}} computed with λref\lambda_{\text{ref}} in the current batch to a Monte Carlo policy gradient estimated on a very large batch?
  • Compared to the current first-order expansion, is considering an optimization objective with a trust region more appropriate?
Reviewer_j1YeOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to determine the lambda parameter of Generalized Advantage Estimation (GAE), which is one common building block for actor-critic reinforcement learning. The key argument of the paper is that λ\lambda shall be chosen for the effectiveness of policy optimization rather than achieving bias-variance tradeoff for the critic itself. By formulating λ\lambda selection based on the quality of the resulting policy updates, this work proposes Gradient-Alignment Generalized Advantage Estimation (GA^2E), which shows that this objective has a tractable first-order characterization. More specifically, GA^2E maximizes the alignment between candidate policy gradients and the true optimization direction, which is unavailable in practice and can be approximated by an estimated reference gradient. GA^2E is then evaluated in both MinAtar and MuJoCo, and the results show that GA^2E can significantly improve over PPO-like algorithms with vanilla GAE.

Critical strengths
  • The problem studied in this work is fundamental to general RL, and the proposed solution is conceptually simple and easy to implement.
  • The empirical performance of GA^2E appears quite strong compared to vanilla GAE over a broad range of λ\lambda parameters.
Critical weaknesses
  • Regarding the algorithm, the proposed λ\lambda selection method appears somewhat heuristic. Specifically, the reference gradient is itself a noisy estimate of the true gradient, and the error can in practice be large and thereby make S(λ)S(\lambda) quite noisy.

  • The chosen baselines are not very strong. Specifically, PPO, DAE, RVL, and EAPO all use the same fixed λ=0.95\lambda=0.95, which is configured without much justification. Moreover, as suggested by Figure 5, the best λ\lambda is likely to be around 0.6 to 0.8 in the environments considered by this paper.

  • Important baselines are missing in the comparison. Specifically, there are two relevant lines of work: (1) GAE with adaptive λ\lambda: (Chen et al., 2024). (2) Online hyperparameter tuning for GAE or TD(λ\lambda): (Xu et al., 2018), (White and White, 2016), and (Zhao et al., 2020). Without a comparison with these baselines, it is hard to fully assess the proposed method.

  • The paper uses PPO-clip as the base algorithm. However, the reference gradient is constructed without any clipping. It is not clear whether this mismatch would affect the performance.

  • The theoretical results (Lemmas 1 and 2, and Theorem 1) appear relatively straightforward and offer limited insight into the algorithm’s design. Equation (20) appears to be motivated independently of these results and can stand on its own.

Some additional comments:

  • Equation 7: λ∗\lambda^* shall depend on θ\theta, and it would be helpful to make this dependency explicit.

  • The reference batch and candidate batch are constructed are of size 1:4. It would be good to elaborate on this choice.

  • Figure 4: Some colors are very similar, making the corresponding curves difficult to distinguish (e.g., λ=0.8\lambda = 0.8 and 0.90.9). Adding distinct markers or line styles would help differentiate the curves and clarify the legend.

References:

Chen et al., “Adaptive bias-variance trade-off in advantage estimator for actor–critic algorithms,” Neural Networks, 2024.

Xu et al., “Meta-Gradient Reinforcement Learning,” NeurIPS 2018

White and White, “A Greedy Approach to Adapting the Trace Parameter for Temporal Difference Learning,” AAMAS 2016

Zhao et al., “META-Learning State-based Eligibility Traces for More Sample-Efficient Policy Evaluation,” AAMAS 2020

Questions for authors
  • As the reference gradient is itself a noisy estimate of the true gradient, the error can in practice be large and thereby make S(λ)S(\lambda) quite noisy. How does GA^2E handle this issue?

  • What are the pros and cons of GA^2E compared to the related work mentioned above?

  • Why choosing λ=0.95\lambda=0.95 for all the PPO-like algorithms?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.