acceptodds
Under review as a conference paper at ICLR 2027

Finite-Time Generalized Advantage Estimation

Abstract

Advantage estimation is a central component of actor-critic reinforcement learning algorithms, such as A2C or PPO. One of the most widely used methods is generalized advantage estimation (GAE). However, its adaptation to practical finite-time rollout implementations is not covered by the infinite-horizon theory commonly used to derive GAE. This paper suggests to explicitly take into account the nature of finite-time rollouts to redefine the weighting of mixtures of -step advantages. We analyze how standard truncated GAE changes the effective weighting of available estimators near rollout endpoints and propose finite-time GAE (FT-GAE), which normalizes geometric weights over the available targets and retains a linear-time backward recursion. It is shown how our reweighting changes estimation error near rollout endpoints analytically under controlled bias-variance models and empirically in a controlled gridworld experiment. Extensive deep RL experiments across diverse benchmark families show substantial improvements when truncated GAE is replaced with FT-GAE. Controlled terminal-signal ablations provide further evidence that its advantage can be particularly pronounced when endpoint signals are informative.

Primary area:
reinforcement learning
Keywords:
Generalized Advantage Estimation (GAE), Reinforcement Learning, Finite-horizon estimation, Proximal policy optimization (PPO), Bias-Variance Trade-off

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_6XAGOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper identifies a theory vs. practice mismatch: episode rollouts in real-world RL applications trivially can never reach infinite time. Therefore, the author's propose FT-GAE, which aims to heal estimations errors near rollout endpoints. The proposed method is tested on 24 different benchmark environments from 5 suits to find that it is most beneficial in scenarios where the most important information is in endpoint signals.

Critical strengths
  1. As it is a minor change to the recursion in the GAE equation, the proposed method is simple, practical and of identical computational cost.

  2. The experiments are thorough and extensive. The used environments are quite diverse. Explicitely testing that retuning λ\lambda does not achieve just similar improvements is something that is oftentimes forgotten and therefore genuinly strengthens the paper's claim.

Critical weaknesses
  1. GAE is used as baseline and only comparator. An obvious alternative might be to lower λ\lambda on the last few steps before an endpoint. Another alternative is Daley et al (2024), as the authors say to have followed its "TD-error decomposition of compound returns" but replace the "equicorrelation model". That replacement itself is never tested.
Questions for authors
  1. I have found Sharma et al. (2017) "Learning to Mix n-Step Returns: Generalizing lambda-Returns for Deep Reinforcement Learning". Your variant looks like a fixed parameter version of what they used. If that is true, this should be at least discussed somewhere in the paper.

  2. Freeway has no terminations as I undestand it. So where does the λ\lambda-sweep gain happen? What explains the gain where no informative terminal signal is present?

  3. You are open about the limitations: only GAE as baseline & FT-GAE not always superior. My main reason for voting "weak" instead of "clear accept" is those missing comparisons (see weaknesses). As the principle of FT-GAE is so simple and cheap, it would be perfect to understand when exactly it helps and when it does not. What were the reasons to not include more comparators?

3 comments

Dear reviewer,

Thank you for your careful review! We address your comments below and in Appendices C–E (see red bars) and invite you to take a look.

W1 / Q3: Additional baselines:

Thank you for your suggestion. We now also compare a linear endpoint-dependent schedule (Lin-Trace) and a PPO adaptation of PiLaR [1] on LunarLander-v3. Mean final returns were 252.17 for Truncated GAE, 275.55 for FT-GAE, 269.65 for Lin-Trace, and 249.03 for PiLaR. Lin-Trace recovered about three quarters of the observed gap between FT-GAE and truncated GAE. Please see our response to Reviewer wfa1 for the schedule definition. We will report the new comparison as an additional ablation in Section 5.

W2: Covariance model:

The mathematical insights can analogously use the equicorrelation assumption of [1]. We added results for the equicorrelation model in Appendix C and observe that the pattern stays the same. We chose the covariance structure in Assumption 4.1 because we expected TD errors farther apart in time to be less correlated. Our geometric decay has the same form as the covariance function of a stationary AR(1) process, a standard model for temporal dependence [2]. We also estimated TD-error covariances at different lags to provide empirical motivation for modeling weaker dependence at longer lags. The estimates averaged across training checkpoints decreased with greater lags (Figure 17).

Q1: Related work:

Thank you for pointing out this connection. We agree that FT-GAE can be viewed as a fixed weighting rule within the weighted-return framework [3] by applying their (Eq. 5) to the fixed logits c(st+i,t+i)=(t+i−1)log⁡λc(s_{t+i}, t+i)=(t+i-1)\log\lambda over the available horizons i=1,…,τ−ti=1,\ldots,\tau-t. We will cite [3] and clarify this relationship along with the other related literature.

Q2: Freeway:

In Freeway, the agent receives a reward only when it completes a crossing. Even when a PPO rollout contains no reward, a critic that has learned to predict future crossings can still provide a learning signal through bootstrapping. At high λ\lambda, FT-GAE places more weight on shorter, bootstrapped returns than Truncated GAE near the rollout boundary. This explains its advantage at high values of λ\lambda. Note that for the best λ\lambda results, the two learning curves nearly overlap.

Q3: When might Truncated GAE help?

At a true episode termination, Truncated GAE assigns the remaining mixture weight to the Monte Carlo return. When successes are rare and the critic fails to predict the delayed reward accurately (e.g. in early training when first observing signals), this extra weight can help credit earlier actions in a successful trajectory. This aligns with prior work which found that Monte Carlo updates can be beneficial in tasks with sparse or delayed rewards [4, 5].

Motivated by [6], we compared Truncated GAE and FT-GAE on MiniGrid DoorKey. Increasing the grid side length from 5 to 8 gives the agent more locations to search the key before it can open the door and reach the goal to observe its only reward. Thus, a larger grid makes both exploration and temporal credit assignment harder.

Grid sizeTraining stepsTruncated GAE returnFT-GAE return
5×5100k0.770.87
8×8100k0.000940.00036
8×8500k0.180.06
8×81M0.310.276
8×82M0.390.46
8×85M0.480.65

Results are averaged over 15 seeds. The first row reports results already reported in the initial version of the paper, where both methods learned within the standard 100,000-step budget, and FT-GAE achieved a higher mean return. On 8×8, both earned almost no reward at that budget. In longer 8×8 experiment, Truncated GAE had the higher mean return up to 1 million steps. FT-GAE had the higher mean at the reported checkpoints from 2 million steps onward. The change in ordering is consistent with a trade-off that shifts during training. Early on, when a successful rollout provides a reward that the critic predicts poorly, tail-mass collapse gives the observed Monte Carlo return more weight and can strengthen the learning signal for earlier actions in the same rollout segment. As the critic learns to predict future success, shorter bootstrapped returns can convey that signal with lower variance, making FT-GAE's weighting more useful. We will add above experiment to the ablations in the revised version.

[1] Daley et al. (2024). Averaging n-Step Returns Reduces Variance in RL.

[2] Brockwell & Davis (2016). Introduction to Time Series and Forecasting.

[3] Sharma et al. (2017). Learning to Mix n-Step Returns: Generalizing λ\lambda-Returns for Deep Reinforcement Learning.

[4] Ostrovski et al. (2017). Count-Based Exploration with Neural Density Models.

[5] Amiranashvili et al. (2018). TD or not TD: Analyzing the Role of Temporal Differencing in Deep Reinforcement Learning.

[6] Kayal et al. (2025). The impact of intrinsic rewards on exploration in Reinforcement Learning.

Thank you for the detailed response. The connection to Sharma et al. and the covariance-model question seem resolved to me, and I appreciate the candid answers on Freeway and on when truncated GAE helps. The DoorKey crossover seems like a useful practical insight.

I have a few follow-up questions:

  1. About Lin-Trace: It seems that a simple linear endpoint-dependent schedule recovers most of the gain on LunarLander. Would you agree that this suggests much of the benefit comes from relying less on long returns near endpoints in general, rather than from the specific renormalized weighting?
  2. How many seeds were used for the new LunarLander comparison, and what are the confidence intervals? It is not clear to me whether the remaining gap between FT-GAE and Lin-Trace is meaningful. The same question applies to the DoorKey 8×8 table, where several differences seem small relative to typical seed variance.
  3. Freeway. From your answer, it seems the Freeway gain is mainly robustness to high λ, since the curves nearly overlap at the best λ, right? Does this also apply to the default-λ results more generally?
  4. Revision. As far as I can see, the revised PDF contains the additions in Appendices C–E, but not yet the new baselines, the DoorKey experiment or the discussion of Sharma et al. Do you plan to add these before the end of the discussion period, and could all changed sentences be highlighted in color as well? This would really help in reviewing the PDF. Thanks.

Thanks for engaging in the discussion! We have updated the main text and highlighted the changes in the revised PDF.

Q4: Revsion:

  • We now cite Sharma et al. (2017) in the related work section and explain how FT-GAE fits into their weighted-return framework in Remark 3.2.
  • In Section 5.6, we report the LunarLander-v3 comparison with Lin-Trace and PiLaR and extend it to include two additional endpoint-dependent schedules. Both achieved mean final returns between those of truncated GAE and FT-GAE.
  • In Section 5.7, we report the DoorKey 8×8 experiment.

Q1: linear endpoint-dependent schedule:

Yes, we agree that the LunarLander-v3 results suggest that much of the benefit comes from placing less weight on long returns near endpoints. At a true episode termination, truncated GAE assigns the remaining mixture weight to the Monte Carlo return. In LunarLander-v3, that return includes a terminal reward of +100 for landing or −100 for crashing. When the outcome is uncertain, this 200-point difference can make the Monte Carlo return highly variable, and giving it extra weight can increase the estimators variance.

FT-GAE was explicitley designed to address tail-mass collapse. It renormalizes the geometric mixture over the available kk-step returns, locally giving shorter, bootstrapped returns more weight. The backward recursion derived from this mixture (Proposition 3.3) yields smaller recursion coefficients near endpoints. This motivated the endpoint-dependent schedules we tested. While FT-GAE retains λ\lambda as its only tuning parameter. The schedules also require tuning the length of the window over which the coefficient is reduced. In our experiment the endpoint-dependent schedules recovered only part of the gap between truncated GAE and FT-GAE. FT-GAE achieved the highest mean final return, and its confidence interval lay entirely above those of all other methods in the comparison (see Q2).

Q2: Experiment details

For LunarLander-v3, we used five seeds per hyperparameter configuration and then trained each method with its selected configuration on 30 fresh seeds. The comparison now includes two additional Step-Trace variants, which set the local trace coefficient to λ/4\lambda/4 or λ/2\lambda/2 within a window of ww steps before the endpoint. The table now includes 95% CI.

MethodMean final return [95% CI]
Truncated GAE252.17 [246.24, 257.44]
FT-GAE275.55 [273.45, 277.33]
Lin-Trace269.65 [267.10, 272.03]
Step-Trace (0.25)266.89 [262.11, 271.08]
Step-Trace (0.5)269.29 [266.36, 271.87]
PiLaR249.03 [240.40, 256.15]

Section 5.6 also reports pairwise estimates of the probability of improvement with 95% confidence intervals. The point estimate for FT-GAE was at least 75% against each comparator.

For the door-key experiment, we trained both methods on 15 additional seeds and now report results for 30 seeds per method in Section 5.7 and below, including 95% bootstrap confidence intervals.

Training stepsTruncated GAEFT-GAERelative change vs. Truncated GAE
100k0.0573 [0.0169, 0.1081]0.0026 [0.0002, 0.0058]−95.4% [−99.4%, −86.2%]
500k0.2591 [0.1292, 0.4205]0.1448 [0.0399, 0.2712]−44.1% [−84.9%, +50.8%]
1M0.3161 [0.1617, 0.4780]0.2428 [0.1059, 0.3958]−23.2% [−69.7%, +79.4%]
2M0.3847 [0.2232, 0.5511]0.4054 [0.2423, 0.5748]+5.4% [−45.8%, +111.9%]
5M0.5162 [0.3516, 0.6834]0.6150 [0.4528, 0.7773]+19.1% [−20.0%, +81.6%]

Q3: Freeway:

Yes, on Freeway the main benefit is greater robustness at high λ\lambda. The learning curves nearly overlap when each estimator uses its best-performing λ\lambda. But this is not true in general for the default λ\lambda results. In Section 5.2, we test whether the gains observed at the default λ\lambda persist after retuning. The results of Table 1 shows that FT-GAE still siginficantly improves over truncaeted GAE in Box2D, MinAtar and Minigrid environments. In all three suites, the 95% confidence intervals for the probability of improvement lie entirely above 0.5

We hope the added comparisons address the main concern you raised in your initial review, and we would appreciate it if you would reconsider your rating.

Reviewer_BZV7OpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper works on the problem of advantage estimation in actor-critic reinforcement learning. One may look k-steps ahead in the rollout to compute the advantage estimator at any time point tt; and increasing kk typically reduces the bias but increases the variance of the estimator. Generalized Advantage Estimation (GAE) forms a geometrically weighted mixture of kk-step advantage estimators. In practice, a truncated GAE is used because rollouts are finite. The paper identifies that such a truncation may assign a large mass on the longest available estimator, and call this issue tail-mass collapse.

The paper introduces finite-time GAE (FT-GAE) which normalizes the geometric weights over the available time horizon, and provides an algorithm (Algorithm 1) using a recursive implementation of its advantage estimator. Section 4.1 provides experimental justification for using FT-GAE over the usual Truncated GAE for advantage estimation and identifies that the main issue FT-GAE solves is the high variance problem in the truncated GAE estimator near the end of a rollout. Section 4.2 also proves this theoretically under a decaying correlation assumption.

Section 5 has several experiments comparing FT-GAE and Truncated GAE over practical RL tasks and ablation analyses.

Critical strengths
  1. Section 4.1 of the paper has an interesting experiment in a time-inhomogeneous GridWorld environment comparing FT-GAE and Truncated GAE. Panels c and d are instructive as they show that the advantage of FT-GAE is in correcting the estimator when few kk-step targets remain.
  2. Section 5 is a strong experimental section showing the advantage of FT-GAE in several tasks and ablation experiments. The endpoint importance ablation is particularly informative. Although the high variance of Truncated GAE near the endpoint has been observed before in literature (see for instance Song et al. (2023) cited in this paper), the particular mechanism of tail-mass collapse and how FT-GAE corrects it is a novel contribution.
Critical weaknesses

None.

Questions for authors
  1. Clarification: I feel that Assumption 4.1 used in the Mathematical Insights section has not been sufficiently explained in the text. In particular, the choice of decaying covariance structure seems hard to interpret. Could the authors provide a more principled explanation of this choice ?
  2. Clarification: Figure 7 in the main text compares Truncated GAE and FT-GAE. The Final return plot for the LunarLander task shows the blue line (Truncated GAE) above the orange line (FT-GAE) for a range of values of λ\lambda. Could the authors explain this result ?
  3. Clarification: The paper claims novelty in the modification of the GAE estimator. A form of the GAE estimator very similar to FT-GAE appeared in the paper Modern Deep Reinforcement Learning (2019) Algorithms by Ivanov & D’yakonov (please see Section 5.4). Could the authors explain the difference of FT-GAE from this method ?
4 comments

Dear reviewer,

Thank you for your review and thoughtful questions. We appreciate your comments on the GridWorld experiment and the endpoint ablation and adressed your covariance point in Appendix C (changes indicated with red sidebars).

Q1. Covariance structure

The geometric decay in Assumption 4.1 has the same form as the covariance function of a stationary AR(1) process, a standard model for temporal dependence [1]. It captures our intuition that TD errors farther apart in time tend to be less correlated. We will clarify this point in the revised version of the paper. To check this intuition, we estimated TD-error covariances at different lags on LunarLander-v3. The estimates, averaged across training checkpoints, decreased with lag (Figure 17 in Appendix C). In Appendix C, we also show that the same qualitative conclusions hold under the equicorrelation model used by Daley et al. [2], which assumes constant correlation between TD-errors.

Q2. Figure 7

The tested λ\lambda values below 0.9 are too low for either method to learn successfully on LunarLander-v3. Truncated GAE achieves a higher mean final return than FT-GAE at some of these settings, but both methods fall short of the 200-point success threshold throughout this range [3]. A likely explanation is that these small λ\lambda values put too much weight on short backups, making both estimators more sensitive to bias in the learned critic. At the same λ\lambda, FT-GAE puts even more weight on short backups near rollout endpoints, which can amplify this bias and may explain its lower returns in parts of this range. Both methods achieve their highest mean final return in the sweep at λ=0.975\lambda = 0.975. At this setting, FT-GAE achieves a higher mean final return and a substantially higher area under the learning curve (AUC), indicating faster learning.

Q3: Related Work:

Thank you for pointing this out. We will cite it alongside the definition of FT-GAE in the revised version. The normalized geometric mixture in Section 5.4 is equivalent to FT-GAE (our Def. 3.1) when their deterministic NN is set to the potentially random number of available steps, τ−t\tau-t. However, the authors do not derive the backward recursion representation which makes the estimator so easily implementable with a simple change. Also, the authors call this mixture truncated GAE and do not mention the commonly used version of GAE, while we use this term for the truncated TD sum in the PPO paper [5, Eq. (11)], which follows the same endpoint weighting convention as the truncated λ\lambda-return [4, Sec. 12.3].

[1] P. J. Brockwell and R. A. Davis. Introduction to Time Series and Forecasting. 3rd ed., Springer, 2016.

[2] B. Daley, M. White, and M. Machado. Averaging n-step returns reduces variance in reinforcement learning. ICML, 2024.

[3] Gymnasium documentation

[4] R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction. 2018.

[5] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal Policy Optimization Algorithms. 2017.

Thank you for your reply. For Figure 7, can you provide experimental evidence or a theoretical proof showing that the worse performance of FT-GAE compared to Truncated GAE at small λ\lambda values is due to bias sensitivity ?

Regarding your reply to Q1, I understand the choice of covariance structure better now. I think that the covariance structure of AR(1) process is a reasonable theoretical assumption.

Thank you for the follow-up.

We can use the mathematical insight of Section 4.2, to show how critic bias can reverse the return-target MSE ranking at small λ\lambda. To isolate this effect, we hold the covariance model fixed and assume a constant critic bias ϵt=ϵ\epsilon_t =\epsilon (this allows to factorize it out of the return-target bias). At a fixed λ\lambda, the difference in average return-target MSE is given by

MSE‾trunc−MSE‾FT=(Vˉtrunc−VˉFT)+ε2(Btrunc−BFT),\overline{\mathrm{MSE}}_{\mathrm{trunc}} - \overline{\mathrm{MSE}}_{\mathrm{FT}} = (\bar{V}_{\mathrm{trunc}} - \bar{V}_{\mathrm{FT}}) + \varepsilon^2 (B_{\mathrm{trunc}} - B_{\mathrm{FT}}),

where Vˉtrunc−VˉFT>0\bar V_{\mathrm{trunc}}-\bar V_{\mathrm{FT}} >0 denotes the variance differences and Btrunc−BFT<0B_{\mathrm{trunc}}-B_{\mathrm{FT}}<0 the difference in squared bias at ε=1\varepsilon=1. Both terms only depend on λ\lambda through the estimators’ weights, and the critic bias enters the MSE difference only through ε2\varepsilon^2. The variance difference is positive because FT-GAE reduces variance, and the squared-bias difference is negative because truncated GAE gives more weight to the longest available backup (Corollary A.3), which discounts the critic error more heavily. Truncated GAE has lower MSE once FT-GAE’s additional squared bias exceeds its fixed variance saving.

The crossover thresholds for ∣ε∣|\varepsilon| are approximately 11.70, 12.10, 12.61, and 13.53 at λ\lambda values of 0.80, 0.85, 0.90, and 0.95, respectively. For example, at the same bias magnitude of ∣ε∣=12.3|\varepsilon|=12.3, truncated GAE has lower return-target MSE at λ=0.80\lambda=0.80 and 0.850.85, while FT-GAE has lower MSE at λ=0.90\lambda=0.90 and 0.950.95.

Thus, at the same critic-bias magnitude, FT-GAE’s additional squared bias can outweigh its variance advantage at the smaller tested λ\lambda values, while its variance advantage still dominates at the larger ones. This reversal in return-target MSE provides a theoretical mechanism for the relative underperformance of FT-GAE at smaller λ\lambda in Figure 7.

Thanks for your reply. I understand the proof that FT-GAE performs worse for small values of λ\lambda. I will keep the score 4 as before.

Reviewer_wfa1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper the authors propose an approach to perform the generalized advantage estimation for finite-time (FT-GAE). The main contribution of this paper is to use the normalized geometric weights over the truncated window (i.e. the available multistep), instead of assigning the remaining tail mass to the longest estimate. The authors argue that it mitigates the instability issue of tail.

Here, authors provide theoretical analysis under controlled bias-variance models, to analyze the estimation error near rollout endpoints. The authors also conduct empirical experiments on exact-reference GridWorld diagnostics, PPO experiments across 24 environments, and ablations covering constant-λ\lambda tuning, terminal rewards, actor/critic targets, and A2C.

Critical strengths

Practical improvement with minimal implementation cost. FT-GAE requires only a local change to the GAE recursion. Reported gains across several benchmark families, including after separate λ\lambda tuning and evaluation on fresh seeds, support its usefulness. Meaningful controlled diagnostics. The GridWorld evaluates both estimators on identical trajectories and critics against exact references.

Together with the termination and actor/critic ablations, this provides evidence beyond aggregate learning curves that endpoint weighting affects estimation and learning.

Critical weaknesses

Critical Weaknesses

  • Theory: limited coverage of random termination.

    • The bias–variance analysis assumes a fixed endpoint, while the implementation also uses random episode terminations. Outcome-dependent weights can introduce bias even with an exact critic. This limits how directly the theory explains gains in the motivating terminal-reward regime.
  • Experiments: insufficient isolation of the specific weighting mechanism.

    • Constant-λ\lambda sweeps do not distinguish FT-GAE from other endpoint-dependent schedules. Simple competing schedules are needed to establish the value of finite geometric normalization.
    • Appendix D matches average backup length at the maximum horizon, leaving potentially substantial differences near realized endpoints. Thus, shorter local backups remain an alternative explanation.
  • Writing and positioning: the precise contribution is insufficiently clarified.

    • The presentation under-develops FT-GAE’s interpretation as a particular endpoint-dependent λ\lambda schedule within the existing trace framework. Making this connection explicit would clarify whether the contribution lies primarily in the weighting rule, its statistical explanation, or its demonstrated practical benefits.
Questions for authors
  • Evidence request: How does FT-GAE compare with simple endpoint-dependent λ\lambda schedules under comparable tuning budgets?
  • Clarification: What additional bias arises at random terminations when V=VπV=V^\pi?
  • Evidence request: Do Appendix D’s gains persist when average backup length is matched at each realized remaining horizon?
2 comments

Dear reviewer,

thank you for your careful review and constructive suggestions. We have addressed most of your points in the revised manuscript (Appendix C,D,E; changes indicated with red sidebars) and invite you to take a look. Thanks a lot for helping us sharpen the article in these details! For your convenience a summary is given below.

Q2/ W1: Bias from outcome-dependent weights

Thanks for pointing this out. At a random termination, the realized endpoint can depend on the same outcomes that determine the TD errors. This can bias FT-GAE even with the exact critic. Using Proposition 3.2 we can calculate the bias as a discounted sum of conditional covariances between each TD error and its weight:

\mathbb{E}^{\pi}\bigl[\widehat{\mathbb{A}}_t^{\mathrm{FT},\lambda} \mid S_t,A_t\bigr]-\mathbb{A}^{\pi}(S_t,A_t) =\sum_{i=1}^{T-t-1}\gamma^i \operatorname{Cov}^{\pi}\left[ \lambda^i\frac{1-\lambda^{\tau-t-i}}{1-\lambda^{\tau-t}} \mathbf{1}_{\\{i \lt \tau-t\\}}, \delta_{t+i}\mid S_t,A_t \right]. \end{aligned}$$ We added the above result and a small MDP in which the bias can be computed in closed form in Section E. To measure its size in practice, we computed advantage estimates using $V^\pi$ for different state action pairs in the gridworld of section 4.1. and computed bias and variance. Averaged across different state, action pairs and policies, FT-GAE's estimated squared bias was ten times lower than its variance and 30 times lower than truncated GAE variance. Thus, the bias from random termination contributed little to FT-GAE's MSE in this experiment. Figure 3(c,d) also shows that FT-GAE has substantially lower RMSE than truncated GAE near rollout endpoints for both advantage estimates and return targets. Extending the analysis in Section 4.2 to random termination would require assumptions about how the endpoint and TD errors depend on each other. This would make the analysis considerably more complex, and its conclusions would depend on the specific model chosen. ## Q1/ W2 / W3: Trace viewpoint and schedule comparison Thank you for your suggestion. We further tested a linear endpoint-dependent schedule, which we call Lin-Trace, with local trace parameter $\lambda\min(1,(\tau-t-1)/w)$. The parameter increases linearly to $\lambda$ as the distance from the endpoint grows. We used the same hyperparameter tuning budget as in Section 5.2 and evaluated the selected Lin-Trace configuration on 30 fresh seeds on LunarLander. We also included PiLaR [1] as a compound-return baseline. | Estimator | Final Return (Mean ± SE) | | --- | ---: | | Truncated GAE | 252,17 ± 2,91 | | **FT-GAE** | **275,55 ± 1,01** | | Linear Schedule | 269,65 ± 1,28 | | PiLaR | 249,03 ± 4,11 | FT-GAE achieved the highest observed mean final return. Lin-Trace recovered about three quarters of the gap between Truncated GAE and FT-GAE, and PiLaR’s mean final return was close to that of Truncated GAE. The above experiment will be added to the ablations in section 5. In addition, we will make the trace interpretation of Proposition 3.2 more explicit in the revision. Normalizing the geometric weights over the available $k$-step advantage estimates yields a local trace parameter that depends on the realized remaining horizon $\tau-t$. We will relate this construction to prior work on adaptive traces of $\lambda$, e.g. [1–4]. We will also express Lin-Trace as a weighted mixture of the available $k$-step advantage estimates and compare the estimators’ trace schedules. We see designing and analyzing other schedules based on the realized endpoint $\tau$ an an interesting direction for future work. ## W2/ Q3: Random walk experiment We repeated the Appendix D experiment using each rollout's realized length $\tau$ to match the average backup length of the estimators. FT-GAE retained an RMSE advantage for long backups at large step sizes, while the gap was smaller than in the original Appendix D experiment. We report these results in Appendix D. [1] B. Daley, M. White, and M. Machado. Averaging n-step returns reduces variance in reinforcement learning. Proceedings of the 41st International Conference on Machine Learning. 2024. [2] M. Zhao, S. Luan, I. Porada, X.-W. Chang, and D. Precup. META-Learning State-based Eligibility Traces for More Sample-Efficient Policy Evaluation. AAMAS, 2020. [3] M. White and A. White. A Greedy Approach to Adapting the Trace Parameter for Temporal Difference Learning. AAMAS, 2016. [4] Y. Yue et al. VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks. 2025. [5] R. Deb, M. Gandhi, and S. Bhatnagar. Schedule Based Temporal Difference Algorithms. Allerton, 2022.

A brief update on Q1/W2/W3.

The endpoint-dependent schedule comparison discussed above is now in Section 5.6 of the revised manuscript. We extended the comparison with Lin-Trace and PiLaR to include new endpoint-dependent schedules that reduce the trace coefficient to λ/4\lambda/4 or λ/2\lambda/2 within a window before the endpoint. While all schedules achieve higher mean final returns than truncated GAE, FT-GAE still achieves the highest mean final return, with an estimated probability of improvement of at least 75% against each comparator. The changes are highlighted in the revised PDF.

We have also added a discussion of the trace viewpoint to Remark 3.4 and the related work section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.