- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper identifies a theory vs. practice mismatch: episode rollouts in real-world RL applications trivially can never reach infinite time. Therefore, the author's propose FT-GAE, which aims to heal estimations errors near rollout endpoints. The proposed method is tested on 24 different benchmark environments from 5 suits to find that it is most beneficial in scenarios where the most important information is in endpoint signals.
Critical strengths
-
As it is a minor change to the recursion in the GAE equation, the proposed method is simple, practical and of identical computational cost.
-
The experiments are thorough and extensive. The used environments are quite diverse. Explicitely testing that retuning does not achieve just similar improvements is something that is oftentimes forgotten and therefore genuinly strengthens the paper's claim.
Critical weaknesses
- GAE is used as baseline and only comparator. An obvious alternative might be to lower on the last few steps before an endpoint. Another alternative is Daley et al (2024), as the authors say to have followed its "TD-error decomposition of compound returns" but replace the "equicorrelation model". That replacement itself is never tested.
Questions for authors
-
I have found Sharma et al. (2017) "Learning to Mix n-Step Returns: Generalizing lambda-Returns for Deep Reinforcement Learning". Your variant looks like a fixed parameter version of what they used. If that is true, this should be at least discussed somewhere in the paper.
-
Freeway has no terminations as I undestand it. So where does the -sweep gain happen? What explains the gain where no informative terminal signal is present?
-
You are open about the limitations: only GAE as baseline & FT-GAE not always superior. My main reason for voting "weak" instead of "clear accept" is those missing comparisons (see weaknesses). As the principle of FT-GAE is so simple and cheap, it would be perfect to understand when exactly it helps and when it does not. What were the reasons to not include more comparators?
3 comments
Thank you for the detailed response. The connection to Sharma et al. and the covariance-model question seem resolved to me, and I appreciate the candid answers on Freeway and on when truncated GAE helps. The DoorKey crossover seems like a useful practical insight.
I have a few follow-up questions:
- About Lin-Trace: It seems that a simple linear endpoint-dependent schedule recovers most of the gain on LunarLander. Would you agree that this suggests much of the benefit comes from relying less on long returns near endpoints in general, rather than from the specific renormalized weighting?
- How many seeds were used for the new LunarLander comparison, and what are the confidence intervals? It is not clear to me whether the remaining gap between FT-GAE and Lin-Trace is meaningful. The same question applies to the DoorKey 8×8 table, where several differences seem small relative to typical seed variance.
- Freeway. From your answer, it seems the Freeway gain is mainly robustness to high λ, since the curves nearly overlap at the best λ, right? Does this also apply to the default-λ results more generally?
- Revision. As far as I can see, the revised PDF contains the additions in Appendices C–E, but not yet the new baselines, the DoorKey experiment or the discussion of Sharma et al. Do you plan to add these before the end of the discussion period, and could all changed sentences be highlighted in color as well? This would really help in reviewing the PDF. Thanks.
Thanks for engaging in the discussion! We have updated the main text and highlighted the changes in the revised PDF.
Q4: Revsion:
- We now cite Sharma et al. (2017) in the related work section and explain how FT-GAE fits into their weighted-return framework in Remark 3.2.
- In Section 5.6, we report the LunarLander-v3 comparison with Lin-Trace and PiLaR and extend it to include two additional endpoint-dependent schedules. Both achieved mean final returns between those of truncated GAE and FT-GAE.
- In Section 5.7, we report the DoorKey 8×8 experiment.
Q1: linear endpoint-dependent schedule:
Yes, we agree that the LunarLander-v3 results suggest that much of the benefit comes from placing less weight on long returns near endpoints. At a true episode termination, truncated GAE assigns the remaining mixture weight to the Monte Carlo return. In LunarLander-v3, that return includes a terminal reward of +100 for landing or −100 for crashing. When the outcome is uncertain, this 200-point difference can make the Monte Carlo return highly variable, and giving it extra weight can increase the estimators variance.
FT-GAE was explicitley designed to address tail-mass collapse. It renormalizes the geometric mixture over the available -step returns, locally giving shorter, bootstrapped returns more weight. The backward recursion derived from this mixture (Proposition 3.3) yields smaller recursion coefficients near endpoints. This motivated the endpoint-dependent schedules we tested. While FT-GAE retains as its only tuning parameter. The schedules also require tuning the length of the window over which the coefficient is reduced. In our experiment the endpoint-dependent schedules recovered only part of the gap between truncated GAE and FT-GAE. FT-GAE achieved the highest mean final return, and its confidence interval lay entirely above those of all other methods in the comparison (see Q2).
Q2: Experiment details
For LunarLander-v3, we used five seeds per hyperparameter configuration and then trained each method with its selected configuration on 30 fresh seeds. The comparison now includes two additional Step-Trace variants, which set the local trace coefficient to or within a window of steps before the endpoint. The table now includes 95% CI.
| Method | Mean final return [95% CI] |
|---|---|
| Truncated GAE | 252.17 [246.24, 257.44] |
| FT-GAE | 275.55 [273.45, 277.33] |
| Lin-Trace | 269.65 [267.10, 272.03] |
| Step-Trace (0.25) | 266.89 [262.11, 271.08] |
| Step-Trace (0.5) | 269.29 [266.36, 271.87] |
| PiLaR | 249.03 [240.40, 256.15] |
Section 5.6 also reports pairwise estimates of the probability of improvement with 95% confidence intervals. The point estimate for FT-GAE was at least 75% against each comparator.
For the door-key experiment, we trained both methods on 15 additional seeds and now report results for 30 seeds per method in Section 5.7 and below, including 95% bootstrap confidence intervals.
| Training steps | Truncated GAE | FT-GAE | Relative change vs. Truncated GAE |
|---|---|---|---|
| 100k | 0.0573 [0.0169, 0.1081] | 0.0026 [0.0002, 0.0058] | −95.4% [−99.4%, −86.2%] |
| 500k | 0.2591 [0.1292, 0.4205] | 0.1448 [0.0399, 0.2712] | −44.1% [−84.9%, +50.8%] |
| 1M | 0.3161 [0.1617, 0.4780] | 0.2428 [0.1059, 0.3958] | −23.2% [−69.7%, +79.4%] |
| 2M | 0.3847 [0.2232, 0.5511] | 0.4054 [0.2423, 0.5748] | +5.4% [−45.8%, +111.9%] |
| 5M | 0.5162 [0.3516, 0.6834] | 0.6150 [0.4528, 0.7773] | +19.1% [−20.0%, +81.6%] |
Q3: Freeway:
Yes, on Freeway the main benefit is greater robustness at high . The learning curves nearly overlap when each estimator uses its best-performing . But this is not true in general for the default results. In Section 5.2, we test whether the gains observed at the default persist after retuning. The results of Table 1 shows that FT-GAE still siginficantly improves over truncaeted GAE in Box2D, MinAtar and Minigrid environments. In all three suites, the 95% confidence intervals for the probability of improvement lie entirely above 0.5
We hope the added comparisons address the main concern you raised in your initial review, and we would appreciate it if you would reconsider your rating.
Dear reviewer,
Thank you for your careful review! We address your comments below and in Appendices C–E (see red bars) and invite you to take a look.
W1 / Q3: Additional baselines:
Thank you for your suggestion. We now also compare a linear endpoint-dependent schedule (Lin-Trace) and a PPO adaptation of PiLaR [1] on LunarLander-v3. Mean final returns were 252.17 for Truncated GAE, 275.55 for FT-GAE, 269.65 for Lin-Trace, and 249.03 for PiLaR. Lin-Trace recovered about three quarters of the observed gap between FT-GAE and truncated GAE. Please see our response to Reviewer wfa1 for the schedule definition. We will report the new comparison as an additional ablation in Section 5.
W2: Covariance model:
The mathematical insights can analogously use the equicorrelation assumption of [1]. We added results for the equicorrelation model in Appendix C and observe that the pattern stays the same. We chose the covariance structure in Assumption 4.1 because we expected TD errors farther apart in time to be less correlated. Our geometric decay has the same form as the covariance function of a stationary AR(1) process, a standard model for temporal dependence [2]. We also estimated TD-error covariances at different lags to provide empirical motivation for modeling weaker dependence at longer lags. The estimates averaged across training checkpoints decreased with greater lags (Figure 17).
Q1: Related work:
Thank you for pointing out this connection. We agree that FT-GAE can be viewed as a fixed weighting rule within the weighted-return framework [3] by applying their (Eq. 5) to the fixed logits c(st+i,t+i)=(t+i−1)logλ over the available horizons i=1,…,τ−t. We will cite [3] and clarify this relationship along with the other related literature.
Q2: Freeway:
In Freeway, the agent receives a reward only when it completes a crossing. Even when a PPO rollout contains no reward, a critic that has learned to predict future crossings can still provide a learning signal through bootstrapping. At high λ, FT-GAE places more weight on shorter, bootstrapped returns than Truncated GAE near the rollout boundary. This explains its advantage at high values of λ. Note that for the best λ results, the two learning curves nearly overlap.
Q3: When might Truncated GAE help?
At a true episode termination, Truncated GAE assigns the remaining mixture weight to the Monte Carlo return. When successes are rare and the critic fails to predict the delayed reward accurately (e.g. in early training when first observing signals), this extra weight can help credit earlier actions in a successful trajectory. This aligns with prior work which found that Monte Carlo updates can be beneficial in tasks with sparse or delayed rewards [4, 5].
Motivated by [6], we compared Truncated GAE and FT-GAE on MiniGrid DoorKey. Increasing the grid side length from 5 to 8 gives the agent more locations to search the key before it can open the door and reach the goal to observe its only reward. Thus, a larger grid makes both exploration and temporal credit assignment harder.
Results are averaged over 15 seeds. The first row reports results already reported in the initial version of the paper, where both methods learned within the standard 100,000-step budget, and FT-GAE achieved a higher mean return. On 8×8, both earned almost no reward at that budget. In longer 8×8 experiment, Truncated GAE had the higher mean return up to 1 million steps. FT-GAE had the higher mean at the reported checkpoints from 2 million steps onward. The change in ordering is consistent with a trade-off that shifts during training. Early on, when a successful rollout provides a reward that the critic predicts poorly, tail-mass collapse gives the observed Monte Carlo return more weight and can strengthen the learning signal for earlier actions in the same rollout segment. As the critic learns to predict future success, shorter bootstrapped returns can convey that signal with lower variance, making FT-GAE's weighting more useful. We will add above experiment to the ablations in the revised version.
[1] Daley et al. (2024). Averaging n-Step Returns Reduces Variance in RL.
[2] Brockwell & Davis (2016). Introduction to Time Series and Forecasting.
[3] Sharma et al. (2017). Learning to Mix n-Step Returns: Generalizing λ-Returns for Deep Reinforcement Learning.
[4] Ostrovski et al. (2017). Count-Based Exploration with Neural Density Models.
[5] Amiranashvili et al. (2018). TD or not TD: Analyzing the Role of Temporal Differencing in Deep Reinforcement Learning.
[6] Kayal et al. (2025). The impact of intrinsic rewards on exploration in Reinforcement Learning.