- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
They paper proposes to use a learned discount factor rather than a constant, pre-specified discount factor, along with a training scheme for the learned discount factor.
Critical strengths
-
The idea is simple and straightforward.
-
The loss which trains the discount has terms that explicitly trade off between large and small discount sizes.
-
The idea of adaptive discount has nontrivial consequences, especially from a theoretical perspective, as gamma defines the learning objective. The authors establish several theoretical results that help alleviate resulting concerns.
-
Results seem positive.
Critical weaknesses
Evaluation and its presentation:
-
Most results present "reward" as the performance metric, and its not clear what this means. I presume this means empirical expected return of final policy at the end of training, which is a very fragile metric (RL algorithms are often unstable, and final policy means very little when looking at the learning curve). I think most tables can (and should) be replaced by learning curves.
-
Relatedly, many of the results in the paper are presented in tables. Tables are perhaps the single least-informative approach for evaluation-presentation and should be avoided whenever possible. Even if the best metric to present is a score, please choose presentations such as bar-charts which allow the reader to reason about the results visually.
-
When introducing a new learned component to replace what was previously a constant / hyperparameter, a natural suspicion is that the learned function simply tunes the HP better. The authors address this by comparing the mean-learned-discount (table 4) to the learned discount's performance (table 7). However, it seems that the range sweeped on for at least one of the combinations (humanoid, PPO), isnt the correct range that covers the learned mean gamma. These results should also be presented in learning curves with proper confidence bounds in my opinion, before this item could be viewed as addressed. In addition, adding additional parameters to the estimator (eg the learned discount) can simply perform better by encoding useful information in these parameters. So in principle, an experiment that adds these params to the actor / critic instead, would also be useful to verify that the contribution to performance does not come simply from larger DNNs.
-
The tables mark certain entries in bold (presumably the best performing). However, we usually specify exactly what is marked in bold, which is usualy the stat.-signif. best-performer. Please state what is marked in bold in the text (if it's there and I missed it, apologies), adhere to this convention (unless we can explain why we should deviate from it), and of course, only in the results that must be presented in tables - as opposed to the those that can be replaced by more informative choices (such as learning curves).
-
Trainig curves are referred to in Appendix G.3, but I can't seem to find them. Am I missing them, or are they perhaps actually missing?
-
The comparison includes several model free algorithms (SAC, PPO). However, at this moment (end of 2026), these are effectively legacy algorithms in my opinion, which limits the significance of the results.
Statistical significance:
-
The paper presents mean and 1 standard deviation over seeds. Standard deviation is not the appropriate stat. signif. metric to evaluate the certainty of the mean, and a proper confidence interval should be used instead (e.g. +-2-SEM for a 95% Gaussian CI).
-
The paper uses 5 seeds for most results. In my experience, 5 seeds are well below the sufficient number of seeds in cont. control (humanoid, ant). In fact, anything below 20 is barely stable, in my experience with SAC in these domains. I suspect this will become visually clear when comparing learning curves for all algorithm variants on the same domains with a proper confidence bound.
Motivation:
-
The motivation included in the introduction is (just barely) sufficient to motivate changes to the discount, in my opinion. However, I think it will be greatly strengthened by demostrating a concrete, measurable problem with constant discounts, or showing that a constant discount explicitly results in unnecessarily large value errors (see next), that effectively "necessasitate" a contribution such as the one proposed by the paper.
-
Part of the motivation used by the paper is a variance / bias trade off. I like it, and I think I understand where the authors are going with it. However, I also think it's insufficiently developed and not made sufficiently precise. Specifically, in TD learning, there are multiple sources of variance and multiple sources of bias. If we assume that we measure error with respect to the true target , then transition uncertainty and reward uncertainty will contribute to the variance (which isnt reduced by a discount), and bootstrap-error will contribute both to variance and bias, which can be reduced by a smaller discount. I think that this analysis should be made more precise, and respectively, what is exactly traded off with a larger / smaller discount factor should be made more precise.
-
The proposed loss is very engineered (regularization terms, etc.), and is effectively a heuristic. Since (some of) the motivation is a bias / variance trade off, it also seems that it is perhaps more natural to set the discount as a function of uncertainty rather than simply learned based on a hueristic loss.
Theoretical analysis:
I like the theoretical analysis included, and I also think that the theoretical consequences of non-constant discount are so wide it will be unreasonable to cover all of them in a single paper (especially in a paper whose objective is a practical contribution to deep RL). However, there are two major theoretical questions left open that I think should be addressed:
-
The paper analyzes a constant state-dependent discount, but proposes a non-stationary state dependent discount (unless I misunderstand, in which case please correct me). I don't think this gap is trivial, and I think this greatly weakens the existing results' contribution to soundness.
-
The main consequence of changing the discount (from my perspective), is that it fundementally redefines the optimal policy in the MDP. The paper does not address the consequences modifying gamma has on the optimal policy, which I also think is a major theoertical question that remains unanswered. That is - the paper establishes convergence, but - convergence to what? It's certainly not the optimal policy defined by gamma = some constant, is it?
In addition, the theoretical analysis is for soft-RL, which I think weakens it significantly. Soft-RL is but a hueristic approach for RL, and does not even actually solve for the true optimal policy. I think that theoretical analysis on actual-RL will be much stronger. For example - are there properties of soft-RL that are necessary for the analysis, precluding a similar analysis for standard-RL?
Significance and soundness:
As a result of the above, it is difficult for me to establish the significance and the soundness of the contributions. This will be greatly alleviated with learning curves with a proper CI (significance), and better motivation, more comprehensive analysis (soundness), or stronger motivation for the loss or the learned discount.
Questions for authors
Please see the weaknesses section.
3 comments
Q5. Making the bias–variance / bootstrap-error trade-off precise and motivating the learned RC loss over heuristic uncertainty-based discounting (Motivation Weaknesses 1, 2, and 3).
We agree with your decomposition that does not reduce one-step reward or transition variance at step , and instead specifically scales downstream bootstrap error (trading truncation bias against propagated approximation bias and estimation variance); we will sharpen Section 1's formulation accordingly. However, setting directly as a function of critic uncertainty is precisely the SAC/PPO-Uncertainty baseline we implemented and tested (Appendix C, Eq. 52), which AdaGamma consistently outperforms across all tasks in Table 1 because ensemble disagreement only measures epistemic spread across heads rather than actual multi-step return fidelity; moreover, our leave-one-out ablation in Table 3 shows that pure RC none (with all three regularizers set to zero) matches the full loss (27.20 ± 0.19 vs. 27.17 ± 0.12), demonstrating that our core objective does not rely on heuristic regularization engineering.
Q6. Theoretical gap between fixed and changing discounts, effects on the optimal policy, and applicability beyond Soft-RL.
Our analysis addresses these questions within its stated tabular scope. First, Corollary 1 (Appendix B.7, Eqs. 47–48) bounds the tracking error of exact fixed-policy evaluation under changing discount functions in terms of their maximum change between successive updates; it does not establish convergence of joint actor–critic–discount training. Second, Theorem 1 establishes convergence to the soft-optimal policy for the specified state-dependent discount function, not the optimal policy for the original constant-discount or undiscounted objective. Theorem 2 bounds the value discrepancy for a fixed policy under different discounts, but does not guarantee preservation of the optimal policy. When evaluation uses undiscounted return, both fixed and learned discounts below one serve as training surrogates. Third, entropy regularization is not essential to the contraction or value-sensitivity results: removing entropy and replacing soft maximization with ordinary maximization yields the corresponding standard tabular results, with the sensitivity constant reduced to the reward bound. However, the uniqueness and continuity arguments for the soft-greedy policy do not transfer unchanged because ordinary greedy policies can have ties. We will clarify these distinctions.
Q7. Missing Bold notation.
The bold entries indicate the best reported mean within each comparison group, not statistically significant superiority. We will state this explicitly in the captions.
I thank the authors for their rebuttal. I have considered the replies and found they have not meaningfully resolved my concerns and I will maintain my original rating.
Specifically:
Empirical results:
Thank you for correcting me and pointing out that Figures 7 and 8 are not in fact missing. A good fix for that problem is to refer to them from the text with \ref, which will link irrespective of where they are placed in the paper.
I think Figures 7 and 8 demonstrate very clearly the fragility of final-performance results. They appear to show that AdaGamma has no statistically significant benefit in any domain tested (In Figures 7 and 8), and further exacerbates my questioning of the value of a table of final performance results presented in table 1.
Theory:
I like the argument made by the authors that "whenever we evaluate with undiscounted returns, every discount is as heuristic as any other". I think it's a good point, which I think should be made in the paper. Nonetheless, the majority of RL theory is done under discounted returns, and thus, a non-stationary, or non-constant in state space discounts and their effect of the optimal policy remain an important question in my opinion.
Please make sure to include the theoretical properties pointed in the rebuttal in the paper, in my opinion they will improve the paper and make it easier to accept. Specifically, the limitations (or not-limitations) of soft-RL-based analysis.
Presentation:
When a reader looks at a performance result, the question that should be answered with minimal effort is "does the contribution produce (statistically) significant benefit over the relevant baselines?". This question is only answered by emphasizing the statistically significant best performers. It is good that the authors clarify their choice of presentation, but I will find it lacking until it addresses the core question of being the best fit to the question asked and answered by the experiments.
First, the training curves referenced in Appendix G.3 are not missing from the submission: due to LaTeX float placement, Figures 7 and 8 were pushed to page 26 (four pages after the Appendix G.3 heading on page 22), and they show full learning trajectories across episodes confirming that AdaGamma's gains hold consistently throughout training rather than only at the final step. Throughout the tables, "Reward" denotes undiscounted episodic evaluation return averaged over evaluation rollouts (evaluated every 104 steps) and boldface marks the highest sample mean; in the revision, we will fix the float placement and cross-references, move primary learning curves and visual bar/interval plots into the main text, and state the metric and bolding conventions explicitly.
Thank you for catching the range mismatch on PPO Humanoid-v4: Table 7 inadvertently evaluated
[0.9950, 0.9990, 0.9999]instead of centering directly on PPO's0.9846learned mean from Table 4—although0.9800(356.19, Table 8),0.9900(411.61, Table 1), and0.9950(456.19, Table 7) closely bracket0.9846and all fall belowPPO-AdaGamma(476.51), and we will check and add the exact0.9846run with learning curves. Regarding parameter count, γϕ(s) is never used by the actor at evaluation time and outputs only a single bounded scalar γϕ(s)∈[γmin,γmax] multiplying the scalar bootstrap target, which cannot serve as extra feature capacity—as directly verified bySAC-CrossValidate, which uses the exact same two-layer 256-unit γϕ network yet underperformsSAC-Fixed-γ(616.25vs.5370.80on Humanoid-v4); we can also check a parameter-matched wider critic control.Our contribution targets a fundamental Bellman/GAE bootstrapping failure mode (self-referential TD collapse when learning γϕ(s)) rather than claiming a new domain-specific SOTA architecture, and SAC and PPO remain the canonical off-policy and on-policy backbones in both academic continuous control and industrial deployments—including our live JD Logistics production recommender serving N≈106 users (Section 4.3), alongside our TRPO and DDPG extensions in Appendix H. Because AdaGamma interfaces solely through the scalar bootstrap discount in value targets (Section 3.2), it integrates directly into newer regularized or ensemble actor–critic variants (such as REDQ or CrossQ) without structural modification.
Because n=5 is explicitly stated, a 95% Gaussian confidence interval (±1.96SEM=±51.96SD≈±0.88SD) is strictly narrower than the ±1SD intervals we reported; under ±2SEM, key improvements such as SAC on Humanoid-v4 (
6907.99 ± 19.46vs.5370.80 ± 10.05Fixed and6606.08 ± 53.39Uncertainty) and PPO on SafetyPointGoal1-v0 (26.31 ± 0.81vs.22.58 ± 1.20Fixed) remain separated by large margins, further supported by statistically significant four-week A/B gains over N≈106 users. Nevertheless, we agree that displaying ±2SEM (or bootstrap 95% CIs) directly on learning curves and running more seeds on high-variance tasks such as Ant-v4 will strengthen statistical certainty, and we will update our uncertainty bands and check expanded seed runs on the core benchmarks.