acceptodds
Under review as a conference paper at ICLR 2027

AdaGamma: State-Dependent Discounting for Temporal Adaptation in Reinforcement Learning

Abstract

The discount factor in reinforcement learning controls both the effective planning horizon and the strength of bootstrapping, yet most deep RL methods use a single fixed value across all states. State-dependent discounting is a natural way to adapt temporal credit assignment to local state structure, but naively learning a discount function inside deep actor–critic methods creates a self-referential target: the learned discount scales the same bootstrap term against which it is trained. Consequently, TD-error-based objectives can admit a degenerate shortcut, reducing target dependence on downstream value estimates instead of learning meaningful temporal structure. We propose AdaGamma, a practical framework for state-dependent discounting in deep actor–critic reinforcement learning. AdaGamma learns a bounded discount function \(\gamma_\phi(s)\) and trains it using a return-consistency objective that matches a one-step adaptive bootstrap to a multi-step return computed with a lagged, stop-gradient reference discount. This removes the direct target manipulation path and encourages the discount to encode state-dependent value propagation. We analyze the Bellman operator induced by a fixed state-dependent discount in finite tabular MDPs, proving contraction, exact soft policy iteration, and Lipschitz sensitivity under standard boundedness assumptions. Empirically, AdaGamma integrates into SAC and PPO, avoids TD-collapse, improves continuous-control benchmarks, and achieves statistically significant gains in a four-week online A/B test on the JD Logistics platform.

Primary area:
reinforcement learning
Keywords:
RL, SAC, recommendation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_mue9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

They paper proposes to use a learned discount factor rather than a constant, pre-specified discount factor, along with a training scheme for the learned discount factor.

Critical strengths
  1. The idea is simple and straightforward.

  2. The loss which trains the discount has terms that explicitly trade off between large and small discount sizes.

  3. The idea of adaptive discount has nontrivial consequences, especially from a theoretical perspective, as gamma defines the learning objective. The authors establish several theoretical results that help alleviate resulting concerns.

  4. Results seem positive.

Critical weaknesses

Evaluation and its presentation:

  1. Most results present "reward" as the performance metric, and its not clear what this means. I presume this means empirical expected return of final policy at the end of training, which is a very fragile metric (RL algorithms are often unstable, and final policy means very little when looking at the learning curve). I think most tables can (and should) be replaced by learning curves.

  2. Relatedly, many of the results in the paper are presented in tables. Tables are perhaps the single least-informative approach for evaluation-presentation and should be avoided whenever possible. Even if the best metric to present is a score, please choose presentations such as bar-charts which allow the reader to reason about the results visually.

  3. When introducing a new learned component to replace what was previously a constant / hyperparameter, a natural suspicion is that the learned function simply tunes the HP better. The authors address this by comparing the mean-learned-discount (table 4) to the learned discount's performance (table 7). However, it seems that the range sweeped on for at least one of the combinations (humanoid, PPO), isnt the correct range that covers the learned mean gamma. These results should also be presented in learning curves with proper confidence bounds in my opinion, before this item could be viewed as addressed. In addition, adding additional parameters to the estimator (eg the learned discount) can simply perform better by encoding useful information in these parameters. So in principle, an experiment that adds these params to the actor / critic instead, would also be useful to verify that the contribution to performance does not come simply from larger DNNs.

  4. The tables mark certain entries in bold (presumably the best performing). However, we usually specify exactly what is marked in bold, which is usualy the stat.-signif. best-performer. Please state what is marked in bold in the text (if it's there and I missed it, apologies), adhere to this convention (unless we can explain why we should deviate from it), and of course, only in the results that must be presented in tables - as opposed to the those that can be replaced by more informative choices (such as learning curves).

  5. Trainig curves are referred to in Appendix G.3, but I can't seem to find them. Am I missing them, or are they perhaps actually missing?

  6. The comparison includes several model free algorithms (SAC, PPO). However, at this moment (end of 2026), these are effectively legacy algorithms in my opinion, which limits the significance of the results.

Statistical significance:

  1. The paper presents mean and 1 standard deviation over seeds. Standard deviation is not the appropriate stat. signif. metric to evaluate the certainty of the mean, and a proper confidence interval should be used instead (e.g. +-2-SEM for a 95% Gaussian CI).

  2. The paper uses 5 seeds for most results. In my experience, 5 seeds are well below the sufficient number of seeds in cont. control (humanoid, ant). In fact, anything below 20 is barely stable, in my experience with SAC in these domains. I suspect this will become visually clear when comparing learning curves for all algorithm variants on the same domains with a proper confidence bound.

Motivation:

  1. The motivation included in the introduction is (just barely) sufficient to motivate changes to the discount, in my opinion. However, I think it will be greatly strengthened by demostrating a concrete, measurable problem with constant discounts, or showing that a constant discount explicitly results in unnecessarily large value errors (see next), that effectively "necessasitate" a contribution such as the one proposed by the paper.

  2. Part of the motivation used by the paper is a variance / bias trade off. I like it, and I think I understand where the authors are going with it. However, I also think it's insufficiently developed and not made sufficiently precise. Specifically, in TD learning, there are multiple sources of variance and multiple sources of bias. If we assume that we measure error with respect to the true target Vπ(s)V^\pi(s), then transition uncertainty and reward uncertainty will contribute to the variance (which isnt reduced by a discount), and bootstrap-error ∣vϕ(s′)−Vπ(s′)∣|v_\phi(s') - V^\pi(s')| will contribute both to variance and bias, which can be reduced by a smaller discount. I think that this analysis should be made more precise, and respectively, what is exactly traded off with a larger / smaller discount factor should be made more precise.

  3. The proposed loss is very engineered (regularization terms, etc.), and is effectively a heuristic. Since (some of) the motivation is a bias / variance trade off, it also seems that it is perhaps more natural to set the discount as a function of uncertainty rather than simply learned based on a hueristic loss.

Theoretical analysis:

I like the theoretical analysis included, and I also think that the theoretical consequences of non-constant discount are so wide it will be unreasonable to cover all of them in a single paper (especially in a paper whose objective is a practical contribution to deep RL). However, there are two major theoretical questions left open that I think should be addressed:

  1. The paper analyzes a constant state-dependent discount, but proposes a non-stationary state dependent discount (unless I misunderstand, in which case please correct me). I don't think this gap is trivial, and I think this greatly weakens the existing results' contribution to soundness.

  2. The main consequence of changing the discount (from my perspective), is that it fundementally redefines the optimal policy in the MDP. The paper does not address the consequences modifying gamma has on the optimal policy, which I also think is a major theoertical question that remains unanswered. That is - the paper establishes convergence, but - convergence to what? It's certainly not the optimal policy defined by gamma = some constant, is it?

In addition, the theoretical analysis is for soft-RL, which I think weakens it significantly. Soft-RL is but a hueristic approach for RL, and does not even actually solve for the true optimal policy. I think that theoretical analysis on actual-RL will be much stronger. For example - are there properties of soft-RL that are necessary for the analysis, precluding a similar analysis for standard-RL?

Significance and soundness:

As a result of the above, it is difficult for me to establish the significance and the soundness of the contributions. This will be greatly alleviated with learning curves with a proper CI (significance), and better motivation, more comprehensive analysis (soundness), or stronger motivation for the loss or the learned discount.

Questions for authors

Please see the weaknesses section.

3 comments

Q1. Definition of the "Reward" metric, reliance on tables over visual plots, boldface convention, and location of the Appendix G.3 training curves (Evaluation Weaknesses 1, 2, 4, and 5).

First, the training curves referenced in Appendix G.3 are not missing from the submission: due to LaTeX float placement, Figures 7 and 8 were pushed to page 26 (four pages after the Appendix G.3 heading on page 22), and they show full learning trajectories across episodes confirming that AdaGamma's gains hold consistently throughout training rather than only at the final step. Throughout the tables, "Reward" denotes undiscounted episodic evaluation return averaged over evaluation rollouts (evaluated every 10410^4 steps) and boldface marks the highest sample mean; in the revision, we will fix the float placement and cross-references, move primary learning curves and visual bar/interval plots into the main text, and state the metric and bolding conventions explicitly.

Q2. Whether gains come from better hyperparameter tuning or extra network parameters, and the PPO Humanoid-v4 fixed-γ\gamma sweep range in Table 7 (Evaluation Weakness 3).

Thank you for catching the range mismatch on PPO Humanoid-v4: Table 7 inadvertently evaluated [0.9950, 0.9990, 0.9999] instead of centering directly on PPO's 0.9846 learned mean from Table 4—although 0.9800 (356.19, Table 8), 0.9900 (411.61, Table 1), and 0.9950 (456.19, Table 7) closely bracket 0.9846 and all fall below PPO-AdaGamma (476.51), and we will check and add the exact 0.9846 run with learning curves. Regarding parameter count, γϕ(s)\gamma_\phi(s) is never used by the actor at evaluation time and outputs only a single bounded scalar γϕ(s)∈[γmin⁡,γmax⁡]\gamma_\phi(s) \in [\gamma_{\min}, \gamma_{\max}] multiplying the scalar bootstrap target, which cannot serve as extra feature capacity—as directly verified by SAC-CrossValidate, which uses the exact same two-layer 256-unit γϕ\gamma_\phi network yet underperforms SAC-Fixed-γ (616.25 vs. 5370.80 on Humanoid-v4); we can also check a parameter-matched wider critic control.

Q3. Significance of evaluating on SAC and PPO as model-free baselines in 2026 (Evaluation Weakness 6).

Our contribution targets a fundamental Bellman/GAE bootstrapping failure mode (self-referential TD collapse when learning γϕ(s)\gamma_\phi(s)) rather than claiming a new domain-specific SOTA architecture, and SAC and PPO remain the canonical off-policy and on-policy backbones in both academic continuous control and industrial deployments—including our live JD Logistics production recommender serving N≈106N \approx 10^6 users (Section 4.3), alongside our TRPO and DDPG extensions in Appendix H. Because AdaGamma interfaces solely through the scalar bootstrap discount in value targets (Section 3.2), it integrates directly into newer regularized or ensemble actor–critic variants (such as REDQ or CrossQ) without structural modification.

Q4. Reporting standard deviation vs. confidence intervals (±2 SEM\pm 2\text{ SEM}) and using 5 random seeds in continuous control (Statistical Significance Weaknesses 1 and 2).

Because n=5n = 5 is explicitly stated, a 95% Gaussian confidence interval (±1.96 SEM=±1.965 SD≈±0.88 SD\pm 1.96\,\text{SEM} = \pm \frac{1.96}{\sqrt{5}}\,\text{SD} \approx \pm 0.88\,\text{SD}) is strictly narrower than the ±1 SD\pm 1\,\text{SD} intervals we reported; under ±2 SEM\pm 2\,\text{SEM}, key improvements such as SAC on Humanoid-v4 (6907.99 ± 19.46 vs. 5370.80 ± 10.05 Fixed and 6606.08 ± 53.39 Uncertainty) and PPO on SafetyPointGoal1-v0 (26.31 ± 0.81 vs. 22.58 ± 1.20 Fixed) remain separated by large margins, further supported by statistically significant four-week A/B gains over N≈106N \approx 10^6 users. Nevertheless, we agree that displaying ±2 SEM\pm 2\,\text{SEM} (or bootstrap 95% CIs) directly on learning curves and running more seeds on high-variance tasks such as Ant-v4 will strengthen statistical certainty, and we will update our uncertainty bands and check expanded seed runs on the core benchmarks.

Q5. Making the bias–variance / bootstrap-error trade-off precise and motivating the learned RC loss over heuristic uncertainty-based discounting (Motivation Weaknesses 1, 2, and 3).

We agree with your decomposition that γ(st)\gamma(s_t) does not reduce one-step reward or transition variance at step tt, and instead specifically scales downstream bootstrap error Vθ(st+1)−Vπ(st+1)V_\theta(s_{t+1}) - V^\pi(s_{t+1}) (trading truncation bias against propagated approximation bias and estimation variance); we will sharpen Section 1's formulation accordingly. However, setting γ(s)\gamma(s) directly as a function of critic uncertainty is precisely the SAC/PPO-Uncertainty baseline we implemented and tested (Appendix C, Eq. 52), which AdaGamma consistently outperforms across all tasks in Table 1 because ensemble disagreement only measures epistemic spread across heads rather than actual multi-step return fidelity; moreover, our leave-one-out ablation in Table 3 shows that pure RC none (with all three regularizers set to zero) matches the full loss (27.20 ± 0.19 vs. 27.17 ± 0.12), demonstrating that our core objective does not rely on heuristic regularization engineering.

Q6. Theoretical gap between fixed and changing discounts, effects on the optimal policy, and applicability beyond Soft-RL.

Our analysis addresses these questions within its stated tabular scope. First, Corollary 1 (Appendix B.7, Eqs. 47–48) bounds the tracking error of exact fixed-policy evaluation under changing discount functions in terms of their maximum change between successive updates; it does not establish convergence of joint actor–critic–discount training. Second, Theorem 1 establishes convergence to the soft-optimal policy for the specified state-dependent discount function, not the optimal policy for the original constant-discount or undiscounted objective. Theorem 2 bounds the value discrepancy for a fixed policy under different discounts, but does not guarantee preservation of the optimal policy. When evaluation uses undiscounted return, both fixed and learned discounts below one serve as training surrogates. Third, entropy regularization is not essential to the contraction or value-sensitivity results: removing entropy and replacing soft maximization with ordinary maximization yields the corresponding standard tabular results, with the sensitivity constant reduced to the reward bound. However, the uniqueness and continuity arguments for the soft-greedy policy do not transfer unchanged because ordinary greedy policies can have ties. We will clarify these distinctions.

Q7. Missing Bold notation.

The bold entries indicate the best reported mean within each comparison group, not statistically significant superiority. We will state this explicitly in the captions.

I thank the authors for their rebuttal. I have considered the replies and found they have not meaningfully resolved my concerns and I will maintain my original rating.

Specifically:

Empirical results:

Thank you for correcting me and pointing out that Figures 7 and 8 are not in fact missing. A good fix for that problem is to refer to them from the text with \ref, which will link irrespective of where they are placed in the paper.

I think Figures 7 and 8 demonstrate very clearly the fragility of final-performance results. They appear to show that AdaGamma has no statistically significant benefit in any domain tested (In Figures 7 and 8), and further exacerbates my questioning of the value of a table of final performance results presented in table 1.

Theory:

I like the argument made by the authors that "whenever we evaluate with undiscounted returns, every discount is as heuristic as any other". I think it's a good point, which I think should be made in the paper. Nonetheless, the majority of RL theory is done under discounted returns, and thus, a non-stationary, or non-constant in state space discounts and their effect of the optimal policy remain an important question in my opinion.

Please make sure to include the theoretical properties pointed in the rebuttal in the paper, in my opinion they will improve the paper and make it easier to accept. Specifically, the limitations (or not-limitations) of soft-RL-based analysis.

Presentation:

When a reader looks at a performance result, the question that should be answered with minimal effort is "does the contribution produce (statistically) significant benefit over the relevant baselines?". This question is only answered by emphasizing the statistically significant best performers. It is good that the authors clarify their choice of presentation, but I will find it lacking until it addresses the core question of being the best fit to the question asked and answered by the experiments.

Reviewer_ajoPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The discounting factor plays a vital role in controlling the credit assignment time horizon, trading variance for bias (i.e., credit assignment over further away steps implies lower bias, but higher variance and vice-versa). Most, RL algorithms applies a single fixed discounting factor across all states. However, target variance, transition reliability and reachable remaining horizon are all state dependent factors thus, benefiting from state-dependent discounting. AdaGamma addresses this limitation by learning a state-dependent discount factor via a separate MLP network. Instead of learning this via the standard TD error, the paper introduces a n-step return computed via a delayed scalar discounting factor. This loss overcomes the degenerate solution of minimising the standard TD-error by simply minimising gamma to avoid predicting long term returns. The authors show how this approach can easily be used with PPO and SAC (or any other RL algorithm), improving performance.

Critical strengths
  1. Address a fundamental issues in RL which is shared by all RL algorithms (i.e., how to flexibly adapt the discount factor).
  2. It includes theoretical guarantees as well as very useful ablations.
  3. The approach can easily be integrated with any RL algorithm.
  4. The proposed approach improves performance over baselines.
  5. The motivation is clearly explained.
Critical weaknesses
  1. No comparison to analogous methods, using an adaptive discounting. I am aware the authors introduce SAC/PPO -Uncertainty as a baseline, but this was their own adaptation to a previous published algorithm, which is not clear how well it should perform under their adaption (i.e., in the author's re-implementation the uncertainty is estimated based on two ensemble only). I think it would also be valuable to compare it schedules adaptive discounting, as they report in the Related Work section.
  2. Some quantities are poorly explained or justified. For instance, it is not clear why the authors introduce the regularisation terms in Eq.13 (see also questions).
  3. While the authors show AdaGamma-RC maintains higher variability in the discount factor compared to AdaGamma-TD, they don't show this per-state variability directly map state properties (e.g., states with more uncertain returns, having lower discount factors) (see question below)
  4. The variance regulariser introduced in eq13 does not promote smoothness, it promotes the discount rate to be constant, which is counterintuitive to the main objective (see question below). To promote smoothness, the discount variance should be encouraged to be low across local state, rather than across all state.
  5. The authors says "For off-policy algorithms, the n-step return is computed from sequences of transitions in the replay buffer and is truncated at episode boundaries". This introduces an issue: in off-policy learning, the vanilla n-step return mixes one steps rewards from the behavioural policy with the estimated long term return of the target policy at state n (i.e., the correct off-policy n-step return requires importance sampling for n>1). While this is not a major issue for small n, the authors should discuss this, especially when testing AdaGamma for higher n (i.e., n>5) .

Minor: In Figure3-b, top left panel, the title should refer to Eq.13 not Eq.15, I believe.

Questions for authors
  1. In equation (11), the loss is defined relative to the discount MLP parameters ϕ\phi. So it is not clear why you need the stop gradient operators, since neither VV nor GG depend on the parametrised discount, since they are estimated relative to a fixed lagging discount γˉ\bar{\gamma}. This is not an issue, it is just if the author could clarify as I found this confusing.
  2. For SAC, does the n-step return GG include the policy entropy term? else this introduces a potential mismatch in the RC-loss, where VV in SAC includes the entropy term, while GG may not. Could the authors clarify?
  3. Table3 shows w/o λvar\lambda_{var} RC achieves higher returns with no significant change in STD-γ\gamma, so why was this regularisation included? it also goes against the main objective of the paper which is state-dependent γ\gamma.
  4. Could you try to directly map the per-state change in discounting with the reliability of the critic estimate at the next step (e.g., for PPO, using the critic error at the next step based on the unbiased MC return)? One of the main claims is that the discount should reduce for states with low critic reliability (and vice-versa). At the moment, it is not convincing that the increased overall variability of the discount for AdaGamma (i.e., from 0.0014 to 0.022) maps to anything meaningful.
1 comment

Q1.Why Stop-gradient is used when optimizing only the discount parameters

With separate network parameters and a reference discount treated as fixed during the update, explicit Stop-gradient does not change the derivative with respect to ϕ\phi. Its purpose is to make the optimization semantics explicit: RC treats value predictions and multi-step targets as fixed supervision rather than updating the Critic through this loss. Also, the Critic is trained using adaptive discounts; it is not necessarily the exact value function under γˉ\bar\gamma. We will clarify that the essential design choice is the reference-based supervisory target, not the Stop-gradient notation alone.

Q2.Whether the SAC multi-step target includes entropy terms

In our SAC implementation, both the one-step bootstrap and the multi-step target include a soft value at their respective bootstrap states, incorporating the entropy term. However, Eq. (10) uses raw rewards at intermediate steps and omits their entropy contributions. Including entropy only at the final bootstrap state does not make this target a fully consistent multi-step soft return. Although evaluating old replay actions under the current policy can introduce noisy log-probability terms, this does not eliminate the mismatch or establish that it is negligible. We will clarify this implementation detail in Section 3.3.2 and evaluate a variant that includes intermediate expected entropy under the current policy.

Q3.Why retain the variance regularizer despite the ablation results

The intended role of λvar\lambda_{\mathrm{var}} is to limit excessive cross-state dispersion, not to create state dependence. Table 3 shows that removing it changes the discount standard deviation only from 0.0216 to 0.0217 while increasing the reported mean return from 27.17 to 27.46, so this experiment does not demonstrate that the term is necessary or beneficial. More importantly, RC-only remains non-degenerate, supporting our claim that RC—not the auxiliaries—accounts for the observed anti-collapse behavior. We will describe the term as optional dispersion control and correct “smoothness,” since global variance does not enforce local smoothness.

Q4.Direct evidence linking discount variation to Critic reliability

In Appendix F.3 (Figure 5), we already show on Ant-v4 that SAC-AdaGamma's learned γϕ(s)\gamma_\phi(s) exhibits structured, physically interpretable variation across torso height and planar velocity—reducing γϕ(s)\gamma_\phi(s) in unstable torso/contact regimes and increasing it during stable locomotion—and our matched-mean fixed-γ\gamma ablations (Tables 7 and 8) verify that removing this per-state variation degrades returns across all tasks. That said, directly correlating per-state γϕ(st)\gamma_\phi(s_t) with next-state critic error ∣V(st+1)−Gt+1MC∣|V(s_{t+1}) - G^{\text{MC}}_{t+1}| on PPO rollouts is a clean quantitative test of our reliability hypothesis, and we will check and add this analysis in the revision.

Q5.Comparisons with adaptive-discount methods and schedules

Our SAC/PPO-Uncertainty baseline (Appendix C) was designed to match AdaGamma's parameter and compute footprint by leveraging twin critics with a learnable sensitivity scale β\beta, and Table 1 shows that this adaptation is in fact a strong baseline that substantially improves over Fixed-γ (e.g., 6606.08 vs. 5370.80 on SAC Humanoid-v4 and 3620.87 vs. 3101.48 on SAC Ant-v4) while still being outperformed by AdaGamma. Global discount schedules adjust γ\gamma only over training iterations while still enforcing a single state-independent scalar across all states at any given policy step—which our dense fixed-γ\gamma sweeps up to 0.9999 (Tables 7 and 8) show cannot match state-dependent AdaGamma.

Q6.Off-policy bias in replay-based multi-step supervision

We agree that computing off-policy nn-step returns from replay trajectories without importance sampling mixes behavioral rewards with target-policy bootstrapping; indeed, avoiding high-variance importance-sampling ratio products while keeping off-policy drift small is a primary reason we use a short horizon n=5n = 5, and this directly explains why larger horizons (n∈{10,20}n \in \{10, 20\} in Table 6) degrade performance and safety cost.

Q7.Incorrect equation reference in Figure 3

The label should refer to Eq. (13), not Eq. (15). We will correct it and check the corresponding labels in Figures 3–4 for the same outdated reference.

Reviewer_KVLCOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

In reinforcement learning, the discount factor determines the degree to which the agent values future rewards. Traditional reinforcement learning algorithms such as SAC and PPO typically use fixed discount factors, but the reliability of future value estimation varies under different states, making it difficult to flexibly balance long-term planning and prediction errors with fixed discount factors. Some studies have learned discount factors for state dependencies through neural networks, but they have found that training discount factors directly using TD error can easily lead to degradation problems: the network can reduce its dependence on future value estimation by lowering the discount factor, thereby reducing training losses, rather than truly learning meaningful state dependent discounts. This motivation is indeed very good. So to solve this problem, they proposed AdaGamma, which uses a small neural network to predict discount factors based on the current state and trains them with a Return Consistency objective. This method utilizes a reference discount factor with slower updates to calculate multi-step cumulative returns, which is used as the supervised target for stopping gradients, to make the one-step value estimate corresponding to the current discount factor as close as possible to the multi-step returns, thereby avoiding direct manipulation of the training objectives. In addition, the author constrains the variation of discount factors through regularization and integrates this method into SAC and PPO.

Critical strengths

This article points out that directly learning the discount factor γ by minimizing the TD error can easily degrade it to a value close to a constant. To alleviate this issue, the author proposes the Return Consistency (RC) objective, which utilizes multi-step returns and reference discount factors to train gamma and reduce the risk of gamma degradation by directly reducing TD errors.

Critical weaknesses

Although the RC method alleviates the problem of discount factor degradation in TD training, there is a significant difference in the learned gamma between states, and the corresponding discount time domain fluctuation is also large, while the actual performance improvement is relatively limited. The paper has not fully explained whether this change is reasonable, nor has it fully explained whether the improvement in partitioning performance comes from changes in the average discount factor or the state dependence itself. There are issues of numerical duplication, abnormal coincidences, and inconsistent hyperparameter configurations in multiple experimental results in the paper. Although it may be a recording or organizing error, these issues have not been explained, which raises doubts about the reliability and reproducibility of the experimental results.

Questions for authors
  1. I noticed that there are several experimental data in the paper that deserve further explanation. Firstly, in Table 11, TRPO CrossValidate and TRPO Unintentty on Ant-v4 are two different methods, but the mean and standard deviation of the reported reward are exactly the same, both being 940.26 ± 457.39. Secondly, in Tables 7 and 8, PPO Fixed - γ=0.9950 and γ=0.9800 on Humanoid-v4 have rewards of 456.19 ± 27.61 and 356.19 ± 27.61, respectively, with a mean difference of exactly 100 and identical standard deviations. In addition, the PPO AdGamma rewards for SafetyPointGoal1-v0 in Tables 1 and 6 (n=5) were 26.31 ± 0.90 and 26.30 ± 0.90, respectively, while the standard deviation and cost results were the same. I suspect that these situations may be due to duplicate citations in the experimental logs, copying or input errors during table organization, or different statistical windows and rounding methods, or other possible reasons. At present, the specific reasons are not clear. We hope the author can verify and explain these values, and if necessary, provide the original results of the corresponding random seeds to confirm the consistency and reliability of the experimental data.
  2. This paper directly learns state dependent gamma through neural networks, but I have doubts about whether this method is truly effective. For example, in the same Ant-v4 environment, the average gamma learned by SAC AdaGamma and PPO AdaGamma are 0.9733 and 0.9990, respectively. Although the values differ by only about 0.026, the future reward weight decays exponentially with the number of steps. If measured by the number of steps where the weight decays to 10%, the two correspond to equivalent discount time domains of about 85 steps and 2301 steps, respectively, with a difference of about 27 times. Intuitively, it is questionable whether this difference is reasonable, and the author has not provided sufficient explanation. Although different algorithms and state distributions may lead to different discount strategies, it is still unclear whether such large differences in time scales help improve decision-making or only come from value estimation errors and optimization processes. Therefore, I have doubts about the practical effectiveness of learning state dependence gamma directly through neural networks.
  3. Although in the experiments of the paper, the RC method avoided the degradation of γ into a constant, I have doubts about whether such a large discount time-domain variation is reasonable and whether it truly helps improve decision-making. According to Table 2, if measured by the number of steps where the future reward weight decays to 10%, and estimated by the mean ± 1 standard deviation of γ, the TD method corresponds to approximately 24.30-25.12 steps, with a difference of only 0.81 steps between the upper and lower bounds, which is indeed fixed and tends to; The RC method corresponds to approximately 44.34-310.01 steps, with a difference of about 7 times between the upper and lower bounds. At the same time, the temporal difference in discounts between TD and RC is so significant that the average reward only increased from 26.35 to 27.17, with an improvement value of 0.82. I suspect that this improvement may mainly come from the larger average gamma of RC, making it more suitable for tasks that require long-term planning, rather than necessarily from state dependency itself. Although the author conducted a fixed gamma comparison in the appendix, they did not fully compare the average gamma (0.971) of RC and the fixed discount corresponding to its mean ± standard deviation under the same settings for this TD collapse experiment.
  4. I would like to know what is the actual fixed gamma used by the standard SAC in JD Logistics Online A/B, as well as the mean and standard deviation of gamma learned by SAC AdaGamma. If there is a significant difference in the average discount factors between the two, may the increase in order revenue mainly come from a more suitable average discount time domain rather than the state dependence itself? Due to the lack of reporting on the actual use and learning of gamma in online experiments in the paper, I believe it is currently difficult to determine the specific source of performance improvement. I hope the author can provide an explanation for this.
  5. I believe that direct learning of γ (s) not only changes the process of value dissemination, but may also alter the optimization preferences of strategies, introducing more difficult to explain factors. I have a question: Does the state dependent discount factor learned by this method really reflect a reasonable time structure, or only compensate for Critic's estimation error? Especially, I speculate that when the value function is consistent with the reference discount factor, the return consistency objective (RC) may be more inclined to learn constant discounts rather than state dependent discounts. Therefore, I am not sure what mechanism enables the learned gamma variation to reflect the reliability of future value estimates, rather than the approximation error of the value function or the bias generated by the joint optimization process. I hope the author can further explain the learning mechanism.
  6. In Table 5, the average gamma of SAC CrossValidate in Ant-v4 environment is 0.9000, while in Table 13, the gamma clipping range of Ant-v4 is [0.9700, 0.9990]. Since this range is a manually set hyperparameter, if it is also applicable to CrossValidate, the average gamma cannot be lower than 0.9700. I noticed that the average gamma of CrossValidate in Table 5 was 0.9000 in all three environments, so I doubt if this method actually used a uniform range of [0.9000, 0.9990] instead of the configuration reported in Table 13. I hope the author can clarify the actual range of gamma clipping used by different methods and explain this inconsistency.
  7. The cost limit for SafetyPointGoal1-v0 in Table 13 is set to 25, but the average cost for SAC AdaGamma in Table 1 is 29.37 ± 18.59, which is higher than this threshold. I want to know if these two values use the same statistical caliber? If they are consistent, does it mean that AdaGamma still does not meet the preset security constraints on average? I hope the author can explain this result
  8. The theoretical results of the paper mainly focus on fixed gamma table MDP, while in actual AdaGamma, gamma, Critic, and strategy vary simultaneously. I want to know to what extent these theoretical results can support the stability of practical algorithms? Especially, is it possible to introduce additional training instability when the gamma network continuously changes the Bellman objective of the value function?
  9. The author alleviates the problem of discount factor degradation caused by direct optimization TD errors by using stop gradient and lagged reference discount factors. However, the multi-step supervision objective of RC still relies on Critic's value estimation, and the reference discount factor is continuously updated through the average output of the gamma network, forming an indirect cyclic dependence. I want to know if stopping gradient and EMA updates really theoretically solves this bootstrap problem, or just delays its impact? Especially when the value estimate of Critic is consistent with the reference discount factor, can constant discount also become a self consistent solution for RC? If so, what mechanism ensures that gamma can learn meaningful state dependencies instead of compensating for value estimation errors? The current theoretical analysis of the paper mainly focuses on the fixed gamma Bellman operator and has not yet covered this joint update mechanism.
3 comments

Q1.Numerical duplication and inconsistencies across tables

Thank you for identifying these entries. We will check the original per-seed logs and table-generation scripts for the duplicated TRPO results, the PPO–Humanoid entries, and the discrepancy between Tables 1 and 6. We will correct these errors in the reviesed paper.

Q2.Different discount horizons learned by SAC and PPO on Ant-v4

The difference is substantial, but the horizon calculated from the mean discount is not the actual horizon under state-dependent discounting: future weights depend on trajectory products ∏kγ(st+k)\prod_k\gamma(s_{t+k}), terminal events, and, for PPO advantages, the additional λ\lambda factors. Moreover, SAC and PPO use different value objectives and training distributions, so their learned discounts need not coincide.

Q3.Whether RC benefits mainly from a higher average discount

Table 2 primarily tests whether the discount module loses state dependence under otherwise matched settings; it does not isolate the contribution of state dependence to the reward improvement. The fixed-discount comparisons in Tables 7–8 use the main benchmark settings, not the 300k-step, fixed-reference collapse protocol, so they do not fully answer your specific concern. We plan to add fixed discounts at the RC mean and its neighboring values under that same protocol. We will also clarify that greater discount dispersion is evidence against collapse, not sufficient evidence of improved decision-making.

Q4.Discount settings and the source of gains in the online experiment

The online SAC baseline uses a fixed discount factor of γ=0.99\gamma=0.99. The only difference between the two experimental arms is the discounting mechanism: fixed discounting versus AdaGamma’s learned state-dependent discounting, with the remaining algorithmic settings and deployment conditions held unchanged. This controlled comparison supports attributing the observed gains to replacing fixed discounting with AdaGamma.

Q5.Whether learned discounts capture temporal structure or compensate for Critic error

As discussed in Sections 1 and 3.3.2, down-weighting bootstrap targets in states where the critic V(st+1)V(s_{t+1}) has larger approximation error or weak multi-step predictive reliability is not an unintended artifact—it is the core estimation-time mechanism of AdaGamma. Minimizing LγRCL_\gamma^{\text{RC}} in Eq. (11) effectively regresses the multi-step residual Gt(n)−rtG_t^{(n)} - r_t onto V(st+1)V(s_{t+1}): although γϕ(s)≡γˉ\gamma_\phi(s) \equiv \bar{\gamma} would be a stationary solution in an idealized tabular limit with zero value-approximation error and exact Bellman consistency everywhere, neural critics under stochastic dynamics and evolving policies always exhibit heterogeneous statewise errors, pulling γϕ(s)\gamma_\phi(s) down in unstable contact/transition states and keeping it high in predictable states (as shown over torso height and velocity on Ant-v4 in Figure 5).

Q6.CrossValidate discounts outside the reported clipping range

Thank you for spotting this mismatch in Table 13. SAC-CrossValidate was run using the default [0.900, 0.999] range stated in Section 3.1 across all three environments (which is why it collapsed to 0.9000 in Table 5), whereas [0.97, 0.999] on Ant-v4 applied to SAC-AdaGamma (consistent with Figure 5) and was mistakenly merged into a shared row in Table 13. We will check our experiment scripts, separate the per-method clipping ranges in Table 13, and verify SAC-CrossValidate under [0.97, 0.999] as well.

Q7.Whether the reported cost satisfies the safety constraint

Both numbers use the exact same statistical definition—cumulative undiscounted episode cost over 1,000 environment steps—and 25 in Table 13 is the soft target threshold of a standard Lagrangian multiplier update (LR = 5e-3), which optimizes a primal–dual reward–cost trade-off rather than enforcing a hard statewise safety shield. Under a finite 1M-step budget, soft Lagrangian RL agents commonly overshoot the target threshold during evaluation (37.46 to 52.53 across all baselines in Table 1); while SAC-AdaGamma (29.37 ± 18.59) does not strictly stay below 25 on average, it lies well within one standard deviation of the limit and cuts the excess cost above 25 by 84% relative to SAC-Fixed-γ (+4.37 vs. +27.53) while achieving the highest reward.

Q8.What the theory establishes about joint-training stability

As explicitly scoped in Section 3.4, our tabular proofs establish operator-level properties—contraction and exact soft policy iteration—rather than end-to-end convergence of joint neural training. To address moving Bellman targets, Theorem 2 establishes Lipschitz continuity of the policy value with respect to the discount function, and Corollary 1 (Appendix B.7) bounds the policy-evaluation tracking error under a time-varying sequence of discount functions. Under exact fixed-policy evaluation, the bound controls tracking error in terms of the maximum change in the discount function between successive updates. In practice, our SAC implementation uses a 100,000-step warmup, discount-network updates every 20 critic steps with a learning rate of 0.0001, and target-network smoothing; our PPO implementation freezes the discount outputs within each rollout's optimization epochs. These measures are intended to mitigate target drift, but do not guarantee that the joint training dynamics satisfy the conditions of the tabular tracking analysis. Figures 7 and 8 provide empirical learning curves rather than a joint-convergence guarantee.

Q9.Whether Stop-gradient and EMA eliminate indirect circular dependence

Stop-gradient removes the direct gradient path through the supervisory target, while EMA slows reference changes; neither eliminates feedback through subsequent Critic and discount updates. However, the observed anti-collapse behavior is not explained solely by EMA: Tables 2–3 disable adaptive reference updates, and RC remains non-degenerate even without the auxiliary penalties. This supports the specific empirical distinction from TD training, not a universal exclusion of constant solutions or a joint-convergence guarantee.

Thank you for the responses. I have considered the replies and found they have not meaningfully resolved my concerns. I will maintain my original rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.