acceptodds
Under review as a conference paper at ICLR 2027

Successor-Error Amplification in Temporal-Difference Learning

Abstract

With function approximation, a temporal-difference (TD) update at one state can change the prediction at its successor, an effect we call *successor shift*. The successor is special because its prediction also appears in the bootstrap target used to form the update. We show that this creates a feedback mechanism in which the successor's value error can amplify itself through the TD update, which we call *successor-error amplification*. Under simple conditions on deterministic chains, this amplification is strong enough that the TD update increases the successor's squared value error in expectation. To mitigate this amplification, we derive Successor-Shift-Controlled TD (SSC-TD), which suppresses successor shift while preserving the norm of the TD update. An adaptive version estimates the shift's average effect on the successor's squared value error, and activates suppression when the estimate indicates harm. Experiments show adaptive SSC-TD lowers lifetime MSE relative to TD on a majority of Atari environments.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, temporal-difference learning, value prediction, function approximation, bootstrapping, successor shift, successor-error amplification, adaptive TD methods, Atari

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_unspOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TD-learning algorithms with function approximation introduce a compound error where an erroneous update to the current state due to wrong value estimation of the next state, lead to further aggravating the erroneous prediction of the next state due to function approximation. The paper shows that in stationary deterministic chains with constant positive feature overlap between consecutive states, this compound error is strong enough to lead the TD update to never decrease the next state squared value error in expectation. To mitigate this issue, the paper introduces Successor-Shift-Controlled TD (SSC-TD), which controls the next state value shift (i.e., reducing the error from propagating by a tunable fraction), while preserving the magnitude of the TD update. However, the paper further argues that fully suppressing the next state value shift is not always beneficial (e.g., when both current and next state values estimates need to be adjusted in the same direction). Thus, they further introduce adaptive SSC-TD, which suppresses the next state value shift during learning, only when the update increases the next-state squared value error on average. The paper shows good performance on Atari Lifetime MSE as well as no divergence in Baird's Counter example (unlike TD-learning).

Critical strengths
  1. Targets a fundamental problem in TD-learning with function approximation (i.e., the compound error due to function approximation and bootstrapping the next state estimated value).
  2. To my knowledge, the derivation of how the TD-update affects the expected change in the next-state value error is novel, albeit limited to to stationary deterministic chains with constant positive feature overlap between consecutive states.
  3. The paper shows good performance on Atari Lifetime MSE (i.e., policy evaluation only) compared to two baselines as well as no divergence in Baird's Counter example (unlike TD-learning).
  4. The paper is clearly written and easy to follow.
Critical weaknesses
  1. The "successor-error shift" identified by the paper is a well-know mechanism in RL, which has been studied extensively and it is even described in Sutton and Barto 2018, RL text-book ("Examples of Off-policy Divergence" 11.2, independently of whether the transitions are on- or off- policy). While the authors mention this in the related work section, the abstract and introduction present this as a core contribution of the paper.
  2. The derived SSC-TD is similar to Constrained TD conceptually (Durugkar & Stone 2017). The authors compare SSC-TD performance with Constrained TD, but the results are buried in the appendix (i.e., Table1) with no reference in the main text.
  3. In the current implementation, the algorithm is limited to policy evaluation with linear function approximation. With nonlinear functions, the feature overlap itself changes during training, which the linear case cannot capture.
  4. There was no attempt to investigate the proposed policy evaluation algorithm in relation to policy improvement (i.e., in a full policy iteration loop).
  5. The stationary deterministic chains with constant positive feature overlap between consecutive states is quite restrictive.

Overall my biggest concerns are reduced novelty (e.g., compared to Constrained TD), limited empirical analysis (e.g., no extension to generalised policy iteration or comparison to residual algorithms) and theoretical results limited to a restrictive case (i.e., stationary deterministic chains with constant positive feature overlap between consecutive states).

Questions for authors
  1. Residual methods solve a similar issue (e.g., Baird's residual algorithm), although they suffer from slower learning. Could the authors also compare SSC-TD to residual algorithms, assessing learning speed and performance ?
  2. On Mixed states, SSC-TD is slower on earlier episodes than standard TD-learning. How does this trend behave during generalised policy iteration (i.e., when the policy is also updated). Does this translate to a slower overall learning ? Especially, since during policy iteration, the policy is updated before the policy evaluation step has "converged". Could the authors integrate adaptive SSC-TD in a generalised policy iteration loop comparing it to standard TD-method that delayed the target network updates (i.e., Mnih et al., 2015).
  3. Could the authors be more explicit on the novel contribution of SSC-TD relative to Constrained TD?
  4. Since SSC-TD does not have a general convergence result under linear function approximation. Does this mean SSC-TD lose the convergence guarantees that standard TD-learning has under linear function approximation and on-policy learning?
2 comments
AuthorsOpenReview ↗

Author Response to Reviewer unsp (Part 1/2: Contributions and Related Work)

We thank the reviewer for the detailed review and constructive feedback. We clarify our contributions, their relationship to prior work, and the scope and limitations of our theoretical and empirical results.

W1 / W2 / Q1 / Q3: Contributions and novelty

W1: The "successor-error shift" identified by the paper is a well-know mechanism in RL.

Successor shift and successor-error amplification are different phenomena. The former is known and the later is novel (current paper). Successor shift is the change in the successor's prediction caused by an update at its predecessor. This is well known, as we acknowledge in the Introduction and Related Work (Sec. 7). Sutton and Barto's (Sec. 11.2) includes examples that show how such feedback can cause TD to diverge.

What is new is the analysis of how this shift affects the successor's value error. To our knowledge, prior work has not isolated and quantified the component by which the successor's own error enters the predecessor's TD target and is reinforced at that same successor through feature overlap. We distinguish this harmful component from useful transfer and establish conditions under which the complete successor shift worsens successor's prediction. Importantly, this mechanism occurs even when on-policy TD converges. It is not merely another explanation of divergence.

W2 / Q3: How does SSC-TD differ from Constrained TD, and where is the comparison? 

Both suppress successor shift, but differ in motivation, derivation, and update rule. Motivated by the success of DQN, Constrained TD (CTD) was designed to limit bootstrap-target movement without delayed target networks. It projects out the entire successor-aligned component, which can nearly eliminate the update when consecutive features align. In contrast, SSC-TD is developed to address successor error amplification, and is derived from an optimization objective that penalizes successor shift (Eq. 11). The SSC-TD update allows partial suppression, and normalizes the update to preserve TD's norm. Unlike CTD, adaptive SSC-TD suppresses only when the estimated shift is harmful, because useful generalization should be retained (Sec. 6.2, Fig. 4).

The empirical difference is substantial. Adaptive SSC-TD outperforms Constrained TD on ALL of the 56 Atari games, and the gap is substantial on some of the games (e.g., Skiing and Tennis); see Table~1-3 in the Appendix E.4. We already discuss CTD in Related Work and refer to its comparison in Sec. 6.3, with full results in Appendix E.4. In the revision we highlight these numbers in the main experiments.

Q1: Comparison with residual-gradient methods.

Residual gradient (RG) is not a standard competitive baseline for value-prediction accuracy. Its slow learning has long been documented (Baird, 1995). More directly, Li (2008) compared TD and RG and found that TD generally achieves lower prediction error, while RG produces smaller TD errors. Recent experiments with neural networks also found TD superior to RG (Yin et al., 2022). This distinction is important because our objective is accurate value prediction, not minimizing the Bellman residual. We therefore use TD as the primary benchmark.

SSC-TD is closely related to residual algorithms, and its update can be expressed as a state-dependent mixture of TD and RG. However the motivation and derivation of the two algorithms are totally different. RG's successor-gradient correction arises from minimizing squared Bellman error, not from analyzing successor shift. RG applies this correction regardless of its effect and can even reverse the shift. SSC-TD instead partially suppresses the shift without reversing it, while adaptive SSC-TD suppresses it only when estimated to be harmful. Our contribution is not mixing TD and RG, but identifying when and how suppression is beneficial.

In the revised paper we clarify this relationship and the established TD–RG comparisons in Related Work.

AuthorsOpenReview ↗

Author Response to Reviewer unsp (Part 2/2: Theory and Experiments)

W5 / Q4: Theory and Analysis

W5: Are the deterministic-chain and constant-overlap assumptions essential?

They are needed for the closed-form net effect (9), not for successor-error reinforcement (7) itself. The decomposition in Eq. (7) holds for stochastic transitions and varying overlap. Positive overlap still amplifies the successor's own error, though the net shift need not be harmful.

To address this concern more fully, we have extended the net-effect analysis in Eq. (9) to stochastic transitions in the revised manuscript (Appendix B). Under stationarity and constant positive overlap, stochastic transitions add the following covariance contribution to Eq. (9): αc Es[Cov⁡(ξ,e(s′)∣s)],ξ=r+γVπ(s′)−Vπ(s).\alpha c\,\mathbb E_s[\operatorname{Cov}(\xi,e(s')\mid s)],\quad \xi=r+\gamma V^\pi(s')-V^\pi(s). This covariance can have either sign. At γ=1\gamma=1 (with centered rewards), the net shift remains harmful unless a negative covariance offsets the nonnegative error-difference term.

Q4: Convergence of SSC-TD under linear function approximation

Standard TD's linear on-policy guarantee does not automatically transfer to SSC-TD, because the expected-update operator changes. We have not established a guarantee for general suppression strengths or adaptive switching. The Baird experiment demonstrates stability in the tested configuration, not universal convergence. We state this explicitly in Section 8.

Experiments

W3: Does SSC-TD extend to nonlinear approximation with changing feature overlap?

We agree that our reported experiments do not establish SSC-TD's effectiveness with learned representations. We focus on linear value prediction, where prediction changes can be measured exactly and the mechanism can be studied without the additional effects of representation learning. We have clarified this scope in the revision and identify nonlinear SSC-TD as an important direction for future work (Sec. 8).

The restriction to linear approximation limits our theoretical guarantees and empirical conclusions, not the underlying mechanism. Successor-error amplification does not require fixed feature overlap. Equation (7) holds for varying overlap, and for differentiable nonlinear value functions, the same decomposition holds to first order with current value gradients replacing features (Sec. 2). Their overlap can change during learning.

W4: Why is SSC-TD not evaluated within policy iteration?

Our primary contribution concerns successor-error reinforcement in value prediction, rather than policy improvement. We use fixed-policy evaluation, following the established Atari prediction benchmark of Javed et al., to isolate the effect of TD updates on prediction accuracy. Whether these improvements also lead to better policy learning is an important question for future work (mentioned in Sec. 8). A reliable Atari control comparison would require additional experiments and evaluation beyond what is feasible during the rebuttal period.

Q2: Does the early slowdown in Mixed States imply slower policy improvement?

The early slowdown in Mixed States occurs with fixed SSC-TD, not adaptive SSC-TD. In the same experiment, adaptive SSC-TD avoids this slowdown and performs at least as well as TD throughout learning (Fig. 4). This is precisely the motivation for adaptation: to preserve useful generalization while suppressing harmful successor shift.

The Mixed States environment has only one action, so policy improvement cannot be evaluated there. Although these results address the concern about early prediction performance, they do not establish how adaptive SSC-TD performs under changing policies.

Reviewer_QoNoOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper identifies successor-error amplification issue in TD learning, that the error in successor is amplified because of model generalization. To mitigate the issue, the authors propose new update method and an adaptive variant to cancel the amplification. The experiments with Atari env demonstrate the effectiveness.

Critical strengths
  1. The paper identifies a new issue in general TD learning and analyzes its influence with strict mathematical analysis.
  2. The paper proposes a simple and effective method to suppress the issue. Furthermore the method has straight math interpretation for understanding.
  3. The paper also analyzes thoroughly when the issue actually harms, and proposes adaptive variant.
Critical weaknesses

The paper only analyzes stationary situation, which may not hold in practical RL settings. The experiments use Markovian sampling, which is different from the analysis.

Questions for authors

The paper focuses on single-transition TD updates. Could large batch mitigate successor-error amplification? If so, how does batch size affect it?

2 comments
AuthorsOpenReview ↗

Author Response to Reviewer QoNo

We thank the reviewer for the positive assessment and constructive questions.

W: The analysis assumes stationarity, whereas the experiments use Markovian sampling.

Successor-error amplification is not limited to stationary settings. Stationarity allows us to derive precise analytical results about the expected effect of successor shift, but the underlying mechanism applies to individual TD updates regardless of the sampling distribution. Indeed, our MountainCar and CartPole experiments (Sec. 6.1) demonstrate substantial amplification on nonstationary episodic trajectories.

To examine the role of sampling directly, we repeated the Atari experiments with transitions sampled independently and uniformly from the recorded trajectories rather than sequentially. We compared TD with fixed SSC-TD, whose suppression does not depend on the order of transitions. The benefits of suppression persist under independent sampling. With single-transition iid sampling (batch size 1), fixed SSC-TD outperformed TD on 53 of 56 Atari games, with a mean game-wise MSE reduction of 42.53% (Appendix E.4, Fig. 9). We further tested independently sampled minibatches of size 10, with results discussed in our response to the next question. In conclusion, neither nonstationary trajectories nor temporal dependence in the sampling process is essential to observing the phenomenon.

Q: Can larger batches eliminate successor-error amplification, and how does batch size affect it?

Successor-error amplification persists even with full-batch TD updates. Larger batches reduce sampling noise and bring the update closer to its expectation, which still contains the error-reinforcing component of successor shift, as per the analysis in Section 3.

Following the reviewer's suggestion, we conducted new Atari experiments using independently sampled transitions from the recorded trajectories, with minibatch size 10. The benefits of suppression persist in this setting. SSC-TD achieved lower prediction MSE on 46 of 56 games (Figure 6). These results show that the benefit of suppression does not depend on sequential sampling. We include this experiment in the main experimental section (Sec. 6).

Thank you for the authors addressing all my concerns. I have no more questions. I will raise the score to 4. Thank you for your hard work. Hope you have a good result.

Reviewer_CRTtOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies successor-error amplification in TD learning: an error in the successor’s value enters the bootstrap target and can reinforce itself through shared features. It analyzes this mechanism on deterministic chains and proposes SSC-TD, which suppresses successor shift while preserving the update norm. An adaptive variant is evaluated on synthetic problems and Atari prediction tasks.

Critical strengths

The paper provides a clear interpretation of how bootstrapping and feature overlap can reinforce successor errors. The distinction between error amplification and useful transfer is informative, and the controlled chain experiments illustrate why suppression should adapt during learning.

Critical weaknesses
  1. The analysis concerns error changes at the sampled successor, which does not establish that suppression improves overall prediction accuracy. Moreover, SSC-TD may fail to preserve the true value function as a fixed point under stochastic transitions, even when representable. A characterization of this bias would strengthen the method’s justification.

  2. The Atari evaluation uses only one trajectory per game and two step sizes. Multiple independent runs, uncertainty estimates, and a broader learning-rate comparison are needed to establish the robustness of the reported improvements.

  3. Under what conditions does suppressing successor shift improve overall value prediction error? Can yo u characterize the bias introduced by SSC-TD under stochastic transitions?

  4. Do the Atari improvements persist across multiple independent trajectories and a broader range of independently tuned step sizes?

Questions for authors

See Weaknesses.

1 comment
AuthorsOpenReview ↗

Author Response to Reviewer CRTt

We thank the reviewer for their constructive feedback.

W1 / W3(a): Does suppressing successor shift improve overall prediction error?

Not necessarily. Positive successor-error amplification identifies a harmful component of the shift, not its net effect on the successor, let alone on all states. Indeed, suppressing the full shift can remove useful generalization even at the successor itself (Sec. 6.2, Fig. 4). This is why we develop adaptive suppression rather than advocate suppressing every shift.

We agree that the successor-level analysis alone does not establish global improvement. We have clarified this distinction in the revised manuscript (Sec. 2). Our experiments address the broader empirical question by measuring overall prediction error, namely RMSVE across all ring states and lifetime MSE on Atari, rather than only the change in the sampled successor's error. SSC-TD improves these measures in most tested settings.

The paper's primary contribution is a quantitative understanding of successor-error reinforcement, a mechanism by which bootstrapping and generalization interact in TD learning, and its distinction from useful transfer. SSC-TD demonstrates how this understanding can guide update design, without claiming to improve global error in every setting. Establishing general guarantees for overall prediction error requires accounting for how updates affect predictions through all states. Our analysis provides a foundation for pursuing this broader goal.

W1 / W3(b): Does SSC-TD introduce fixed-point bias under stochastic transitions, and can this bias be characterized?

As noted in Sec. 8, SSC-TD may not preserve the true value function as a fixed point under stochastic transitions, even when the function is exactly representable. This happens because the sampled successor influences both the TD error and the update direction. Their correlation can produce a nonzero expected update at the true values. The same issue arises in single-sample residual-gradient updates, is known as the double-sampling problem, for which standard resolutions exist in the literature.

The bias has a simple characterization. Let qη(s,s′)q_\eta(s,s') denote the SSC-TD direction for fixed suppression η\eta, and let w∗w^{*} represent VπV^\pi. The expected update at the true values is

=\mathbb E_s[\operatorname{Cov}(r+\gamma V^\pi(s'),q_\eta(s,s')\mid s)].$$ The term $V^\pi(s)$ cancels because it is constant given $s$. If the reward depends only on the current state, the bias is simply proportional to the covariance between the successor value and the update direction. It vanishes for deterministic transitions or $\eta=0$. For small $\eta$, the displacement of the mean-update fixed point is $O(\eta)$ provided the expected-update operator remains well-conditioned. One standard remedy is double sampling, using independent transitions from the same state for the TD error and the entire SSC-TD direction, including normalization. This removes the covariance and restores the true-value fixed point. An auxiliary estimate of the conditional mean TD error offers another possible route, related to the use of auxiliary estimates in gradient-TD methods. We have added this characterization and the potential remedies to Sec. 8. ### **W2 / W4: Do the Atari improvements persist across independent trajectories and a wider range of step sizes?** We thank the reviewer for this suggestion. We have expanded the Atari evaluation to 20 independent trajectories per game and a broader sweep of step sizes, $\alpha\in\{ 10^{-6},3\times10^{-6},10^{-5},3\times10^{-5},10^{-4}\}$. In the revised manuscript, we now report mean lifetime MSE with 95% confidence intervals, as well as performance across step sizes. The overall pattern and conclusions remain consistent with the original experiments. The improvements remain consistent across the tested step sizes. At $\alpha=10^{-5}$, adaptive SSC-TD reduced lifetime MSE relative to TD on 47 of 56 games, tied on 1, and increased it on 8, with a mean game-wise relative MSE reduction of 15.03%. At $\alpha=3\times10^{-6}$ and $3\times10^{-5}$, it improved on 46 and 48 games, respectively. In the step-size selection process, we chose the step size that minimized TD's prediction error among the values tested, then used that same step size for SSC-TD without optimizing it separately. Thus, TD had the benefit of step-size selection, making this a conservative comparison for SSC-TD. Our aim was to compare the update rules under matched conditions rather than maximize benchmark performance through extensive tuning. We have incorporated these new results and confidence intervals into the main experimental section of the revised manuscript in Sec. 6.3 (Fig. 5), with additional step-size comparisons in Appendix E.4 (Figs. 7–8).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.