acceptodds
Under review as a conference paper at ICLR 2027

Deep Reinforcement Learning and The Tale of Two Temporal Difference Errors

Abstract

The temporal difference (TD) error was first formalized in Sutton1988, where it was first characterized as the difference between temporally successive predictions, and later, in that same work, formulated as the difference between a bootstrapped target and a prediction. Since then, these two interpretations of the TD error have been used interchangeably, with the latter eventually being adopted as the critic loss in deep reinforcement learning (RL) architectures. In this work, we show that these two interpretations of the TD error are not always equivalent. In particular, we show that increasingly nonlinear deep RL architectures can cause these interpretations of the TD error to yield increasingly different numerical values. Then, building on this insight, we show how choosing one interpretation of the TD error over the other can affect the performance of deep RL algorithms that utilize the TD error to compute quantities beyond the value function, such as with deep differential (i.e., average-reward) RL methods. All in all, our results show that the default interpretation of the TD error as the difference between a bootstrapped target and a prediction does not always hold in deep RL settings.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Deep Reinforcement Learning, Average-Reward Reinforcement Learning, Temporal Difference Error

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_N14dOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates two formulations of the temporal-difference error: the conventional Bellman residual (explicit TD error) and the change in value predictions induced by a learning update (implicit TD error). The authors show that these quantities can differ under function approximation, and investigate using implicit TD errors for average-reward updates in differential Q-learning and reward centering, as well as advantage estimation in A2C. The empirical results suggest that the implicit formulation can improve training stability and performance.

Critical strengths
  • The paper presents an interesting perspective on temporal-difference errors, with potentially useful implications for deep RL algorithms.

  • The theoretical analysis is generally intuitive and easy to follow, particularly the characterization of the discrepancy under linear function approximation.

  • The empirical results are promising, showing improvements in stability and learning performance when using implicit TD errors in several RL algorithms.

Critical weaknesses
  1. Lack of a unified experimental narrative.

The paper investigates three somewhat loosely connected applications of implicit TD errors. For differential Q-learning and reward centering, implicit TD is used to update the average-reward estimate, whereas for A2C, it serves as an alternative advantage estimate. These are substantially different roles, and the paper does not sufficiently connect them or explain why the implicit formulation should be preferable in both settings. The experimental section would benefit from a more coherent motivation and organization.

Furthermore, all three methods still use explicit TD errors to train their critics. The implicit formulation is only introduced for auxiliary updates or actor training. This distinction should be emphasized more clearly throughout the paper, including in its overall framing.

  1. Limited experimental coverage and insufficient ablations.

Each of the three applications is evaluated on only one environment, making it difficult to assess the generality of the reported improvements. Additional experiments across different tasks would substantially strengthen the empirical claims.

Furthermore, the paper does not sufficiently investigate why implicit TD improves performance. For example, in A2C, implicit TD can be approximated as the explicit TD error scaled by the squared norm of the critic gradient, suggesting that the improvements might result from adaptive advantage scaling rather than a fundamentally better advantage estimator. Comparisons against normalized or clipped explicit TD advantages would help clarify the mechanism.

  1. Experimental comparisons do not fully isolate the investigated factors.

The experiments in Figure 1 do not clearly isolate the effect of network nonlinearity. In the inverted pendulum experiment, both network architecture and batch updates change, while the Breakout experiment changes network size without changing the number of layers. More controlled comparisons would strengthen the conclusion that increased nonlinearity causes greater divergence between the two TD errors.

Questions for authors
  1. What is the historical and theoretical justification for the implicit TD error formulation in Equation (6)?

This is my most significant question regarding the paper, and the answer would substantially affect my assessment of its novelty and contribution.

My understanding is that in Sutton (1988), temporally successive predictions refer to predictions made at successive points along a trajectory, rather than predictions of the same state-action pair before and after a learning update. Equation (6), on the other hand, appears to arise from rearranging the tabular Q-learning update rule.

Could the authors clarify whether Equation (6) has previously been established as an alternative formulation of the TD error, and provide the relevant references?

If this formulation is indeed historically established as an interpretation of the TD error, the paper's investigation of their nonequivalence would be well motivated. However, if it is instead a newly introduced quantity, the nonequivalence seems largely expected, as the two expressions measure fundamentally different things. In that case, I believe the paper's central motivation and claimed significance would need substantial reconsideration.

This would not necessarily invalidate the empirical findings, but I would view the work differently: as proposing a new learning signal based on changes in value predictions, rather than resolving a discrepancy between two established interpretations of TD error.

  1. What explains the improved performance of implicit TD advantage estimation in A2C?

Does the improvement come from a better advantage estimate, or from the implicit gradient-dependent scaling of explicit TD errors? A comparison against advantage normalization, clipping, or gradient-norm-weighted explicit TD errors would be informative.

  1. How does implicit TD compare against direct average-reward estimation?

For differential Q-learning and reward centering, why should implicit TD updates be preferable to estimating the average reward directly from observed rewards, for example using an exponential moving average? A corresponding baseline would be useful.

  1. Could the authors clarify Proposition 3.5?

Equation (15) appears to provide a sufficient condition for equality between the batch-averaged TD errors, but not a necessary condition, since different error terms may cancel. Could the authors clarify whether the stated necessity is justified?

  1. Could the authors clarify the A2C implementation?

Appendix C.3 describes using replay buffers and target networks for A2C, whereas Algorithms 3 and 7 present single-transition updates. How are these reconciled in the implementation, and are any off-policy corrections used for replayed actor updates?

6 comments
AuthorsOpenReview ↗

Rebuttal (1/3)

We thank the reviewer for their insightful and detailed review of our work. Below we address the reviewer’s concerns and questions. Due to space constraints, we split our response across multiple comments:

Each of the three applications is evaluated on only one environment, making it difficult to assess the generality of the reported improvements. Additional experiments across different tasks would substantially strengthen the empirical claims.

Since the submission deadline, we have run additional experiments, such that each class of algorithm is run for three different environments each (rather than one environment each, as was the case in the submitted draft). Among the 6 new experiments, a statistically-significant performance improvement (when using the implicit TD error) was observed in 4 out of the 6 experiments, with statistically-equivalent performance observed in the other two experiments. These new results are included in Figure 3 of the updated draft.

All three methods still use explicit TD errors to train their critics. The implicit formulation is only introduced for auxiliary updates or actor training. This distinction should be emphasized more clearly throughout the paper, including in its overall framing.

We kindly note that the submitted draft of the paper alluded to this distinction in the abstract (line 21 of the submitted draft; line 21 of the updated draft), introduction (line 52 of the submitted draft; line 52 of the updated draft), Section 4 (lines 292 and 296 of the submitted draft; lines 292 and 296 of the updated draft), Section 4.4 (line 424 of the submitted draft; line 430 of the updated draft), and Section 5 (line 469 of the submitted draft; line 496 of the updated draft).

To address the Reviewer’s concern and make this distinction more clear, we have added a clarification statement at the start of Section 4 that states that all three methods still use explicit TD errors to train their critics (see lines 299-302 of the updated draft).

The paper investigates three somewhat loosely connected applications of implicit TD errors. These are substantially different roles, and the paper does not sufficiently connect them or explain why the implicit formulation should be preferable in both settings. The experimental section would benefit from a more coherent motivation and organization.

The aim of Section 4, in which we present the three applications of the implicit TD error, is to show that the choice of TD error can affect performance. We do not claim that the implicit TD error is always preferable, only that the choice between the two TD errors should be made purposefully in any deep RL setting in which the TD error is used to compute quantities beyond the value function. This is precisely what connects the three classes of algorithms discussed in the paper: they all use the TD error to compute quantities beyond the value function (this is mentioned several times in the paper; see our above response). Hence, by showing three district classes of algorithms in which the choice of TD error matters (a claim that has been further validated by the new experiments performed), we are able to show that the implication of the implicit vs. explicit TD error is a universal one that goes beyond a single class of algorithms.

AuthorsOpenReview ↗

Rebuttal (2/3)

The paper does not sufficiently investigate why implicit TD improves performance. For example, in A2C, implicit TD can be approximated as the explicit TD error scaled by the squared norm of the critic gradient, suggesting that the improvements might result from adaptive advantage scaling rather than a fundamentally better advantage estimator. Comparisons against normalized or clipped explicit TD advantages would help clarify the mechanism.

We respectfully disagree with the Reviewer on this. For the differential and reward centering algorithms, in addition to the empirical results, we provide a theory-based discussion on why the implicit TD error mitigates the observed instability (see Appendix A in the submitted and updated drafts). For A2C, we empirically compare the algorithm when using the explicit vs. implicit TD error as the advantage estimate while keeping all other components fixed. This allows us to isolate the effect of using either of the TD errors, and hence, observe which of the two TD errors is a better advantage estimator.

The Reviewer also suggests comparing the implicit TD error against normalized or clipped explicit TD errors. While we agree that this can be an interesting direction for future work, it goes beyond the scope of this work, as our aim is to compare the standard explicit TD error to the implicit TD error. Moreover, even if it is the case that such modifications to the explicit TD error improve performance, it does not invalidate our claim that the choice of which TD error to use matters. Again, we do not claim that the implicit TD error is always preferable, only that the choice between the two TD errors should be made purposefully in any deep RL setting in which the TD error is used to compute quantities beyond the value function.

The experiments in Figure 1 do not clearly isolate the effect of network nonlinearity. In the inverted pendulum experiment, both network architecture and batch updates change, while the Breakout experiment changes network size without changing the number of layers. More controlled comparisons would strengthen the conclusion that increased nonlinearity causes greater divergence between the two TD errors.

While we agree that Figure 1 does not, by itself, clearly isolate the effect of nonlinearity, we disagree that these effects are not discussed elsewhere. In particular, the effect of adding nonlinearities and batch updates are discussed (independently) in Sections 3.2 and 3.3, respectively, such that the results in Figure 1a) validate these claims from an empirical perspective. In Figure 1b) we isolate how adding more hidden units (while keeping the numbers of layers unchanged) causes greater divergence between the two TD errors. While we agree that performing more experiments (e.g. by changing the number of layers) would yield more insights, it would not change the central conclusions from our work: that the difference between the two TD errors increases with more nonlinearity, and that this difference affects performance. As such, we leave such further exploration for future work.

To this end, we have updated Section 5 of the paper to clarify that future work also includes performing additional experiments to study the effects of individual sources of nonlinearity (see lines 514-515 of the updated draft).

AuthorsOpenReview ↗

Rebuttal (3/3)

What is the historical and theoretical justification for the implicit TD error formulation in Equation (6)?

We are happy to provide the requested clarification:

In Sutton (1988), the TD error is characterized as “the difference between temporally successive predictions” numerous times (e.g. in the abstract of that paper), before being quantified as the difference between a bootstrapped target and a prediction (e.g. see Section 6.4 of that paper). In other words, it is argued in Sutton (1988) (again see, for example, Section 6.4 of that paper) that the difference between temporally successive predictions can be characterized via the explicit TD error. Here, we begin to see the conflation between the two quantities.

This conflation can most recently be seen in [1, 2], where the authors (which include Sutton), characterize the TD error as an n-step version of Equation (6) (i.e., the implicit TD error). More specifically, in both works, the authors use the implicit interpretation of the TD error to prove the convergence of the average-reward estimate of their (tabular) algorithms (e.g. see Equation B.12 in [1]). Crucially, however, they use the explicit TD error in the algorithm pseudocode itself, thereby also conflating the two quantities. For example, despite using the implicit TD error interpretation to show convergence, the pseudocode for Algorithms 1-3 of [2] uses the explicit TD error for the average-reward update.

As such, [1,2] provide historical and theoretical justification for the implicit TD error formulation in Equation (6), as well as show that the two interpretations of the TD error continue to be conflated, thereby motivating the need for our work.

We are happy to further discuss this point with the Reviewer, given its importance to the Reviewer's assessment of our paper.

[1] Wan et al. 2021 “Learning and Planning in Average-Reward Markov Decision Processes.”

[2] Naik et al. 2024 “Reward Centering”

What explains the improved performance of implicit TD advantage estimation in A2C?

In our A2C experiments, we kept all aspects of the experimental setup and algorithm consistent except for using the explicit vs. implicit TD error as the advantage estimate. Accordingly, we isolated the effect of using the implicit vs. explicit TD error, such that any gains in performance (if any) can be attributed to which of the TD errors is a better advantage estimate. While the additional experiments suggested by the Reviewer would be insightful, they would not change the conclusions or claims of the paper. Accordingly, we leave such efforts to future work.

How does implicit TD compare against direct average-reward estimation?

This is a great question, and we are happy to clarify. The primary advantage of using TD error-based average-reward estimation, rather than direct average-reward estimation, is that direct average-reward estimation only works in the on-policy setting, whereas TD error-based average-reward estimation works in both on-policy and off-policy settings (this property was shown in Appendix D of [1]). Hence, for most algorithms of interest, using TD error-based average-reward estimation is the appropriate choice. In our work, we are interested in seeing the effect of using the explicit vs. implicit TD error, so a comparison with direct average-reward estimation is out of scope for the research question being considered.

[1] Wan et al. 2021 “Learning and Planning in Average-Reward Markov Decision Processes.”

Could the authors clarify Proposition 3.5?

This is a good catch by the Reviewer. We have updated Equation (15) to state a necessary condition.

Could the authors clarify the A2C implementation?

This is another good catch by the Reviewer. Appendix C.3 contains a copy/paste error related to the use of replay buffers which has now been resolved in the updated draft. No replay buffers were used; Algorithms 3 and 7 denote the implementations used in our experiments (we note that this is the standard A2C algorithm implementation, with the exception of which TD error is used as the advantage estimate). Given that the standard A2C algorithm is an on-policy algorithm, off-policy correction is not required. We again thank the reviewer for identifying this typo.

We again thank the reviewer for this high-quality review which has undoubtedly helped us improve the paper. We are happy to further discuss with the reviewer as needed.

Reviewer_Ktd9OpenReview ↗
Rating
3
Confidence
1
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the difference between two TD signals in reinforcement learning: the difference between the bootstrapped target and the current value prediction, and the change in the value prediction for the same input across one parameter update, divided by the learning rate. The authors refer to these as the explicit and implicit TD errors, respectively; they analyze the equivalence of the two under tabular updates, and how function approximation and batch updates break this equivalence. Building on this, the authors keep the original training procedure for the value network and use the implicit signal for the average-reward update or as the advantage estimate for the policy. The paper demonstrates the divergence between the two signals through linear and neural-network experiments, and validates this use in differential Q-learning, reward centering, and A2C on Breakout, Pong, and Ant, respectively. With separate hyperparameter tuning for each method and evaluation over multiple random seeds, the results show that this substitution improves learning performance or stability in the settings tested. The appendices further provide an explanation for the improved stability, algorithm pseudocode, and experimental details.

Critical strengths
  1. The line of reasoning is clear, and the modification is simple yet insightful. The paper distinguishes the "target residual" from the "actual change in predictions," and then uses the latter for the average-reward or policy updates. The value of the contribution lies in this specific use. The paper also ties the motivation, derivation, and application together well.
  2. The experiments provide concrete support, and hyperparameter tuning was taken seriously. The authors demonstrate gains across three classes of algorithms.
Critical weaknesses
  1. The authors offer a possible explanation for the improved stability in Appendix A and acknowledge that further study is needed. However, their justification for treating the implicit update as a reliable reference relies on results from the tabular setting. It remains unclear whether this justification extends to deep neural networks. The theoretical claims in Section 4.1 therefore seem somewhat stronger than the analysis supports.
Questions for authors
  1. Could the authors clarify why the implicit update can be treated as a reliable reference when using deep neural networks? If the analysis in Appendix A is intended as a possible explanation motivated by tabular results, I suggest making this explicit in Section 4.1.
3 comments

We thank the reviewer for their review of our work. Below we address the reviewer’s concerns and questions:

The authors offer a possible explanation for the improved stability in Appendix A and acknowledge that further study is needed. However, their justification for treating the implicit update as a reliable reference relies on results from the tabular setting. It remains unclear whether this justification extends to deep neural networks. The theoretical claims in Section 4.1 therefore seem somewhat stronger than the analysis supports.

We are happy to clarify. In short, the discussion in Appendix A does not assume a specific kind of function approximator. Rather, Appendix A serves as a theory-based discussion that examines how using the explicit TD error for average-reward estimation can add instability to the update. Indeed, as noted by the Reviewer, if a deep neural network is used as the function approximator, then the results from the tabular setting regarding the stability of the implicit average-reward update do not directly apply (as is the case with most theoretical results in the deep RL setting). This is precisely why we do not formalize any of the results in Appendix A as lemmas, propositions, or theorems, given that making such statements would require additional theoretical work which is seldom done in the deep RL setting.

To address the Reviewer’s concern, we have added a clarification statement in the updated draft of the paper (see lines 369-372 of the updated draft) that clarifies that the analysis is intended as a possible explanation for the instability observed motivated by tabular results. We also added a clarifying remark in Appendix A (see lines 623-626 of the updated draft).

Could the authors clarify why the implicit update can be treated as a reliable reference when using deep neural networks? If the analysis in Appendix A is intended as a possible explanation motivated by tabular results, I suggest making this explicit in Section 4.1.

The analysis in Appendix A is intended as a possible explanation motivated by tabular results. We have made this clarification in the updated draft of the paper (see lines 369-372 and 623-626 of the updated draft).

We again thank the reviewer for their review of our paper. We are happy to further discuss with the reviewer as needed.

Thank you for the clarification and revisions. The updated Section 4.1 and Remark A.1 address my concern about the scope of the theoretical explanation. I remain positive about the method and empirical results. Given my limited background in this area, I maintain my rating of 3 (weak acceptance) and confidence of 1.

We again thank the reviewer for their time and effort. We are glad to hear that we have addressed the reviewer's concern and are happy to address any additional questions that may come up in the discussion period.

Reviewer_KdnrOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

In this paper, the authors compare the two interpretations of the TD errors, and analyze their equivelence under tabular and function approximation settings. Based on the results, the authors propose using implicit TD errors for average-reward updates in differential RL, and also the advantage estimation in A2C algorithm. Experimentrs on three tasks report improved learning stability and performance by applied the above changes.

Critical strengths

One of the strength in this paper may provide a mathematical characterization of the differences between explicit and implicit TD errors under function approximation, but the insight behind this is limited.

Critical weaknesses
  1. The explicit TD error measures the Bellman prediction error, whereas the implicit TD error measures the prediction change induced by an explicit TD-based parameter update. Their nonequivalence under function approximation is therefore not surprising. The paper does not sufficiently justify why these quantities should be treated as alternative interpretations of the same TD error or why their difference constitutes a significant research problem.
  2. The implicit TD error defined in this paper appears to differ from the notion of temporally successive predictions in Sutton (1988), as it measures prediction changes before and after parameter updates at the same state-action pair. The authors do not sufficiently clarify this distinction or justify treating the two formulations as alternative interpretations of TD error. Moreover, the motivation for applying implicit TD only to average-reward estimation and policy updates, while retaining explicit TD for critic learning, remains unclear.
Questions for authors
  1. Could the authors clarify the proof of Proposition 3.5? Eq. (15) appears to be sufficient but not necessary for the two batch-averaged TD errors to be equal, since terms from different samples may cancel out.
  2. Sutton (1988) compares predictions across successive observations, while this paper compares predictions before and after a parameter update. Could the authors explain how these two definitions are related?
  3. Since implicit TD is computed using parameters updated by explicit TD, what is the benefit of introducing implicit TD? Also, why is it only used for average-reward and policy updates, but not for critic learning?
4 comments
AuthorsOpenReview ↗

Rebuttal (1/2)

We thank the reviewer for their review of our work. Below we address the reviewer’s concerns and questions. Due to space constraints, we split our response across multiple comments:

Sutton (1988) compares predictions across successive observations, while this paper compares predictions before and after a parameter update. Could the authors explain how these two definitions are related?

We are happy to clarify. As a tangible example, consider the tabular Q-learning algorithm (3) in our paper. Indeed, for tabular Q-learning, at each observation, only one state-action pair has a non-zero (implicit or explicit) TD error. However, this does not mean that the implicit TD error is undefined for the other state-action pairs during this observation. Rather, as per Equation (3c), the implicit TD error for these other state-action pairs, during this observation, equals zero. Accordingly, we can always define the implicit TD error for all state-action pairs across successive observations, such that our definition for the implicit TD error encompasses the notion of “predictions across successive observations”.

In other words, we can always compare predictions across successive observations using the implicit TD error, such that this difference (i.e., the implicit TD error) equals zero if there are no parameter updates.

The paper does not sufficiently justify why these quantities should be treated as alternative interpretations of the same TD error.

We are happy to clarify. The two interpretations of the TD error have historically been assumed to be interchangeable. For example, in Sutton (1988), it is argued (see, for example, Section 6.4 of that paper) that the “difference between temporally successive predictions” can be characterized via the explicit TD error. Here, we begin to see a conflation between the two notions of the TD error.

This same conflation can most recently be seen in [1, 2], where the authors characterize the TD error as an n-step version of the implicit TD error. More specifically, in both works, the authors use the implicit interpretation of the TD error to prove the convergence of the average-reward estimate of their (tabular) algorithms (e.g. see Equation B.12 in [1]). Crucially, however, they use the explicit TD error in the algorithm pseudocode itself, thereby also conflating the two quantities. For example, despite using the implicit TD error interpretation to show convergence, the pseudocode for Algorithms 1-3 of [2] (some of which use a nonlinear function approximator) uses the explicit TD error for the average-reward update.

As such, [1,2] provide evidence that the two interpretations of the TD error are assumed to be interchangeable, thereby motivating the need for this work, in which we show that this equivalence does not hold in deep RL settings.

[1] Wan et al. 2021 “Learning and Planning in Average-Reward Markov Decision Processes.”

[2] Naik et al. 2024 “Reward Centering”

The paper does not sufficiently justify why the difference between the two TD errors constitutes a significant research problem.

We strongly disagree with the Reviewer on this. In the paper, we show that this difference can cause significant instability in differential (i.e., average-reward) and reward centering algorithms, as evidenced by the results shown in Figure 2. Moreover, we show that this difference can affect the overall performance of various classes of RL algorithms in which the TD error is used to compute quantities beyond the value function, as evidenced by the results shown in Figure 3.

We also note to the Reviewer that since the submission deadline, we have run additional experiments, such that each class of algorithm is run for three different environments each (rather than one environment each, as was the case in the submitted draft). Among the 6 new experiments, a statistically-significant performance improvement (when using the implicit TD error) was observed in 4 out of the 6 experiments, with statistically-equivalent performance observed in the other two experiments. These new results are included in Figure 3 of the updated draft. Accordingly, these new results strengthen our claim that the difference between the TD errors can have a substantial impact in the performance of the various classes of deep RL algorithms considered.

AuthorsOpenReview ↗

Rebuttal (2/2)

The motivation for applying implicit TD only to average-reward estimation and policy updates, while retaining explicit TD for critic learning, remains unclear.

In this work, we limit our exploration to classes of algorithms that use the TD error to compute quantities beyond the value function, such that we explore the implications of using either TD error to compute these non-value function quantities. We agree with the Reviewer that exploring how the implicit TD error could be used to update the critic is a captivating research question, and indeed mention this as a direction for future work in Section 5 of our paper (see lines 480-485 of the submitted draft; lines 516-522 of the updated draft).

Could the authors clarify the proof of Proposition 3.5? Eq. (15) appears to be sufficient but not necessary for the two batch-averaged TD errors to be equal, since terms from different samples may cancel out.

This is a good catch by the Reviewer. We have updated Equation (15) to state a necessary condition.

Since implicit TD is computed using parameters updated by explicit TD, what is the benefit of introducing implicit TD? Also, why is it only used for average-reward and policy updates, but not for critic learning?

The benefits can be most clearly seen in Figures 2 and 3. In Figure 2, we see that using the implicit TD error to update the average-reward estimate mitigates the instability caused by updating the estimate with the explicit TD error (see Appendix A for a more detailed discussion). In Figure 3, we see that the choice of the implicit vs. explicit TD affects the performance of various classes of algorithms (in particular, we see a statistically-significant increase in performance in the majority of the experiments considered when using the implicit TD error). For clarity, we do not claim that using the implicit TD error is always preferable, only that the choice between the two TD errors should be made purposefully in any deep RL setting in which the TD error is used to compute quantities beyond the value function.

We agree that exploring how the implicit TD error can be used to update the critic is a compelling research question. However, addressing this research question would be a substantial (and worthwhile) effort that would require its own standalone research project. Accordingly, we leave such efforts to future work.

We again thank the reviewer for their review of our paper. We are happy to further discuss with the reviewer as needed.

Thank you for the clarification. However, my major concern regarding the motivation remains unresolved. That is, the authors seek to prove that two inherently different quantities are different, despite being aware of their conceptual distinction. I also agree with Reviewer N14d's interpretation of Sutton (1988). As the authors themselves acknowledge, several important algorithmic questions remain open. I hope the authors can further explore these questions in future work. For now, I will keep my original rating.

We thank the reviewer for their response.

We emphasize that the motivation for this work is to bring awareness to the fact that there exists a distinction between these two quantities, given that these two quantities have historically been conflated in the literature.

For example, consider Algorithm 6 (Differential Q-learning with linear function approximation) in Wan et al. (2021). This is a non-tabular algorithm that uses the explicit TD error to update the average-reward estimate. As we have shown in our work, the two TD errors are not generally equivalent in linear function approximation settings. Hence, by using the explicit TD error to update the average-reward estimate, Algorithm 6 of Wan et al. (2021) conflates the two interpretations of the TD error. The same situation can be seen in Algorithm 3 of Naik et al. (2024), which uses the explicit TD error to update the average-reward estimate in a DQN-style algorithm. Altogether, these two works illustrate that the two interpretations of the TD error continue to be conflated in non-tabular settings, thereby emphasizing the need for our work, in which we are the first to show that these interpretations of the TD error are not generally equivalent in non-tabular settings and can result in the instability shown in Figure 2.

Moreover, as per our discussion with Reviewer N14d, Sutton (1988) offers several distinct interpretations of the notion of “the difference between temporally successive predictions”, including one that is captured by the implicit TD error discussed in our work.

We again thank the reviewer for their time.

Reviewer_UUMaOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper focus on interpretation of temporal difference error in deep reinforcement learning. The explicit TD error defined as the difference between bootstrapped target and the critic prediction. The implicit TD error defined as the change in the critic prediction after a parameter update. Authors theoretically analyze why these two quantities can differ under function approximation, nonlinear networks, and batch updates. They further investigate the use of implicit TD errors in differential Q-learning, reward-centering algorithms, and A2C, where TD errors are used to update quantities beyond the critic. Experiment results demonstrate differences between the two TD errors and show improvements in learning stability and performance.

Critical strengths
  1. This paper discuss a distinction between the learning error and the actual learning response of a value function approximator. I find this distinction interesting, as it encourages a more careful examination of how TD errors should be interpreted and used in rl algorithms.

  2. This paper proposed use of implicit TD errors for average reward estimation and actor updates is conceptually simple and may provide useful insights for designing more stable learning mechanisms.

  3. The experiments cover several RL settings, including Q-learning, reward centering, and actor-critic. The results suggest that implicit TD errors can improve learning performance and stability.

Critical weaknesses
  1. The implicit TD errors can improve learning stability. However, the implicit TD error also depends on the learning rate and optimizer. A more controlled comparison with normalized or adaptively scaled explicit TD errors would help explain the mechanism underlying the observed improvements.

  2. Although paper evaluates several rl algorithms, each algorithmic comparison is conducted on a limited number of environments. Additional experiments would help establish the generality of the conclusions.

Questions for authors
  1. The experiments show promising results on Breakout, Pong, and Ant. Have the authors observed similar improvements across additional environments or different critic architectures and optimization settings?

  2. Could the authors further clarify whether the observed benefits arise specifically from the implicit TD error formulation or can be explained by gradient-dependent scaling or adaptive normalization effects?

2 comments

We thank the reviewer for their review of our work. Below we address the reviewer’s concerns and questions. Given that the questions asked by the reviewer also encompass the concerns, we only respond to the questions:

The experiments show promising results on Breakout, Pong, and Ant. Have the authors observed similar improvements across additional environments or different critic architectures and optimization settings?

Since the submission deadline, we have run additional experiments, such that each class of algorithm is run for three different environments each (rather than one environment each, as was the case in the submitted draft). Among the 6 new experiments, a statistically-significant performance improvement (when using the implicit TD error) was observed in 4 out of the 6 experiments, with statistically-equivalent performance observed in the other two experiments. These new results are included in Figure 3 of the updated draft.

Altogether, these experiments cover a diverse range of experimental settings (e.g. discrete vs. continuous action-spaces, batch updates vs. no batch updates, discounted RL vs. average-reward RL, etc.). While our experiments do not cover all possible architectures and optimization settings, we argue that, collectively, the observed results support our claim that the choice of which TD error to use matters in deep RL settings.

Could the authors further clarify whether the observed benefits arise specifically from the implicit TD error formulation or can be explained by gradient-dependent scaling or adaptive normalization effects?

In our experiments, we kept all aspects of the experimental setup and algorithm consistent except for using the explicit vs. implicit TD error to compute a quantity beyond the value function (the average-reward estimate or the advantage estimate). Accordingly, we isolated the effect of using the implicit vs. explicit TD error, such that any gains in performance (if any) can be attributed to which of the TD errors was used. Additionally, we note that for the differential and reward centering algorithms, we provide a theory-based discussion (Appendix A in the submitted and updated drafts) that motivates why the using the implicit TD error mitigates the instability caused by using the explicit TD error to update the average-reward estimate (thereby strengthening the argument that the observed gains are due to the implicit TD error formulation). For the A2C algorithms, we interpret any observed gain in performance as being due to using a better advantage estimator.

While the additional experiments suggested by the Reviewer would be insightful, they would not change the conclusions or claims of the paper (that the difference between the two TD errors exists and can affect performance). Accordingly, we leave such efforts to future work.

We again thank the reviewer for their review of our paper. We are happy to further discuss with the reviewer as needed.

Thank you for adding experiments across three environments per algorithm class. This addresses my concern about limited environment coverage. However, the results do not establish that the implicit TD error is a better advantage estimator. It measures the critic’s update response and depends on gradient scaling and the optimizer, while the paper provides no A2C-specific theory or scale-controlled comparison to show more accurate advantage estimates. I appreciate the expanded experiments, but this concern remains, so I will maintain my original rating.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Discussion Summary

We thank the reviewers and AC for their time and consideration of our paper. As the rebuttal period comes to an end, we wanted to summarize the key points brought up during the discussion and the key updates made to the paper:

New empirical results: Some reviewers wanted additional experiments to be conducted to further validate the empirical results shown in the submitted draft. In this regard, we ran additional experiments such that each class of algorithm was run for three different environments each (rather than one environment each, as was the case in the submitted draft). Among the 6 new experiments, a statistically-significant performance improvement (when using the implicit TD error) was observed in 4 out of the 6 experiments, with statistically-equivalent performance observed in the other two experiments. These new results are included in Figure 3 of the updated draft.

Historical precedent of conflation between the two TD errors: Some reviewers wanted clarification on the historical precedent for the conflation between the two TD errors. In this regard, we note that the algebraic relationship between the TD error and changes in tabular Q-values that is exploited in Wan et al. (2021) and Naik et al. (2024) precisely corresponds to the implicit TD error (i.e., Equation (6)) discussed in our work. Moreover, Algorithm 6 in Wan et al. (2021) and Algorithm 3 in Naik et al. (2024) collectively illustrate a historical precedent for conflating the two interpretations of the TD error in non-tabular settings. More specifically, in these algorithms, the authors use the explicit TD error to update the average-reward estimate, despite using the implicit TD error to show convergence in the tabular setting, thereby showing a conflation between the two TD errors.

Importantly, we emphasize that the primary motivation and benefit of this work is that it brings to light the difference between the two interpretations of the TD error, as well as the practical consequences of conflating the two interpretations. Most notably, we identify a previously undisclosed critical issue with average-reward and value-based reward centering algorithms (the instability shown in Figure 2), as well as provide a theoretically-motivated solution for this issue. This hence motivates the need for this work, so that our findings can be communicated to the broader average-reward and reward centering communities.

We again thank the reviewers and AC for their time and consideration.

Best,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.