- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates two formulations of the temporal-difference error: the conventional Bellman residual (explicit TD error) and the change in value predictions induced by a learning update (implicit TD error). The authors show that these quantities can differ under function approximation, and investigate using implicit TD errors for average-reward updates in differential Q-learning and reward centering, as well as advantage estimation in A2C. The empirical results suggest that the implicit formulation can improve training stability and performance.
Critical strengths
-
The paper presents an interesting perspective on temporal-difference errors, with potentially useful implications for deep RL algorithms.
-
The theoretical analysis is generally intuitive and easy to follow, particularly the characterization of the discrepancy under linear function approximation.
-
The empirical results are promising, showing improvements in stability and learning performance when using implicit TD errors in several RL algorithms.
Critical weaknesses
- Lack of a unified experimental narrative.
The paper investigates three somewhat loosely connected applications of implicit TD errors. For differential Q-learning and reward centering, implicit TD is used to update the average-reward estimate, whereas for A2C, it serves as an alternative advantage estimate. These are substantially different roles, and the paper does not sufficiently connect them or explain why the implicit formulation should be preferable in both settings. The experimental section would benefit from a more coherent motivation and organization.
Furthermore, all three methods still use explicit TD errors to train their critics. The implicit formulation is only introduced for auxiliary updates or actor training. This distinction should be emphasized more clearly throughout the paper, including in its overall framing.
- Limited experimental coverage and insufficient ablations.
Each of the three applications is evaluated on only one environment, making it difficult to assess the generality of the reported improvements. Additional experiments across different tasks would substantially strengthen the empirical claims.
Furthermore, the paper does not sufficiently investigate why implicit TD improves performance. For example, in A2C, implicit TD can be approximated as the explicit TD error scaled by the squared norm of the critic gradient, suggesting that the improvements might result from adaptive advantage scaling rather than a fundamentally better advantage estimator. Comparisons against normalized or clipped explicit TD advantages would help clarify the mechanism.
- Experimental comparisons do not fully isolate the investigated factors.
The experiments in Figure 1 do not clearly isolate the effect of network nonlinearity. In the inverted pendulum experiment, both network architecture and batch updates change, while the Breakout experiment changes network size without changing the number of layers. More controlled comparisons would strengthen the conclusion that increased nonlinearity causes greater divergence between the two TD errors.
Questions for authors
- What is the historical and theoretical justification for the implicit TD error formulation in Equation (6)?
This is my most significant question regarding the paper, and the answer would substantially affect my assessment of its novelty and contribution.
My understanding is that in Sutton (1988), temporally successive predictions refer to predictions made at successive points along a trajectory, rather than predictions of the same state-action pair before and after a learning update. Equation (6), on the other hand, appears to arise from rearranging the tabular Q-learning update rule.
Could the authors clarify whether Equation (6) has previously been established as an alternative formulation of the TD error, and provide the relevant references?
If this formulation is indeed historically established as an interpretation of the TD error, the paper's investigation of their nonequivalence would be well motivated. However, if it is instead a newly introduced quantity, the nonequivalence seems largely expected, as the two expressions measure fundamentally different things. In that case, I believe the paper's central motivation and claimed significance would need substantial reconsideration.
This would not necessarily invalidate the empirical findings, but I would view the work differently: as proposing a new learning signal based on changes in value predictions, rather than resolving a discrepancy between two established interpretations of TD error.
- What explains the improved performance of implicit TD advantage estimation in A2C?
Does the improvement come from a better advantage estimate, or from the implicit gradient-dependent scaling of explicit TD errors? A comparison against advantage normalization, clipping, or gradient-norm-weighted explicit TD errors would be informative.
- How does implicit TD compare against direct average-reward estimation?
For differential Q-learning and reward centering, why should implicit TD updates be preferable to estimating the average reward directly from observed rewards, for example using an exponential moving average? A corresponding baseline would be useful.
- Could the authors clarify Proposition 3.5?
Equation (15) appears to provide a sufficient condition for equality between the batch-averaged TD errors, but not a necessary condition, since different error terms may cancel. Could the authors clarify whether the stated necessity is justified?
- Could the authors clarify the A2C implementation?
Appendix C.3 describes using replay buffers and target networks for A2C, whereas Algorithms 3 and 7 present single-transition updates. How are these reconciled in the implementation, and are any off-policy corrections used for replayed actor updates?
6 comments
Rebuttal (2/3)
The paper does not sufficiently investigate why implicit TD improves performance. For example, in A2C, implicit TD can be approximated as the explicit TD error scaled by the squared norm of the critic gradient, suggesting that the improvements might result from adaptive advantage scaling rather than a fundamentally better advantage estimator. Comparisons against normalized or clipped explicit TD advantages would help clarify the mechanism.
We respectfully disagree with the Reviewer on this. For the differential and reward centering algorithms, in addition to the empirical results, we provide a theory-based discussion on why the implicit TD error mitigates the observed instability (see Appendix A in the submitted and updated drafts). For A2C, we empirically compare the algorithm when using the explicit vs. implicit TD error as the advantage estimate while keeping all other components fixed. This allows us to isolate the effect of using either of the TD errors, and hence, observe which of the two TD errors is a better advantage estimator.
The Reviewer also suggests comparing the implicit TD error against normalized or clipped explicit TD errors. While we agree that this can be an interesting direction for future work, it goes beyond the scope of this work, as our aim is to compare the standard explicit TD error to the implicit TD error. Moreover, even if it is the case that such modifications to the explicit TD error improve performance, it does not invalidate our claim that the choice of which TD error to use matters. Again, we do not claim that the implicit TD error is always preferable, only that the choice between the two TD errors should be made purposefully in any deep RL setting in which the TD error is used to compute quantities beyond the value function.
The experiments in Figure 1 do not clearly isolate the effect of network nonlinearity. In the inverted pendulum experiment, both network architecture and batch updates change, while the Breakout experiment changes network size without changing the number of layers. More controlled comparisons would strengthen the conclusion that increased nonlinearity causes greater divergence between the two TD errors.
While we agree that Figure 1 does not, by itself, clearly isolate the effect of nonlinearity, we disagree that these effects are not discussed elsewhere. In particular, the effect of adding nonlinearities and batch updates are discussed (independently) in Sections 3.2 and 3.3, respectively, such that the results in Figure 1a) validate these claims from an empirical perspective. In Figure 1b) we isolate how adding more hidden units (while keeping the numbers of layers unchanged) causes greater divergence between the two TD errors. While we agree that performing more experiments (e.g. by changing the number of layers) would yield more insights, it would not change the central conclusions from our work: that the difference between the two TD errors increases with more nonlinearity, and that this difference affects performance. As such, we leave such further exploration for future work.
To this end, we have updated Section 5 of the paper to clarify that future work also includes performing additional experiments to study the effects of individual sources of nonlinearity (see lines 514-515 of the updated draft).
Rebuttal (3/3)
What is the historical and theoretical justification for the implicit TD error formulation in Equation (6)?
We are happy to provide the requested clarification:
In Sutton (1988), the TD error is characterized as “the difference between temporally successive predictions” numerous times (e.g. in the abstract of that paper), before being quantified as the difference between a bootstrapped target and a prediction (e.g. see Section 6.4 of that paper). In other words, it is argued in Sutton (1988) (again see, for example, Section 6.4 of that paper) that the difference between temporally successive predictions can be characterized via the explicit TD error. Here, we begin to see the conflation between the two quantities.
This conflation can most recently be seen in [1, 2], where the authors (which include Sutton), characterize the TD error as an n-step version of Equation (6) (i.e., the implicit TD error). More specifically, in both works, the authors use the implicit interpretation of the TD error to prove the convergence of the average-reward estimate of their (tabular) algorithms (e.g. see Equation B.12 in [1]). Crucially, however, they use the explicit TD error in the algorithm pseudocode itself, thereby also conflating the two quantities. For example, despite using the implicit TD error interpretation to show convergence, the pseudocode for Algorithms 1-3 of [2] uses the explicit TD error for the average-reward update.
As such, [1,2] provide historical and theoretical justification for the implicit TD error formulation in Equation (6), as well as show that the two interpretations of the TD error continue to be conflated, thereby motivating the need for our work.
We are happy to further discuss this point with the Reviewer, given its importance to the Reviewer's assessment of our paper.
[1] Wan et al. 2021 “Learning and Planning in Average-Reward Markov Decision Processes.”
[2] Naik et al. 2024 “Reward Centering”
What explains the improved performance of implicit TD advantage estimation in A2C?
In our A2C experiments, we kept all aspects of the experimental setup and algorithm consistent except for using the explicit vs. implicit TD error as the advantage estimate. Accordingly, we isolated the effect of using the implicit vs. explicit TD error, such that any gains in performance (if any) can be attributed to which of the TD errors is a better advantage estimate. While the additional experiments suggested by the Reviewer would be insightful, they would not change the conclusions or claims of the paper. Accordingly, we leave such efforts to future work.
How does implicit TD compare against direct average-reward estimation?
This is a great question, and we are happy to clarify. The primary advantage of using TD error-based average-reward estimation, rather than direct average-reward estimation, is that direct average-reward estimation only works in the on-policy setting, whereas TD error-based average-reward estimation works in both on-policy and off-policy settings (this property was shown in Appendix D of [1]). Hence, for most algorithms of interest, using TD error-based average-reward estimation is the appropriate choice. In our work, we are interested in seeing the effect of using the explicit vs. implicit TD error, so a comparison with direct average-reward estimation is out of scope for the research question being considered.
[1] Wan et al. 2021 “Learning and Planning in Average-Reward Markov Decision Processes.”
Could the authors clarify Proposition 3.5?
This is a good catch by the Reviewer. We have updated Equation (15) to state a necessary condition.
Could the authors clarify the A2C implementation?
This is another good catch by the Reviewer. Appendix C.3 contains a copy/paste error related to the use of replay buffers which has now been resolved in the updated draft. No replay buffers were used; Algorithms 3 and 7 denote the implementations used in our experiments (we note that this is the standard A2C algorithm implementation, with the exception of which TD error is used as the advantage estimate). Given that the standard A2C algorithm is an on-policy algorithm, off-policy correction is not required. We again thank the reviewer for identifying this typo.
We again thank the reviewer for this high-quality review which has undoubtedly helped us improve the paper. We are happy to further discuss with the reviewer as needed.
Rebuttal (1/3)
We thank the reviewer for their insightful and detailed review of our work. Below we address the reviewer’s concerns and questions. Due to space constraints, we split our response across multiple comments:
Each of the three applications is evaluated on only one environment, making it difficult to assess the generality of the reported improvements. Additional experiments across different tasks would substantially strengthen the empirical claims.
Since the submission deadline, we have run additional experiments, such that each class of algorithm is run for three different environments each (rather than one environment each, as was the case in the submitted draft). Among the 6 new experiments, a statistically-significant performance improvement (when using the implicit TD error) was observed in 4 out of the 6 experiments, with statistically-equivalent performance observed in the other two experiments. These new results are included in Figure 3 of the updated draft.
All three methods still use explicit TD errors to train their critics. The implicit formulation is only introduced for auxiliary updates or actor training. This distinction should be emphasized more clearly throughout the paper, including in its overall framing.
We kindly note that the submitted draft of the paper alluded to this distinction in the abstract (line 21 of the submitted draft; line 21 of the updated draft), introduction (line 52 of the submitted draft; line 52 of the updated draft), Section 4 (lines 292 and 296 of the submitted draft; lines 292 and 296 of the updated draft), Section 4.4 (line 424 of the submitted draft; line 430 of the updated draft), and Section 5 (line 469 of the submitted draft; line 496 of the updated draft).
To address the Reviewer’s concern and make this distinction more clear, we have added a clarification statement at the start of Section 4 that states that all three methods still use explicit TD errors to train their critics (see lines 299-302 of the updated draft).
The paper investigates three somewhat loosely connected applications of implicit TD errors. These are substantially different roles, and the paper does not sufficiently connect them or explain why the implicit formulation should be preferable in both settings. The experimental section would benefit from a more coherent motivation and organization.
The aim of Section 4, in which we present the three applications of the implicit TD error, is to show that the choice of TD error can affect performance. We do not claim that the implicit TD error is always preferable, only that the choice between the two TD errors should be made purposefully in any deep RL setting in which the TD error is used to compute quantities beyond the value function. This is precisely what connects the three classes of algorithms discussed in the paper: they all use the TD error to compute quantities beyond the value function (this is mentioned several times in the paper; see our above response). Hence, by showing three district classes of algorithms in which the choice of TD error matters (a claim that has been further validated by the new experiments performed), we are able to show that the implication of the implicit vs. explicit TD error is a universal one that goes beyond a single class of algorithms.