Tabular TD Descends the Absolute Bellman Error

Temporal-difference (TD) learning is the standard method for policy evaluation in reinforcement learning, but its expected update is not, in general, the negative gradient of any global objective, even in the tabular setting. This has limited the use of optimization ideas to interpret TD and to streamline its analysis. Classical contraction arguments establish convergence without requiring a strict decrease in a global error measure after each one-state update. In this paper, we show that expected tabular TD descends the stationary-weighted absolute Bellman error. As our main contribution, we prove that an expected update of any single state strictly decreases this measure whenever the updated state's Bellman error is nonzero, for any step size in $(0,1]$. This monotonicity yields convergence under standard cumulative-step-size conditions. We then use the same descent property to derive a finite-sample bound on the expected absolute value error of sampled TD under independent sampling. We also give a local-optimization interpretation of this descent property: each expected tabular TD update can be written as the solution of a trust-region subproblem for a linear model of the absolute Bellman error. Although we focus on the tabular case, this perspective may offer a starting point for new algorithms, including ones that use function approximation.