Understanding the theoretical properties of projected Bellman equation, linear Q-learning, and approximate value iteration

In this paper, we study the theoretical properties of the projected Bellman equation (PBE) and two algorithms to solve this equation: linear Q-learning and approximate value iteration (AVI). We identify two sufficient conditions for the existence of PBE solution: strictly negatively row dominating diagonal (SNRDD) assumption and a condition motivated by the convergence of AVI. The SNRDD condition further provides a unified view on the convergence of Q-learning algorithms: tabular, linear, and regularized Q-learning algorithms. We also examine its relationship with the convergence of AVI. Finally, we present examples under $ϵ$-greedy behavior policy showing the limitations of existing analyses of PBE solutions.