1. The theoretical novelty is not adequately distinguished from prior work on regularized GTD and standard Tikhonov regularization.
The paper's regularized MSPBE objective appears to be covered by the general differentiable convex regularization framework of Yu (2017). In particular, the positive-definite, feature-weighted quadratic penalty used by R-GTD satisfies the relevant condition in Yu's Assumption 5.5(c), even when the feature interaction matrix is singular.
Importantly, that assumption requires the origin to be the unique minimizer of a reward-free, rescaled limiting objective. It does not require the minimizer of the original, reward-dependent regularized MSPBE to be the origin. The manuscript's discussion of this restriction therefore appears to mischaracterize the prior result.
I recognize that R-GTD is not the same stochastic algorithm as Yu's method: it introduces an additional slack variable and uses sample-based estimates in its updates. However, the two formulations can target the same regularized solution. The paper needs to explain what additional theoretical guarantee follows from the three-variable formulation, rather than from the already-studied regularized objective.
The geometric characterization also appears closely related to standard Tikhonov regularization. After an appropriate change of coordinates using the positive-definite feature Gram matrix, the optimization problem reduces to ordinary ridge-regularized least squares. The uniqueness of the regularized solution, convergence to the minimum-norm unregularized solution, and the reported asymptotic rate then follow from standard linear-algebra arguments. A GTD-specific interpretation is useful, but the manuscript does not clearly establish what is technically new beyond this classical result.
Reference: Huizhen Yu, "On Convergence of some Gradient-based Temporal-Differences Algorithms for Off-Policy Learning," arXiv:1712.09652, Section 5.3, Assumption 5.5, and Theorem 5.5.
2. The practical value of the selected solution and the additional algorithmic variable is not sufficiently established.
Selecting the minimum-norm MSPBE minimizer does not necessarily produce the best approximation to the true value function. Theorem 4.4 decomposes the approximation error but leaves a solution-selection term uncontrolled. Consequently, the bound does not establish that this term becomes small as regularization vanishes.
The paper's own example in Appendix M illustrates the issue. The limiting R-GTD parameter is (-2, 2), whereas the parameter corresponding to the true projected value is (80/3, 92/3). Both belong to the unregularized MSPBE solution set, yet they represent substantially different approximate values. This does not contradict the theorem, but it shows that minimum-norm selection alone is not an accuracy guarantee.
The experimental evidence also does not isolate the benefit of the proposed three-variable dynamics. The largest improvements occur in stress tests with deliberately large initialization components along null or nearly singular directions. These tests illustrate the shrinkage mechanism, but provide limited evidence of an advantage under more typical initializations or when shrinkage removes useful value components.
Performance on the other benchmarks is mixed. R-GTD and the Yu baseline achieve very similar mean errors on Baird's example, while the Yu baseline achieves a lower mean error on Frozen Lake. More importantly, the implementation of the "Yu (2017)" baseline is not described precisely enough to determine its exact update, regularizer, regularization strength, or tuning procedure.
A particularly important missing comparison is a standard two-variable GTD algorithm using the same feature-weighted regularization penalty as R-GTD. Without this matched-objective comparison, it is difficult to tell whether any improvement comes from the choice of regularizer or from the new three-variable update.
Finally, the experiments emphasize final parameter error. Value-space error, MSPBE, convergence behavior, and distance to each algorithm's own regularized solution would help separate optimization performance from regularization bias.
Overall, the main concern is not that the proposed algorithm is necessarily incorrect, but that its theoretical and practical contributions are not yet convincingly separated from existing regularization techniques.