- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper studies Bellman Residual Minimization (BRM) for offline reinforcement learning and proposes Off-GLADIUS, a smoothed and debiased primal--dual method. The main theorem combines a global optimization/generalization result with a local bound that converts Bellman residual into -estimation error.
The most interesting result is the local estimate
$|Q-Q^\star|_{2,\pi^\star}^2\le $
$ \frac{C_\Pi}{(1-\sqrt{\gamma e^{2\delta/\lambda}})^2} R_b(Q),$
which gives conditions under which a small Bellman residual implies a small error. The method is tested on three discrete-action control tasks and on a tabular problem where is known exactly.
Critical strengths
-
The local residual-to- bound is useful and gives a clear dependence on coverage, discounting, entropy temperature, and the local neighborhood.
-
Off-GLADIUS has a simple objective. It directly minimizes a debiased Bellman residual and does not require a pessimism penalty, density-ratio estimation, or a target network.
-
The tabular experiment is useful because is known and the optimization error can be directly checked.
Critical weaknesses
-
The novelty appears narrower than stated. Kang et al. [1] already study deep estimation using minimax Bellman-error minimization, and Uehara et al. [2] study soft minimax offline RL under realizability and partial coverage. The main new contribution therefore seems to be the specific local residual-to- bound and its combination with existing optimization results.
-
I have a concern about the main neural-network theorem. The condition $ J_\xi J_\xi^\top \succeq \mu I$ does not obviously imply strong concavity in the full parameter space of an overparameterized auxiliary network. This point is important because the cited stability result uses a strong-concavity modulus.
-
The exact debiasing identity requires the conditional expectation $ \xi_Q(s,a)=\mathbb{E}[V_Q(s')\mid s,a].$ It is not clear that the parameterized auxiliary class can represent this function. If not, an additional approximation-error term seems necessary.
-
The analysis assumes that all iterates remain in compact parameter regions, but the algorithm has no projection step and no bounded-iterate result is proved.
-
The theory and implementation are different. The theory uses smooth networks and harmonic-step SGDA, while the experiments use ReLU, Adam, fixed learning rates, and multiple inner updates. Therefore, the claim that the theory covers ``the procedure actually run'' seems too strong.
[1] L. Kang, X. Liao, J. Liu, and Y. Luo. \textit{Deep Estimation for with Minimax Bellman Error Minimization}. Information Sciences, 648:119565, 2023.
[2] M. Uehara, N. Kallus, J. D. Lee, and W. Sun. \textit{Offline Minimax Soft-Q-learning Under Realizability and Partial Coverage}. NeurIPS, 2023.
Questions for authors
-
What is the precise new result compared with Kang et al. [1] and Uehara et al. [2]? Is the main novelty the local residual-to- inequality?
-
How does imply the strong concavity required by the stability theorem for an overparameterized network?
-
What assumption guarantees that the auxiliary class can represent or approximate ? If it cannot, where does this approximation error enter the final theorem?
-
How is the assumption that all SGDA iterates remain in compact parameter sets justified?
-
Does the theorem cover the ReLU/Adam/multiple-inner-step algorithm used in the experiments? If not, the statement ``the procedure actually run'' should be weakened.
2 comments
As the discussion period ends today, we would be grateful if you could let us know whether our response and the revised paper address your concerns. If any concern remains, we would be glad to clarify it before the deadline; if not, we would appreciate it if you could take the revision into account in your evaluation.
We thank the reviewer for the thorough and careful examination of our theoretical analysis. Below we respond to each of the points raised.
Q1 / W1 (novelty relative to Kang et al. [1] and Uehara et al. [2]).
We thank the reviewer again for these references, which help us position our contribution more precisely.
Both works take the empirical minimax solution as given. Kang et al. [1] analyze the hard-max debiased objective of Antos et al. [3], define Q^∈argminQmaxOL^(Q,O) without any algorithm for computing it, and convert residual to Q∗ error via the Chen–Jiang route [4] under all-policy concentrability and Bellman completeness of the Q-class (their Assumptions 4.1–4.2), neither of which we assume. Uehara et al. [2] obtain L2 rates for a Lagrangian objective that is linear in the dual l. As their Remark 1 notes, this is tractable when L is an RKHS or linear class, where the problem reduces to ERM, and no optimization guarantee is given for neural dual classes. We have added a discussion of both works to the related work (Appendix B).
Our contribution is thus (a) a convergence guarantee for the first-order procedure itself, by carrying the geometry of Kang et al. [5] to offline RL and combining it with the stability analysis of Kang and Jang [6], and (b) the local inequality converting residual to Q∗ error under single-policy concentrability and realizability alone.
W2 / Q2 (strong concavity for an overparameterized ξ).
The latest arXiv version of Kang and Jang [6] establishes its stability bound under the PL condition in the dual variable, and this is the condition that Assumption 5.3(ii) provides, since JξJξ⊤⪰μI implies PL in θ2. The O(1/N) rate of Theorem 5.4 therefore holds under Assumption 5.3(ii) as stated. The reviewer’s concern applies to the earlier version of [6], which stated the bound under ρ-strong concavity in the dual. That condition cannot hold for an overparameterized ξ, because strong concavity in θ2 requires Jξ⊤Jξ⪰μI, and this matrix is rank-deficient when the number of parameters exceeds the number of samples. We have made the dependence on PL in the dual explicit in Appendix A.
W3 / Q3 (approximating E[VQ(s′)∣s,a]).
Kang et al. [5, Theorem 10] establish two-sided PL of the empirical objective, and under this condition stochastic alternating GDA converges to a global saddle point (Yang et al. [8, Theorem 3.3]). The auxiliary iterates therefore converge to the inner maximizer, which by the bias–variance decomposition is the best approximation of E[VQ(s′)∣s,a] within the auxiliary class.
W4 / Q4 (bounded iterates).
Liu et al. [7, Theorem 6 and 7] show that once the PL$^*$ condition holds on a ball around the initialization whose radius exceeds a threshold set by the initial loss and the PL constant, every iterate with a sufficiently small step size stays in that ball. Since our analysis covers a range of parameterizations, including linear classes and wide neural networks, we need to state only the conditions that the proof uses rather than a mechanism specific to one of them. Containment of the iterates is one such condition, and we agree that it should have been stated explicitly rather than left in the preamble of Assumption 5.3. We have therefore revised Assumption 5.3 to state it as condition (iii).
W5 / Q5 (theory vs. implementation).
There is a gap between the analysis and the practical techniques standard in deep RL, and our experiments use those the reviewer lists (ReLU, Adam with a learning-rate schedule, multiple inner steps). We have therefore weakened “the procedure actually run” and similar phrasing and added a “Practical implementation” paragraph in Appendix C. An additional experiment shows that plain SGDA reaches the same final return as Adam on all three tasks (Appendix C.3, Figure 3).
Revisions (marked in blue).
References.
[3] Antos, Szepesvári, Munos, Machine Learning, 2008.
[4] Chen & Jiang, ICML 2019.
[5] Kang, Yoganarasimhan, Jain, EC 2025.
[6] Kang & Jang, arXiv:2508.18741.
[7] Liu, Zhu, Belkin, ACHA, 2022.
[8] Yang, Kiyavash, He, arXiv:2002.09621.
We are happy to answer any further questions on the revised parts, and we hope the revisions can be reflected in the evaluation.