acceptodds
Under review as a conference paper at ICLR 2027

Globally Convergent Offline Reinforcement Learning with Smoothed Bellman Residual Minimization

Abstract

Bellman Residual Minimization (BRM) enforces pointwise Bellman consistency and remains stable under neural parameterization, yet it is rarely used in offline reinforcement learning. Recent work shows that the debiased objective, though nonconvex minimax, admits a Polyak–Łojasiewicz–strongly-concave landscape (Kang et al., 2025), but only for offline inverse RL, and only up to the residual, which is not known to control value error. We propose Off-GLADIUS, which carries this geometry to offline RL, and establish global convergence for the procedure actually run. We then close the remaining gap by proving a local metric-subregularity inequality for the soft Bellman fixed point that converts residual excess risk into a finite-sample bound on value error under the optimal policy, assuming realizability and single-policy concentrability. Empirically, Off-GLADIUS is competitive with CQL and ahead of OptiDICE and prior BRM methods on three discrete-action control benchmarks. Carrying no pessimism term, density ratio, or target network, it also degrades far more gracefully as the logged data turns adversarial.

Primary area:
reinforcement learning
Keywords:
offline reinforcement learning, Bellman residual minimization, global convergence, minimax optimization, value function estimation, Polyak–Łojasiewicz condition, function approximation, offline policy optimization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_2FWQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies Bellman Residual Minimization (BRM) for offline reinforcement learning and proposes Off-GLADIUS, a smoothed and debiased primal--dual method. The main theorem combines a global optimization/generalization result with a local bound that converts Bellman residual into Q⋆Q^\star-estimation error.

The most interesting result is the local estimate

$|Q-Q^\star|_{2,\pi^\star}^2\le $

$ \frac{C_\Pi}{(1-\sqrt{\gamma e^{2\delta/\lambda}})^2} R_b(Q),$

which gives conditions under which a small Bellman residual implies a small Q⋆Q^\star error. The method is tested on three discrete-action control tasks and on a tabular problem where Q⋆Q^\star is known exactly.

Critical strengths
  1. The local residual-to-Q∗Q^* bound is useful and gives a clear dependence on coverage, discounting, entropy temperature, and the local neighborhood.

  2. Off-GLADIUS has a simple objective. It directly minimizes a debiased Bellman residual and does not require a pessimism penalty, density-ratio estimation, or a target network.

  3. The tabular experiment is useful because Q∗Q^* is known and the optimization error can be directly checked.

Critical weaknesses
  1. The novelty appears narrower than stated. Kang et al. [1] already study deep Q⋆Q^\star estimation using minimax Bellman-error minimization, and Uehara et al. [2] study soft minimax offline RL under realizability and partial coverage. The main new contribution therefore seems to be the specific local residual-to-Q∗Q^* bound and its combination with existing optimization results.

  2. I have a concern about the main neural-network theorem. The condition $ J_\xi J_\xi^\top \succeq \mu I$ does not obviously imply strong concavity in the full parameter space of an overparameterized auxiliary network. This point is important because the cited stability result uses a strong-concavity modulus.

  3. The exact debiasing identity requires the conditional expectation $ \xi_Q(s,a)=\mathbb{E}[V_Q(s')\mid s,a].$ It is not clear that the parameterized auxiliary class can represent this function. If not, an additional approximation-error term seems necessary.

  4. The analysis assumes that all iterates remain in compact parameter regions, but the algorithm has no projection step and no bounded-iterate result is proved.

  5. The theory and implementation are different. The theory uses smooth networks and harmonic-step SGDA, while the experiments use ReLU, Adam, fixed learning rates, and multiple inner updates. Therefore, the claim that the theory covers ``the procedure actually run'' seems too strong.

[1] L. Kang, X. Liao, J. Liu, and Y. Luo. \textit{Deep Estimation for Q∗Q^* with Minimax Bellman Error Minimization}. Information Sciences, 648:119565, 2023.

[2] M. Uehara, N. Kallus, J. D. Lee, and W. Sun. \textit{Offline Minimax Soft-Q-learning Under Realizability and Partial Coverage}. NeurIPS, 2023.

Questions for authors
  1. What is the precise new result compared with Kang et al. [1] and Uehara et al. [2]? Is the main novelty the local residual-to-Q∗Q^* inequality?

  2. How does JξJξ⊤⪰μIJ_\xi J_\xi^\top \succeq \mu I imply the strong concavity required by the stability theorem for an overparameterized network?

  3. What assumption guarantees that the auxiliary class can represent or approximate E[VQ(s′)∣s,a]\mathbb{E}[V_Q(s')\mid s,a]? If it cannot, where does this approximation error enter the final theorem?

  4. How is the assumption that all SGDA iterates remain in compact parameter sets justified?

  5. Does the theorem cover the ReLU/Adam/multiple-inner-step algorithm used in the experiments? If not, the statement ``the procedure actually run'' should be weakened.

2 comments

We thank the reviewer for the thorough and careful examination of our theoretical analysis. Below we respond to each of the points raised.

Q1 / W1 (novelty relative to Kang et al. [1] and Uehara et al. [2]).

We thank the reviewer again for these references, which help us position our contribution more precisely.

Both works take the empirical minimax solution as given. Kang et al. [1] analyze the hard-max debiased objective of Antos et al. [3], define Q^∈arg⁡min⁡Qmax⁡OL^(Q,O)\hat Q\in\arg\min_Q\max_O\hat{\mathcal L}(Q,O) without any algorithm for computing it, and convert residual to Q∗Q^* error via the Chen–Jiang route [4] under all-policy concentrability and Bellman completeness of the QQ-class (their Assumptions 4.1–4.2), neither of which we assume. Uehara et al. [2] obtain L2L^2 rates for a Lagrangian objective that is linear in the dual ll. As their Remark 1 notes, this is tractable when LL is an RKHS or linear class, where the problem reduces to ERM, and no optimization guarantee is given for neural dual classes. We have added a discussion of both works to the related work (Appendix B).

Our contribution is thus (a) a convergence guarantee for the first-order procedure itself, by carrying the geometry of Kang et al. [5] to offline RL and combining it with the stability analysis of Kang and Jang [6], and (b) the local inequality converting residual to Q∗Q^* error under single-policy concentrability and realizability alone.

W2 / Q2 (strong concavity for an overparameterized ξ\xi).

The latest arXiv version of Kang and Jang [6] establishes its stability bound under the PL condition in the dual variable, and this is the condition that Assumption 5.3(ii) provides, since JξJξ⊤⪰μIJ_\xi J_\xi^\top\succeq\mu I implies PL in θ2\theta_2. The O(1/N)O(1/N) rate of Theorem 5.4 therefore holds under Assumption 5.3(ii) as stated. The reviewer’s concern applies to the earlier version of [6], which stated the bound under ρ\rho-strong concavity in the dual. That condition cannot hold for an overparameterized ξ\xi, because strong concavity in θ2\theta_2 requires Jξ⊤Jξ⪰μIJ_\xi^\top J_\xi\succeq\mu I, and this matrix is rank-deficient when the number of parameters exceeds the number of samples. We have made the dependence on PL in the dual explicit in Appendix A.

W3 / Q3 (approximating E[VQ(s′)∣s,a]\mathbb E[V_Q(s')\mid s,a]).

Kang et al. [5, Theorem 10] establish two-sided PL of the empirical objective, and under this condition stochastic alternating GDA converges to a global saddle point (Yang et al. [8, Theorem 3.3]). The auxiliary iterates therefore converge to the inner maximizer, which by the bias–variance decomposition is the best approximation of E[VQ(s′)∣s,a]\mathbb E[V^Q(s')\mid s,a] within the auxiliary class.

W4 / Q4 (bounded iterates).

Liu et al. [7, Theorem 6 and 7] show that once the PL$^*$ condition holds on a ball around the initialization whose radius exceeds a threshold set by the initial loss and the PL constant, every iterate with a sufficiently small step size stays in that ball. Since our analysis covers a range of parameterizations, including linear classes and wide neural networks, we need to state only the conditions that the proof uses rather than a mechanism specific to one of them. Containment of the iterates is one such condition, and we agree that it should have been stated explicitly rather than left in the preamble of Assumption 5.3. We have therefore revised Assumption 5.3 to state it as condition (iii).

W5 / Q5 (theory vs. implementation).

There is a gap between the analysis and the practical techniques standard in deep RL, and our experiments use those the reviewer lists (ReLU, Adam with a learning-rate schedule, multiple inner steps). We have therefore weakened “the procedure actually run” and similar phrasing and added a “Practical implementation” paragraph in Appendix C. An additional experiment shows that plain SGDA reaches the same final return as Adam on all three tasks (Appendix C.3, Figure 3).

Revisions (marked in blue).

  • Appendix B (W1 / Q1): a discussion of [1] and [2] in the related work.
  • Appendix A (W2 / Q2): the dependence on PL in the dual made explicit.
  • Assumption 5.3 (W4 / Q4): containment of the iterates moved from the preamble and stated as condition (iii).
  • Throughout (W5 / Q5): “the procedure actually run” and similar phrasing weakened.
  • Section 4 and Appendix C (W5 / Q5): a new “Practical implementation” paragraph, referenced after Algorithm 1, and a plain SGDA vs. Adam comparison (Figure 3).

References.

[3] Antos, Szepesvári, Munos, Machine Learning, 2008.

[4] Chen & Jiang, ICML 2019.

[5] Kang, Yoganarasimhan, Jain, EC 2025.

[6] Kang & Jang, arXiv:2508.18741.

[7] Liu, Zhu, Belkin, ACHA, 2022.

[8] Yang, Kiyavash, He, arXiv:2002.09621.

We are happy to answer any further questions on the revised parts, and we hope the revisions can be reflected in the evaluation.

As the discussion period ends today, we would be grateful if you could let us know whether our response and the revised paper address your concerns. If any concern remains, we would be glad to clarify it before the deadline; if not, we would appreciate it if you could take the revision into account in your evaluation.

Reviewer_wTLdOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Off-GLADIUS, an extension of GLADIUS from offline inverse reinforcement learning to the offline reinforcement learning setting. It develops a theoretical framework for analyzing the global convergence of smoothed Bellman residual minimization using off-policy data and presents structured convergence proofs. The proposed method is evaluated against several state-of-the-art baselines, with the reported results demonstrating competitive performance on the considered tasks.

Critical strengths

S1. The paper is clearly written and well organized. The problem formulation and proposed solution are presented in a systematic manner. In particular, the theoretical framework is logically structured, and the inclusion of a proof sketch in the main text helps readers follow the central convergence argument. This clarity is particularly valuable given the technical nature of the contribution.

S2. The paper includes comparisons with relevant offline RL algorithms, and the reported return curves indicate competitive performance toward the end of training. These comparisons provide useful empirical evidence for evaluating the proposed method.

Critical weaknesses

W1. The empirical evaluation is limited in scope and reveals weaknesses in early-stage optimization. The experiments in Section 6.1 cover only three classic-control tasks from Gymnasium, without corresponding evaluations on other relevant offline RL benchmarks. Moreover, the return curves exhibit slow initial improvement and instability during the early iterations. Although the abstract and introduction characterize the method as competitive with CQL, OptiDICE, SBEED, and BRM-FPI, the reported results appear to support this claim primarily toward the end of training. These limitations weaken the empirical evidence supporting the method's competitiveness and generalizability.

W2. There is a discrepancy between the optimization objective analyzed in the paper and the loss function used in the default implementation. Algorithm 1 minimizes E[ℓ]\mathbb{E}[\ell], where ℓ\ell denotes the single-sample objective consisting of the squared TD error minus a dual correction term. However, the default implementations for CartPole and LunarLander minimize E[∣ℓ∣]\mathbb{E}[|\ell|]. The relevant code can be found at off-gladius-code-632A/methods/off_gladius/off_gladius_cartpole.py:265, with the default configuration for LunarLander specified at off-gladius-code-632A/methods/off_gladius/off_gladius_lunarlander.py:455. Taking the absolute value changes the optimization objective and can reverse the gradient contribution when ℓ<0\ell < 0. Consequently, the alternating updates for QQ and ξ\xi no longer directly correspond to the original unified minimax objective. Therefore, the BRM risk identification and subsequent convergence results established in Appendix A.1 for the original objective do not directly extend to the default implementation. This discrepancy raises a substantive concern regarding the alignment between the theoretical analysis and the experimental implementation.

Questions for authors

Q1. Why the empirical evaluation is restricted to three Gymnasium classic-control tasks rather than including a broader set of offline RL benchmarks, such as OGBench? In particular, how should the reported results be interpreted in terms of generalizability to more diverse offline RL settings?

2 comments

We thank the reviewer for the careful reading, and for the constructive suggestions on the scope of the evaluation. We have uploaded a revised paper in which the changes are marked in blue.

W1 / Q1 (scope of the evaluation and early-stage behavior).

Scope. The theoretical guarantee of our method holds for finite action spaces, where the log-sum-exp is exact and Lemma A.2 (full action support) and the constant of Lemma A.5 apply. Continuous-control benchmarks such as OGBench therefore lie outside the setting of our analysis. Extending the method to continuous actions is a substantial problem in its own right, enough for a separate paper, and it is the direction of our future work (Section 7).

Additional experiment. Within discrete-action offline RL, we have broadened the evaluation with three MinAtar games under an analogous mixed-dataset protocol, comparing against all baselines of Section 6.1 and Behavior Cloning (mean ± std over 5 seeds):

GameCQLOptiDICEBRM-FPISBEEDBCOff-GLADIUS
Breakout7.99 ± 0.422.69 ± 0.423.11 ± 1.594.91 ± 1.659.95 ± 0.4812.09 ± 0.70
Freeway18.61 ± 0.5629.38 ± 3.505.01 ± 5.8845.55 ± 1.5934.03 ± 1.2848.07 ± 2.63
Space Invaders22.73 ± 0.7816.02 ± 0.886.52 ± 1.8426.49 ± 0.8916.90 ± 2.0738.00 ± 2.19

All methods use a learning rate of 3×10−43\times10^{-4}, a batch size of 256, and 10510^5 training steps (5×1035\times10^3 for BC), with a single hyperparameter swept per method (Appendix C.9). Off-GLADIUS attains the highest return on all three games and is the only method that exceeds BC on every game. These results have been added to Section 6.

On generalizability, CartPole, Acrobot, and LunarLander are the standard environments on which theory-oriented work on soft Bellman operators and minimax offline RL checks that its guarantees carry over to practice (Appendix F, “Empirical Validation on Discrete Control Benchmarks”), and the MinAtar results extend that check to pixel-based games. Together, these results show that Off-GLADIUS is competitive across the discrete-action regime.

Early-stage behavior. The slow start is largely explained by the dual correction, which is accurate only once ξ\xi tracks E[VQ(s′)∣s,a]\mathbb E[V^Q(s')\mid s,a]. Early in training ξ\xi lags the moving VQθ1V^{Q_{\theta_1}} (see our response to W2), and keeping ξ\xi closer to its inner optimum with larger KξK_\xi improves return and across-seed stability up to Kξ=10K_\xi=10 (Appendix D.2). We agree there is room to improve the early phase further. The contribution of the paper is a guarantee on where the procedure converges, and this is where its significance lies.

W2 (implemented E[∣ℓ∣]\mathbb E[|\ell|] vs. analyzed E[ℓ]\mathbb E[\ell]).

The implementation uses a safeguard that departs from the analyzed objective. We agree that this should be made explicit, and the revised paper now describes it in Appendix C.3.

Why the implementation uses ∣ℓ∣|\ell|. At initialization, ξ\xi is far from its inner optimum E[VQ(s′)∣s,a]\mathbb E[V^Q(s')\mid s,a], so the correction term can exceed the TD term and the objective becomes negative. With ξ\xi lagging, the objective has no lower bound in that direction, and minimizing it diverges rather than reducing the Bellman residual. We therefore use ∣ℓ∣|\ell| to block this negative direction, and take KξK_\xi dual steps per primal step so that ξ\xi converges toward its inner optimum and the correction becomes accurate.

It is a stabilizer, not an objective. When ξ\xi is at its inner optimum mQ:=E[VQ(s′)∣s,a]m_Q:=\mathbb E[V^Q(s')\mid s,a], writing bQ:=TQ−Qb_Q:=\mathcal TQ-Q for the Bellman residual, ∣ℓ∣=∣bQ∣⋅∣bQ+2γ(VQ(s′)−mQ)∣|\ell|=|b_Q|\cdot|b_Q+2\gamma(V^Q(s')-m_Q)|, so E[∣ℓ∣]\mathbb E[|\ell|] vanishes exactly when the Bellman residual does, and Q∗Q^* remains a minimizer under realizability. The two objectives differ only in how they weight nonzero residuals, and on samples with ℓ≥0\ell\ge0 their gradients coincide.

Revisions.

  • Section 6 and Appendix C.9 (W1/Q1): the MinAtar results above, with their experimental setup and hyperparameters.
  • Appendix C.3 (W2): a new “Practical implementation” paragraph describing the ∣ℓ∣|\ell| safeguard and why it is needed.
  • Section 4 (W2): a sentence after Algorithm 1 pointing to this paragraph, so that the departure from the analyzed objective is stated in the main text.

We appreciate the reviewer’s close look at the experiments and the code. We would be glad to clarify anything further.

I thank the authors for the response and the revised manuscript, which remains very clearly written. The revision addresses part of my concerns.

On the evaluation, the revision adds three MinAtar games with a BC reference (Section 6, Table 2, Appendix C.9), and the response explains why continuous-control benchmarks fall outside the finite-action analysis. This is a useful extension within the setting the theory covers.

Regarding W2, the ∣ℓ∣|\ell| safeguard is now described in Appendix C.3, with a pointer after Algorithm 1. This explains why ∣ℓ∣|\ell| is used, but not why the convergence guarantee carries over to it. The equivalence argument holds only when ξ\xi is at its inner optimum, whereas the safeguard is introduced for the phase where ξ\xi is not, and no comparison between ℓ\ell and ∣ℓ∣|\ell| is reported.

Since the central claim is a guarantee for the algorithm as run, while the experiments optimize a different objective from the one analyzed, I am keeping my current score.

Reviewer_7hZ5OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses a long-standing open question in offline reinforcement learning: whether smoothed Bellman Residual Minimization (BRM) can achieve global convergence to the optimum under off-policy data, and whether this can yield a meaningful Q∗Q^*-estimation guarantee. The authors propose Off-GLADIUS, which extends GLADIUS from offline inverse RL to offline RL via a QQ-function-centric reformulation, and establish global convergence for the procedure actually run—not an assumed minimizer. A local metric-subregularity inequality converts the residual guarantee into an explicit Q∗Q^*-estimation bound under realizability and single-policy concentrability. Empirically, Off-GLADIUS is competitive with CQL and ahead of OptiDICE on three discrete-action control benchmarks, is the only one of the three BRM methods that remains stable on every task, and degrades more gracefully than CQL as logged data turns adversarial.

Critical strengths

[S1] The research question is important and long-standing: whether BRM can achieve global convergence in offline RL has remained unresolved, and this paper directly targets that gap by supplying the missing optimization guarantee.

[S2] The theoretical contribution is complete and layered: the paper proves convergence of the population Bellman-residual excess risk, converts it into an explicit Q∗Q^*-estimation bound, and clearly identifies the conditions under which the conversion holds.

Critical weaknesses

[W1] The empirical evaluation is too narrow to support the generality claims. All experiments are conducted on three low-dimensional discrete-action Gymnasium tasks (CartPole-v1, Acrobot-v1, LunarLander-v3) plus an 8×88\times 8 tabular FrozenLake. For a paper whose central claim is global convergence of offline RL, this is a significant gap. The paper itself acknowledges that the guarantee is limited to finite action spaces, but the empirical section does not test whether the algorithm remains stable and competitive in the continuous-control regime where the theory does not directly apply. This is precisely the regime in which the community will want to know whether Off-GLADIUS is usable, and where prior BRM methods have been most extensively evaluated. In particular, the paper should have conducted experiments on a broader set of environments to demonstrate the algorithm's generality—e.g., hopper, walker2d, halfcheetah as considered by OptiDICE, and other environments in D4RL.

[W2] Missing modern and foundational baselines. The comparison set is CQL, OptiDICE, BRM-FPI, and SBEED. While BRM-FPI and SBEED are the appropriate prior BRM methods to isolate the optimization gap, the absence of Decision Transformer (DT; Chen et al., 2021), Behavior Cloning (BC), and Diffusion-QL (Wang et al., 2022) means the paper does not situate Off-GLADIUS relative to the dominant paradigms in offline RL. DT and Diffusion-QL both bypass Bellman residual optimization entirely and perform strongly on D4RL; BC is the standard lower-bound reference. Without these, the reader cannot judge whether the Bellman-consistency route remains practically competitive, and the paper's value proposition—that one can obtain a global convergence guarantee without sacrificing empirical performance—is left untested against the strongest alternatives.

Questions for authors

See Weaknesses

2 comments

We sincerely thank the reviewer for the careful review. Below we respond to W1 (continuous actions) and W2 (baselines).

W1 (continuous actions). Our guarantee is limited to finite action spaces, and the paper states this explicitly (Sections 1, 2.1, 7 and Appendix F). We address the continuous case in three parts.

(i) Discrete actions are practically relevant. For example, clinical offline RL uses discrete dose grids (25 actions in Komorowski et al., 2018; Raghu et al., 2017), and LLM policies are softmaxes over a finite vocabulary whose logits can be read as token-level Q-values [1]. Dynamic discrete choice models in econometrics also share the same soft Bellman equation (Ermon et al., 2015; Kang et al., 2025).

(ii) Why the extension is not immediate. Off-GLADIUS needs no policy network because the policy and the soft value are closed-form functions of QQ: πQ∝eQ\pi^Q \propto e^{Q} and VQ(s)=log⁡∑a∈AeQ(s,a)V^Q(s) = \log \sum_{a \in \mathcal{A}} e^{Q(s,a)}. Our analysis uses this exact VQV^Q in the debiasing identity (Lemma 3.1), the Jacobian conditioning (Lemma A.10), and the residual-to-Q∗Q^{\ast} bound (Lemma A.5). With continuous actions, VQ(s)=log⁡∫AeQ(s,a) daV^Q(s) = \log \int_{\mathcal{A}} e^{Q(s,a)}\,da has no closed form, and each standard workaround gives up one of three ingredients: an exact (or unbiased) VQV^Q, realizability, or a QQ-only formulation behind our global convergence.

  • Sampling. A natural approach draws a1,…,aMa_1, \dots, a_M from a proposal q(⋅∣s)q(\cdot \mid s) and uses V^(s)=log⁡(1M∑m=1MeQ(s,am)/q(am∣s))\hat V(s) = \log\big(\frac{1}{M} \sum_{m=1}^{M} e^{Q(s,a_m)} / q(a_m \mid s)\big). Since log⁡\log is concave, Jensen's inequality gives E[V^(s)]≤VQ(s)\mathbb{E}[\hat V(s)] \le V^Q(s), with a gap that depends on QQ. The auxiliary ξ\xi removes the variance of the target (Lemma 3.1) but not this bias in its mean, so the debiased objective is minimized by the fixed point of a biased operator rather than by Q∗Q^{\ast}. For practical MM, the gap shrinks only if qq concentrates on high-QQ actions (Haarnoja et al., 2017; [2]). Offline, such a qq is drawn to actions the data do not cover, where QQ is extrapolated (Fujimoto et al., 2019; Kumar et al., 2019), and qq becomes a third learned component outside our (Q,ξ)(Q, \xi) game.
  • Discretization. Discretizing each action dimension into KK bins yields a finite action set, but the number of joint actions grows as KdK^d: with K=11K = 11, Hopper (d=3d = 3) has 1,331 actions, while Walker2d and HalfCheetah (d=6d = 6) have about 1.8 million, each requiring a QQ output and a log-sum-exp at every next state.
  • A separate policy. SBEED (Dai et al., 2018) avoids the integral by learning π\pi as a separate variable, at the cost of an extra policy variable and a guarantee only to a stationary point.

(iii) Scope and ongoing work. As the reviewer notes (S2), we view the finite-action result as self-contained: it supplies the missing global convergence guarantee for BRM in offline RL (S1) and converts it into a Q∗Q^{\ast}-estimation bound. Moving a discrete-action method to continuous actions has typically been a contribution in its own right (DQN to DDPG [3]; MuZero to Sampled MuZero, [4]). We are pursuing this extension, starting with discretization on low-dimensional continuous tasks (continuous CartPole, Pendulum, continuous LunarLander), where KdK^d stays small. Within discrete actions, we have broadened the evaluation with three MinAtar games (Section 6, Table 2), where Off-GLADIUS exceeds BC and all baselines.

W2 (baselines). We ran BC, DT and Diffusion-QL on our three tasks with the same 7:3 dataset and protocol. For discrete actions, DT uses a classification head, as in its Atari experiments [5], and Diffusion-QL [6] maps each sampled continuous action to the nearest discrete action.

Off-GLADIUSCQLOptiDICEBCDTDiffusion-QL
CartPole-v1491.4 ± 5.6490.8 ± 16.9484.1 ± 25.0496.1 ± 6.4500.0 ± 0.0500.0 ± 0.0
Acrobot-v1−93.5 ± 5.6−94.8 ± 11.6−180.2 ± 34.2−154.2 ± 22.0−106.9 ± 3.0−81.5 ± 0.8
LunarLander-v3215.9 ± 12.6192.5 ± 11.5−56.8 ± 19.3237.4 ± 4.9246.8 ± 2.8256.0 ± 2.4

First three columns from Table 1; BC, DT and Diffusion-QL over 5 seeds. Imitation is strong on this PPO-heavy mixture, but Off-GLADIUS remains competitive throughout: near the 500 maximum on CartPole-v1, second only to Diffusion-QL on Acrobot-v1 (where BC drops to −154.2), and above the solved threshold of 200 on LunarLander-v3.

We hope these responses address the reviewer’s concerns, and we are happy to answer any further questions.

References not in the submission.

[1] Rafailov et al., COLM 2024.

[2] Kalashnikov et al., CoRL 2018.

[3] Lillicrap et al., ICLR 2016.

[4] Hubert et al., ICML 2021.

[5] Chen et al., NeurIPS 2021.

[6] Wang et al., ICLR 2023.

Thank you for the response and the updated manuscript. However, I am afraid the new results do not fully address my concerns and are somewhat selective.

While the authors added DT, Diffusion-QL, and BC baselines, Table 1 shows that Off-GLADIUS is actually outperformed by BC, DT, and Diffusion-QL on LunarLander-v3, while CartPole-v1 is completely saturated.

Most critically, in the paper's claimed "advantage scenario" (Coverage Failure, Section 6.2), these newly added strong baselines (BC, DT, Diffusion-QL) are conspicuously absent. The authors argue that Off-GLADIUS is robust to degraded data, yet this is only tested against CQL and OptiDICE. Without evaluating BC or DT under coverage failure, the claim that this method offers a practical advantage over simple imitation or generative models remains unproven.

Therefore, I maintain my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.