acceptodds
Under review as a conference paper at ICLR 2027

Lifted Bellman Linear Programming for Offline Reinforcement Learning

Abstract

Offline reinforcement learning (RL) typically trains a critic by minimizing a regression loss against bootstrapped value targets, which are stabilized by target networks updated through exponential moving averages (EMA). Furthermore, multi-step targets incorporate behavior-policy actions and therefore require off-policy correction. We instead impose in-sample Bellman optimality on the critic through inequality constraints. We formulate the Lifted Bellman Linear Program (LBLP), which lifts the linear programming characterization of Bellman optimality to the joint space so that every constraint involves only state-action pairs in the dataset. Its unique minimizer is the in-sample optimal pair, and constraints along -step segments of dataset trajectories leave this minimizer unchanged for any rollout policy and any horizon. Under deterministic dynamics, this minimizer lies between the best return of the dataset and the optimal value. Relaxing the constraints into hinge penalties recovers the same solution above a finite penalty coefficient in the tabular case. Approximate Lifted Bellman Unconstrained Minimization (ALBUM) implements this relaxation with neural networks and detaches the -step rollout targets by stop gradient. Its objective contains no squared regression onto bootstrapped targets, so it trains without target networks or EMA updates. Under deterministic dynamics, the -step constraints allow discounted returns along dataset trajectories to serve as lower bounds without off-policy correction or action chunking, providing a direct mechanism for long-horizon value propagation. On OGBench, ALBUM uses a single critic with a Gaussian policy, matches the average performance of FQL, and substantially outperforms ReBRAC, while using the fewest parameters and the least peak GPU memory among all measured methods.

Primary area:
reinforcement learning
Keywords:
Reinforcement learning, Offline RL, Linear Programming

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_KDbsOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Lifted Bellman Linear Programming (LBLP), an offline reinforcement learning method that learns value functions by enforcing Bellman inequalities rather than minimizing squared Bellman errors. The formulation jointly optimizes Q and V functions and introduces multi-step trajectory constraints to accelerate value propagation. The authors establish that the exact LP and its hinge-penalty relaxation recover the in-sample optimal value functions under specified conditions.

For practical implementation, the paper introduces ALBUM, which approximates the formulation using neural networks and stop-gradient multi-step targets. On OGBench, ALBUM achieves performance comparable to FQL while requiring fewer parameters and less GPU memory, without target networks or critic ensembles.

Critical strengths

The LP formulation provides an interesting alternative to conventional Bellman regression. The theoretical results establish the exactness of the tabular formulation and characterize how multi-step constraints can accelerate value propagation.

The empirical performance and computational efficiency are promising. ALBUM achieves an average success rate of 43.4, comparable to FQL's 43.6, while using approximately half as many parameters and substantially less GPU memory. Its ability to train without target networks or critic ensembles is particularly interesting.

Critical weaknesses
  1. (Major weakness) The main theoretical contribution does not adequately explain the behavior of the actual algorithm.

Propositions 4.3 and 4.7 characterize the exact LP and its tabular relaxation, while Theorem 4.5 analyzes an idealized update. However, ALBUM uses neural function approximation and a detached update that differs from the gradient of the proposed objective. The theoretical results therefore do not establish convergence of the implemented algorithm. More importantly, Appendix L.11 shows that the learned critics do not satisfy the theoretical formulation: the Bellman inequality holds on only 50%–81% of samples, and the learned mean values exceed the empirical optimum in nine of ten environments. Thus, the empirical success of ALBUM cannot be directly explained by the exactness results. The paper needs stronger evidence connecting its theoretical formulation to the observed improvements.

  1. (Major weakness) The main theoretical contribution does not adequately explain the behavior of the actual algorithm.

Propositions 4.3 and 4.7 characterize the exact LP and its tabular relaxation, while Theorem 4.5 analyzes an idealized update. However, ALBUM uses neural function approximation and a detached update that differs from the gradient of the proposed objective. The theoretical results therefore do not establish convergence of the implemented algorithm. More importantly, Appendix L.11 shows that the learned critics do not satisfy the theoretical formulation: the Bellman inequality holds on only 50%–81% of samples, and the learned mean values exceed the empirical optimum in nine of ten environments. Thus, the empirical success of ALBUM cannot be directly explained by the exactness results. The paper needs stronger evidence connecting its theoretical formulation to the observed improvements.

  1. (Major weakness) The paper does not establish why ALBUM can avoid the instability associated with the well-known deadly triad in reinforcement learning.

The deadly triad arises from combining function approximation, bootstrapping, and off-policy learning, potentially causing divergence. ALBUM retains all three ingredients despite replacing squared Bellman regression with inequality-based penalties. Proposition 5.1 provides only a necessary condition preventing unbounded descent along a particular direction, not a general stability or convergence guarantee. This is particularly important because one of the paper's main practical advantages is training without target networks or critic ensembles, which are commonly used to stabilize off-policy learning. Although the experiments demonstrate stability in the evaluated environments, they do not establish whether the proposed formulation fundamentally mitigates the deadly triad. Controlled comparisons on classical off-policy divergence examples would be informative.

  1. (Major weakness) The deterministic dynamics assumption substantially limits the theoretical justification of the practical algorithm.

The interpretation of sampled multi-step returns as valid lower bounds relies on deterministic transitions. Under stochastic dynamics, sampled returns need not satisfy the corresponding expected Bellman inequalities. Although the exact LP formulation is more general, its sampled neural implementation does not inherit the same guarantees. Since stochastic transitions are common in reinforcement learning, the paper should demonstrate how ALBUM behaves under transition noise. Without such evidence, the applicability of the proposed multi-step learning mechanism remains unclear.

Questions for authors

Given that the learned neural critics are generally infeasible for LBLP, what evidence establishes that the empirical performance improvements arise from the proposed LP formulation rather than from the particular heuristic updates? Since ALBUM retains all three ingredients of the deadly triad, can the authors provide theoretical or empirical evidence that its inequality-based updates are more stable than conventional off-policy TD learning? In particular, how does ALBUM perform on classical divergence examples such as Baird's counterexample? How does ALBUM perform under stochastic transitions, where sampled multi-step returns may no longer provide valid lower bounds? What explains the substantial sensitivity to rollout horizons and penalty coefficients despite satisfying Proposition 5.1? Can the authors provide a principled procedure for selecting these parameters?

7 comments
AuthorsOpenReview ↗

Response to Reviewer KDbs (1/6)

weaknesses

Answer W1.

We share this concern, and it is the reason Appendix L.11 compares the learned critic with the in-sample optimum.

Summary.

Our theory does not establish convergence of the neural update. However, the theory characterizes the solution that the update targets, why the stop gradient preserves this solution, and when the update diverges ((i)). A fraction of satisfied constraints near 50% is the reference level for an approximate critic, and the TD critics of FQL and ReBRAC stay near this level, whereas ALBUM satisfies more constraints with smaller violations ((ii)). Learned values above GDG_D also occur for both TD critics on scene and puzzle-3x3 ((iii)).

Details.

(i) Theoretical relation of the implemented update to LBLP. We agree that our theory does not fully establish convergence of the neural update, as is the case for deep off-policy methods with nonlinear function approximation, and Appendix A states this. However, Appendix A also divides our results into two groups according to whether the neural parameterization affects them, and the second group concerns the implemented update itself.

ResultWhat it describesNeural parameterization
Proposition 4.3, Proposition 4.4Solution of the program on MDM_DNot covered
Proposition 4.7Recovery of the solution by the hinge relaxation, tabularNot covered
Theorem 4.5Acceleration of an idealized coordinatewise iterationNot covered
Structure of equation 5Every critic term is evaluated at a pair in DDCovered
Proposition 5.1Boundedness along a common shift of all outputs, which any network with a bias in its final layer admitsCovered
Proposition I.4, Corollary I.5(QD∗,VD∗)(Q^{\ast}_D, V^{\ast}_D) is a stationary point of the detached update field, with QQ and VV as free variables (Definition I.1)Detached rule, not the parameterization

The detached update of ALBUM differs from the gradient of the proposed objective only in the KK-step term. The objective terms, the one-step hinge, and the epigraph hinge, which define the relaxed LBLP without the KK-step term (Lemma 4.2), are differentiated without stop gradient. Under the conditions of Appendix I, (QD∗,VD∗)(Q^{\ast}_D, V^{\ast}_D) remains a stationary point of the detached update field (Proposition I.4, Corollary I.5), so the stop gradient preserves the solution. Proposition 5.1 gives a necessary condition for the boundedness of the neural update, whose violation certifies divergence, as Section 6.1 confirms, but within the region where the condition holds, the condition does not predict how well the update performs. The theory explains the behavior of the implemented update in three respects, the solution that the update targets, why the stop gradient preserves this solution, and when the update diverges.

AuthorsOpenReview ↗

Response to Reviewer KDbs (2/6)

(ii) Fraction of satisfied constraints. Appendix L.11 states that the learned pair is not feasible for LBLP. This statement was intended as a statement of scope. Our exactness results characterize the solution of the program, not the values that the neural critic attains (Appendix A). Infeasibility itself is expected of any approximate critic, just as a TD critic does not satisfy the Bellman equation exactly. However, we realize that Appendix L.11 reported the fraction of satisfied constraints without a reference level and without the size of the violations. Under deterministic dynamics and continuous states, each state of DD carries a single action, and QD∗(s,a)=r+γVD∗(s′)Q^{\ast}_D(s,a) = r + \gamma V^{\ast}_D(s') holds with equality on every pair. Every one-step constraint is therefore active at the solution. Near the solution, the fraction of violated constraints is determined by the sign of the approximation error and not by its size. The reference level is 50% when positive and negative errors are equally likely. To measure the fraction and the size of the violations against this reference, we compared the constraint residual gBg_B of ALBUM with the TD error δ\delta of FQL and ReBRAC. Each quantity is computed with the training target of each method on the same 10510^5 dataset pairs after 1M steps over 4 seeds.

TaskMethodgB≤0g_B \le 0 or δ≤0\delta \le 0 (%)max⁡(gB,0)\max(g_B, 0) or max⁡(δ,0)\max(\delta, 0): Meanmax⁡(gB,0)\max(g_B, 0) or max⁡(δ,0)\max(\delta, 0): p90max⁡(gB,0)\max(g_B, 0) or max⁡(δ,0)\max(\delta, 0): p99
antmaze-largeALBUM67.90.050.130.66
FQL54.30.100.301.07
ReBRAC51.80.070.220.60
sceneALBUM64.10.320.963.59
FQL50.30.591.745.37
ReBRAC49.60.812.516.21
puzzle-3x3ALBUM68.70.371.114.60
FQL48.21.193.3612.01
ReBRAC48.81.825.5516.41

The fraction for the TD critics is close to the reference level, and ALBUM violates fewer constraints, by a smaller mean amount than the positive TD error of both TD critics. In the revision, Appendix L.11 states this comparison level and reports the residual comparison above.

(iii) Learned values above VD∗V^{\ast}_D. We realize that Appendix L.11 closes with the infeasibility of the learned pair, and this framing presented the departure from VD∗V^{\ast}_D as a failure. Since each state of DD carries a single action, VD∗(s)=GD(s)V^{\ast}_D(s) = G_D(s) is the return of the behavior trajectory through ss. The behavior trajectories reach the goal far less often than ALBUM, which succeeds in 92.5% of the episodes on antmaze-large. The value at a state is determined jointly with the values at neighboring states of other trajectories through shared parameters. A value above GDG_D is therefore consistent with generalization from neighboring trajectories with higher returns. This generalization lies outside our exactness results. To determine whether this behavior is specific to the LP formulation, we evaluated the critics of FQL and ReBRAC against GDG_D on the same dataset pairs, with the relative gap mean⁡(Q−GD)/∣mean⁡(GD)∣\operatorname{mean}(Q - G_D)/|\operatorname{mean}(G_D)| for each method. Both TD critics also exceed GDG_D on scene and puzzle-3x3, so a mean above VD∗V^{\ast}_D is not specific to ALBUM (Appendix L.11).

TaskMethodMean of Q−GDQ - G_DRelative gap
antmaze-largeALBUM21.8 ± 0.313.7 ± 0.2%
FQL-2.4 ± 0.2-2.7 ± 0.2%
ReBRAC2.7 ± 0.13.0 ± 0.1%
sceneALBUM217.6 ± 3.243.1 ± 0.6%
FQL94.0 ± 1.633.0 ± 0.6%
ReBRAC63.2 ± 2.322.2 ± 0.8%
puzzle-3x3ALBUM350.3 ± 2.648.6 ± 0.4%
FQL174.5 ± 2.543.0 ± 0.6%
ReBRAC128.3 ± 4.231.6 ± 1.0%
AuthorsOpenReview ↗

Response to Reviewer KDbs (3/6)

Answer W2. We thank the reviewer for this question and would like to clarify the scope of our claims. The paper does not claim that ALBUM resolves or mitigates the deadly triad in general. To our knowledge, such a general guarantee is not available for any deep off-policy method with nonlinear function approximation. Our claim is narrower. ALBUM does not require target networks or critic ensembles. Appendix L.1 of the submission already explains this claim by tracing each device to the failure mode it was introduced to control and showing that the LBLP critic objective does not contain that failure mode, and the revised Section 6.1 states this mechanism in the main text. On Baird's counterexample in the new Appendix L.1.1, TD methods diverge without a target network, whereas ALBUM remains bounded at every step size (Please see (iii)).

(i) Critic ensembles. The upward bias that critic ensembles counteract arises because the bootstrapped target queries the maximum of the estimated values at the next state, either through max⁡a′Q(s′,a′)\max_{a'} Q(s',a') or through a policy action chosen to maximize the estimated QQ. Selecting the largest of noisy estimates biases the target upward, and Double Q-learning (van Hasselt, 2010) and its successors (van Hasselt et al., 2016; Fujimoto et al., 2018) reduce this bias by decorrelating the estimates over which the maximum is taken. In offline RL, the queried action can lie outside the dataset, and ensembles such as SAC-N and EDAC (An et al., 2021) also counteract the resulting overestimation. In LBLP, the bootstrapped quantity is V(s′)V(s'), which is a decision variable rather than the result of selecting an action. The objective lowers VV, and the epigraph constraint V(s)≥Q(s,a)V(s) \geq Q(s,a) bounds it from below only through QQ at dataset actions. The critic update therefore contains no step that selects an action by maximizing estimated values, and it never evaluates QQ at an action outside the dataset.

(ii) Target networks. Target networks in DQN and DDPG (Mnih et al., 2015; Lillicrap et al., 2015) stabilize the regression of QQ onto a target that depends on QQ itself. A squared loss produces an update proportional to the error against a target that moves with the parameters, and the slowly updated copy limits this feedback. Neither LBLP nor its relaxation in ALBUM involves such a regression. Every bootstrapped quantity enters only as a bound inside a hinge, so a violation contributes a gradient with a fixed coefficient, independent of the size of the constraint residual.

(iii) Controlled comparison. Following the reviewer's suggestion, we ran Baird's counterexample with linear features and γ=0.99\gamma = 0.99, and Appendix L.1.1 reports the learning curves. All rewards are zero, so Q∗=V∗=0Q^{\ast} = V^{\ast} = 0 is also the LBLP solution. The table reports the final max error max⁡s,amax⁡(∣Q(s,a)∣,∣V(s)∣)\max_{s,a}\max(|Q(s,a)|, |V(s)|) after 10510^5 SGD steps, mean ± std over 5 seeds, and the initial error is 12. The last column uses the decaying step size ηt=0.1/(1+t/103)\eta_t = 0.1/(1+t/10^3), and div. denotes ∥θ∥>108\Vert \theta\Vert \gt 10^8 in all 5 seeds. Without a target network, TD(0), Q-learning, and clipped double Q-learning diverge. A target network (τ=0.005\tau = 0.005) mitigates divergence, but its error increases throughout training at a step size of 10−310^{-3} and remains above 7 under the decaying schedule. ALBUM and subgradient descent on the relaxed LBLP objective remain bounded in every setting. At a constant step size they settle in a neighborhood of Q∗Q^{\ast} whose radius scales with the step size, as expected for updates on a nonsmooth objective. Under the decaying schedule they reach the smallest errors among all methods. This difference is consistent with the mechanism in (ii). These results concern linear features, and for neural networks with the detached KK-step term, we do not have a stability guarantee. On OGBench, Section 6.1 shows that removing target networks and critic ensembles degrades ReBRAC and FQL, whereas ALBUM trains without either.

Methodη=10−3\eta = 10^{-3}η=10−2\eta = 10^{-2}η=10−1\eta = 10^{-1}ηt\eta_t
TD(0)div.div.div.div.
TD(0) + target(4.6±3.3)×104(4.6 \pm 3.3)\times10^{4}0.031 ± 0.0020.077 ± 0.0017.5 ± 3.4
Q-learning2000 ± 150div.div.div.
Q-learning + target1400 ± 99168 ± 480.27 ± 0.0112.0 ± 1.0
Clipped double Q2000 ± 150div.div.div.
Clipped double Q + target1300 ± 97143 ± 380.26 ± 0.0111.6 ± 0.7
Relaxed LBLP0.027 ± 0.0010.27 ± 0.012.7 ± 0.20.027 ± 0.002
ALBUM0.054 ± 0.0010.53 ± 0.015.4 ± 0.20.053 ± 0.001

[refrences] Other cited works are listed in the references of the paper.

(An et al., 2021) An, G., Moon, S., Kim, J. H., & Song, H. O. (2021). Uncertainty-based offline reinforcement learning with diversified q-ensemble. Advances in neural information processing systems, 34, 7436-7447.

Reviewer_fggUOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Lifted Bellman Linear Programming (LBLP), a primal linear programming formulation for offline reinforcement learning over the joint (Q,V) space. The formulation uses only state-action pairs observed in the dataset and uniquely identifies the in-sample Bellman-optimal pair in the tabular setting. The authors further introduce redundant K-step rollout constraints, which preserve the solution while potentially accelerating value propagation. Based on a hinge relaxation of LBLP, the paper develops ALBUM, a neural offline RL algorithm that uses a single critic, no target network, and stop-gradient rollout targets. Experiments on OGBench show that ALBUM achieves an average performance comparable to FQL and better than ReBRAC, while using fewer parameters and less GPU memory.

Critical strengths
  1. The LBLP formulation is conceptually clean and potentially useful. Lifting the Bellman program to the joint (Q,V) space allows all constraints to remain within the dataset support while directly identifying both value functions.
  2. ALBUM provides a favorable performance-efficiency trade-off. It obtains an average OGBench score close to FQL using only one Q network and one V network, without target networks or critic ensembles. The ablations and computational-cost measurements provide useful empirical support for this advantage.
Critical weaknesses
  1. There is a gap between the tabular theory and the implemented neural algorithm. The exactness and acceleration results apply to the tabular program or an idealized coordinate-wise iteration. In contrast, the detached neural update is not generally the gradient of a scalar LBLP objective. Table 17 also shows that the learned critic does not satisfy all LBLP constraints. This weakens the claim that ALBUM directly implements the relaxed LBLP.
  2. Several important guarantees depend on deterministic dynamics. Under stochastic transitions, a sampled K-step return is not necessarily a lower bound on the in-sample optimal value. The empirical evaluation is also limited to OGBench, so the robustness of the main mechanism under stochastic dynamics remains unclear.
Questions for authors
  1. In what precise formal sense does ALBUM implement the relaxed LBLP, given that the stop-gradient update is generally not the gradient of the stated scalar objective and the learned critic is not LBLP-feasible in Table 17?
  2. How robust is ALBUM when transition or reward stochasticity is introduced? A controlled stochasticity experiment, or a clearer narrowing of the claims to deterministic settings, would materially affect my assessment.
4 comments
AuthorsOpenReview ↗

Response to Reviewer fggU (1/2)

weaknesses

Answer W1 and Q1. We agree that ALBUM does not directly implement the relaxed LBLP in the sense of descending its scalar objective, and that the relation between the two needs to be stated precisely. We address the update and Table 17 separately, and the revision replaces the claim of direct implementation with the relation below.

(i) Relation of the implemented update to the relaxed LBLP. The table below summarizes how Appendix A divides our results according to whether the neural parameterization affects them.

ResultWhat it describesNeural parameterization
Proposition 4.3, Proposition 4.4Solution of the program on MDM_DNot covered
Proposition 4.7Recovery of the solution by the hinge relaxation, tabularNot covered
Theorem 4.5Acceleration of an idealized coordinatewise iterationNot covered
Structure of equation 5Every critic term is evaluated at a pair in DDCovered
Proposition 5.1Boundedness along a common shift of all outputs, which any network with a bias in its final layer admitsCovered
Proposition I.4, Corollary I.5(QD∗,VD∗)(Q^{\ast}_D, V^{\ast}_D) is a stationary point of the detached update field, with QQ and VV as free variables (Definition I.1)Detached rule, not the parameterization

The first group confirms the reviewer's observation. The exactness and acceleration results describe the tabular program and an idealized iteration, and they do not cover the neural update. The second group concerns the implemented loss itself, so these results hold for ALBUM as implemented. The last row addresses the detached update. The detached update differs from the gradient of the scalar objective only in the KK-step term, and under the conditions of Appendix I, (QD∗,VD∗)(Q^{\ast}_D, V^{\ast}_D) remains a stationary point of the detached update field. Proposition I.4 and Corollary I.5 concern the update rule rather than the network, and we do not claim convergence of the neural update. ALBUM therefore approximates the relaxed LBLP in the following precise sense. The loss evaluates the same constraints, the update preserves the solution as a stationary point, and the coefficients satisfy the necessary condition for bounded updates. In the revision, Section 5 states this relation in place of direct implementation, and the abstract, introduction, and conclusion use the same wording.

(ii) Regarding Table 17 of Appendix L.11. We realize that Table 17 reported the fraction of satisfied constraints without the level that the fraction should be compared against. Under deterministic dynamics and continuous states, every one-step constraint is active at the LBLP solution, holding with equality, so near the solution the fraction of violated constraints is determined by the sign of the approximation error and not by its size. Even an accurate approximation of the solution therefore satisfies about 50% of these constraints, which is the level that the fraction should be compared against. For this diagnostic, we reran ALBUM, FQL, and ReBRAC over 4 seeds and compared the constraint residual gBg_B of ALBUM with the TD error δ\delta of FQL and ReBRAC on the same 10510^5 dataset pairs after 1M steps.

TaskMethodgB≤0g_B \le 0 or δ≤0\delta \le 0 (%)max⁡(gB,0)\max(g_B, 0) or max⁡(δ,0)\max(\delta, 0): Meanmax⁡(gB,0)\max(g_B, 0) or max⁡(δ,0)\max(\delta, 0): p90max⁡(gB,0)\max(g_B, 0) or max⁡(δ,0)\max(\delta, 0): p99
antmaze-largeALBUM67.90.050.130.66
FQL54.30.100.301.07
ReBRAC51.80.070.220.60
sceneALBUM64.10.320.963.59
FQL50.30.591.745.37
ReBRAC49.60.812.516.21
puzzle-3x3ALBUM68.70.371.114.60
FQL48.21.193.3612.01
ReBRAC48.81.825.5516.41

The TD critics stay near 50%, whereas ALBUM satisfies more one-step constraints with a smaller mean violation. A fraction below 100% is therefore shared by any approximate critic. In the revision, Appendix L.11 states this comparison level and reports the residual comparison above.

AuthorsOpenReview ↗

Response to Reviewer fggU (2/2)

Answer W2 and Q2. We agree with the reviewer that several of our guarantees depend on the deterministic assumption, and that their extension to stochastic settings is a natural question. We are glad to address both directions that the reviewer suggests, the narrowing of the claims and a controlled stochasticity experiment.

(i) Narrowing of the claims. Proposition 4.4, Theorem 4.5, Proposition I.4, and the interpretation of sampled KK-step returns as lower bounds (Remark D.1) require deterministic dynamics, and the submission qualifies each of these claims where it appears, including the abstract, introduction, and conclusion. Lemma 4.2, Proposition 4.3, Proposition 4.7, and Proposition 5.1 hold for any finite discounted MDP. Appendix A collects these qualifications, and in the revision, a new Limitations paragraph in the Conclusion states them in the main text and lists the extension to stochastic transitions as future work.

(ii) Experiment. The new Appendix L.12 explains in detail that under stochastic transitions the constraints of LBLP remain exact, and that stochasticity enters through the sampled target, which can shift the minimizer of the sampled hinge objective above QD∗Q^{\ast}_D. To measure this effect in practice, each action is perturbed as clip⁡(a+ϵ,−1,1)\operatorname{clip}(a + \epsilon, -1, 1) with ϵ∼N(0,σ2I)\epsilon \sim \mathcal N(0, \sigma^2 I) before execution, following the stochastic environment wrapper in the paper and official code of Abraham et al. (2026). The datasets are recollected under the noisy dynamics with the OGBench collection policies, about 1M transitions each. Storing each action before adding the noise makes the transitions of the datasets stochastic. ALBUM and ReBRAC are trained with their default hyperparameters for 10610^6 steps and evaluated under the same noisy dynamics.

TaskMethodσ=0\sigma = 0σ=0.01\sigma = 0.01σ=0.05\sigma = 0.05σ=0.25\sigma = 0.25
antmaze-largeALBUM92.591.8 (-1%)95.2 (+3%)89.8 (-3%)
ReBRAC95.896.2 (0%)91.8 (-4%)92.5 (-3%)
sceneALBUM93.291.0 (-2%)87.0 (-7%)46.2 (-50%)
ReBRAC52.051.5 (-1%)59.0 (+13%)31.5 (-39%)
cube-singleALBUM89.889.0 (-1%)84.8 (-6%)75.5 (-16%)
ReBRAC92.090.5 (-2%)86.2 (-6%)30.2 (-67%)

The columns σ>0\sigma \gt 0 report the final evaluation over 8 seeds, and the column σ=0\sigma = 0 is taken from our paper. Up to σ=0.05\sigma = 0.05, neither method changes substantially on any task. At σ=0.25\sigma = 0.25, both methods decrease on the two manipulation tasks, ALBUM more on scene and ReBRAC much more on cube-single. If the bias of the sampled KK-step returns were dominant, ALBUM would decrease consistently more than ReBRAC, whose critic uses no sampled KK-step returns. Instead, ALBUM is more robust than ReBRAC on cube-single and keeps the higher success rate on both manipulation tasks at σ=0.25\sigma = 0.25. We therefore interpret the observed performance degradation as arising primarily from the increased task difficulty induced by noise, rather than from the bias in the sampled KK-step returns.

We did not add reward noise in the current revision, and we would like to share our concern with the reviewer. The rewards of OGBench are integer-valued, so additive noise changes the reward scale and the success criterion together, and isolating the intended effect of reward stochasticity requires a more careful design than the action perturbation. If the reviewer considers this experiment important, we are willing to run it on a subset of tasks during the discussion period and report the results. We hope that the experiment and the narrowed statement of the claims address the reviewer's question on robustness under stochastic dynamics.

[reference] (Abraham et al., 2026): Abraham, A. A., Shi, L. X., & Finn, C. (2026). Long-Horizon Q-Learning: Accurate Value Learning via n-Step Inequalities. arXiv preprint arXiv:2605.05812.

questions

Answer Q1. Please see our answer to W1.

Answer Q2. Please see our answer to W2.

Thank you for your detailed response. I would keep my positive score.

We sincerely appreciate the reviewer's positive assessment and the time spent reading our response. We are glad that our response addressed the questions raised in the review. We also thank the reviewer for the constructive feedback, which has helped us improve the quality of the paper.

Reviewer_TdrxOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes the Lifted Bellman Linear Program (LBLP) and its neural relaxation ALBUM for offline RL. The main idea is to lift the LP characterization of Bellman optimality to the joint QV space through the epigraph constraint, so that every constraint involves only dataset pairs. Under deterministic dynamics, the paper shows GD<QD<Q∗G_D < Q_D < Q^*, and that the hinge relaxation is exact above a finite penalty. ALBUM minimizes ωQE[Q]+ωVE[V]\omega_Q\mathbb{E}[Q]+\omega_V\mathbb{E}[V] plus three hinge penalties, with a stop-gradient on the KK-step target. It trains a single Q and V, with no target networks or ensembles. On 10 OGBench state-based domains with a Gaussian DDPG+BC actor, ALBUM matches FQL and well above ReBRAC, and uses the fewest parameters and the least peak memory.

Critical strengths
  • A lightweight critic that stays stable without the usual tricks. ALBUM uses one QQ and one VV with no target network or ensemble.
  • Good results on long-horizon tasks with a Gaussian policy. ALBUM reaches 61.2 vs. 29.6 for FQL on puzzle-3x3, 64.4 vs. 55.8 on scene, and 28.1 vs. 26.2 for ReBRAC on antmaze-giant. I also appreciated the honest limitations section (App. A), which states where the theory stops applying.
Critical weaknesses
  1. The key component is very close to LQL [1], which is not cited. LQL [1] already addresses long-horizon value propagation with a hinge on nn-step lower bounds along logged trajectories, with stop-gradient targets. Its justification is the same as Lemma 4.2 here: a realized action sequence lower-bounds what the optimal policy can achieve. ALBUM's λK[sg(y^K)−Q]+\lambda_K[\mathrm{sg}(\hat y_K)-Q]+ term is nearly identical. The main differences are three:
    • It uses a linear rather than a squared hinge.
    • It replaces the TD regression with the LP descent term plus a one-step hinge.

As a result, the conceptual step over [1] seems incremental. The paper's own hinge-IQL result does not show which part is essential, because it changes the base critic rather than comparing against [1]. Please cite [1] and compare against it under the same codebase and actor. This would isolate what the LP formulation adds beyond the nn-step hinge.

  1. The method is complex and its optimization is fragile. Eq. 5 has five critic coefficients (ωQ,ωV,λB,λE,λK\omega_Q,\omega_V,\lambda_B,\lambda_E,\lambda_K), plus KK and αBC\alpha_{BC}. The objective pushes QQ and VV down linearly and relies on the hinges to stop them, so stability depends on how the coefficients balance. With λK=0\lambda_K=0, Vmin⁡V_{\min} diverges to about −106-10^6.

  2. Empirical gains are modest, and lack the results of transfer to generative-policy methods. ALBUM matches FQL on average but trails it on humanoidmaze-medium, antsoccer and cube-double. The contribution is a way to train the critic, so it should be shown inside recent generative-policy methods, as [1] does with FQL, QC-FQL and Best-of-N. Here the only such test is IFQL on three domains with 4 seeds (Table 13). Please report FQL and QC-FQL (or DQC) with the ALBUM critic on all 10 domains.

Questions for authors

See Weaknesses.

8 comments
AuthorsOpenReview ↗

Response to Reviewer Tdrx (1/4)

Answer W1.

We sincerely appreciate the reviewer's concern regarding the conceptual relation to other works. Since the review does not include a reference for LQL, we identify it as “Long-Horizon Q-Learning: Accurate Value Learning via n-Step Inequalities” (Abraham et al., 2026). Our response is based on this paper.

Summary. ALBUM and LQL share the KK-step lower bound and differ as follows. (i) ALBUM is the relaxation of LBLP, so the linear hinge follows from exact penalty theory, and the objective contains no regression onto a bootstrapped target. (ii) The KK-step constraints are redundant at the LBLP solution, and the role of the KK-step term is analyzed within LBLP. (iii) The solution of ALBUM is the in-sample optimal pair and does not depend on the actor, whereas the TD regression of LQL has QπQ^\pi of the current actor as its fixed point. (iv) With the same actor, ALBUM and LQL attain comparable success rates, and removing the target network and the critic ensemble from LQL lowers its success rate on scene and puzzle-3x3, whereas ALBUM trains without them.

Details. We agree that ALBUM and LQL share the KK-step lower bound along logged trajectories and its justification. Our related work section already distinguishes ALBUM from methods that use multi-step returns as auxiliary terms added to a regression objective, such as Optimality Tightening (He et al., 2017), and LQL is a recent method of this kind.

(i) Origin of the objective. ALBUM is not obtained by replacing the TD regression of LQL or of TD methods with hinge penalties such as Optimality Tightening. ALBUM is the relaxation of LBLP, the primal linear program that we propose for in-sample learning. The constraints of LBLP determine its solution and carry positive dual variables. A penalty preserves such a solution at a finite coefficient only when the subdifferential of the penalty at zero contains the dual variables (Bertsekas, 2015, Section 1.5). The linear hinge meets this condition above a finite threshold (Proposition 4.7), whereas a squared hinge does not. The linear hinge is therefore a consequence of classical exact penalty theory rather than a heuristic design choice. In LQL, the TD regression determines the solution, so the squared hinge is consistent with its design. The program LBLP also explains why ALBUM uses no target network and no critic ensemble (revised Section 6.1 and Appendix L.1). The part of the ALBUM objective that defines the solution contains no regression onto a bootstrapped target and never evaluates the critic at actions outside the dataset. ALBUM represents the value of a state by a separate function VV, and the epigraph constraints determine VV from the actions in the dataset. In contrast, LQL keeps TD regression as its base loss, together with a target network and a two-critic ensemble, and has no separate state-value function. The difference is therefore not the replacement of a loss term. TD regression is a semi-gradient iteration toward a bootstrapped target that moves with the parameters, and target networks were introduced to control the resulting feedback. However, relaxed LBLP under linear features is subgradient descent on a convex objective. On Baird's counterexample (new Appendix L.1.1), TD-based methods diverge without a target network, whereas relaxed LBLP and ALBUM remain bounded at every step size.

(ii) Role of the KK-step term. The KK-step constraints are redundant at the LBLP solution (Lemma 4.2). The KK-step term is nevertheless essential in practice, and its role is analyzed within the same framework. The KK-step term relaxes the condition for bounded updates (Proposition 5.1) and accelerates value propagation in an idealized iteration (Theorem 4.5). With the detached KK-step target, the LBLP solution remains a stationary point of the update under the conditions in Appendix I. However, adding a rollout penalty to a TD loss requires a separate argument that the solution is unchanged. LQL provides this argument for Q∗Q^{\ast} with an optimal bootstrap action a∗(s)a^{\ast}(s), whereas Lemma 4.2 holds for any rollout policy and any horizon.

(iii) Solution that each loss identifies. ALBUM is an in-sample method in the family of IQL (Kostrikov et al., 2022). The unique minimizer of LBLP is the in-sample optimal pair (QD∗,VD∗)(Q^{\ast}_D, V^{\ast}_D) (Proposition 4.3), and ALBUM approximates this program through its relaxation. The critic loss of ALBUM contains no action of the actor, so the solution that ALBUM targets is independent of the actor. In contrast, LQL in continuous control evaluates the current actor rather than performing in-sample control, as in actor-critic methods such as ReBRAC (Tarasov et al., 2023) and FQL (Park et al., 2025). The TD target of LQL contains the action π(s)\pi(s) of the actor, so the TD regression of LQL has QπQ^\pi of the current actor as its fixed point.

AuthorsOpenReview ↗

Response to Reviewer Tdrx (2/4)

(iv) Empirical comparison. Following the reviewer's suggestion, we also compared ALBUM with LQL under the offline RL protocol of the submission. The critic loss, the critic ensemble, the target network, and the trajectory sampling of LQL follow the original implementation, with trajectories of length 8. Each batch contains 32 trajectories, so LQL uses 256 transitions per step, the same number as ALBUM. The discount factor is 0.995 for both methods. LQL uses the Gaussian actor of ALBUM, and the evaluation protocol and the tuning budget are identical. Only the behavioral cloning coefficient is tuned, over the grid {0.001,0.003,0.01,0.03,0.1,0.3}\lbrace 0.001, 0.003, 0.01, 0.03, 0.1, 0.3\rbrace shared with ALBUM, on the default task of each environment. We report all ten domains with five tasks and eight seeds each, after 1M training steps. These results evaluate the critic of LQL outside its intended setting and do not represent the performance of LQL in its target setting.

Task categoryALBUMLQL
antmaze-large-navigate89.8 ± 1.084.5 ± 2.9
antmaze-giant-navigate28.1 ± 5.13.3 ± 1.3
humanoidmaze-medium-navigate35.5 ± 1.857.8 ± 3.5
humanoidmaze-large-navigate1.6 ± 0.53.0 ± 2.7
antsoccer-arena-navigate36.1 ± 3.856.4 ± 3.7
cube-single-play83.2 ± 1.727.2 ± 6.0
cube-double-play13.3 ± 2.816.1 ± 2.8
scene-play64.4 ± 5.562.7 ± 2.7
puzzle-3x3-play61.2 ± 6.490.1 ± 3.7
puzzle-4x4-play20.6 ± 2.57.6 ± 4.1
Average43.440.9
Cost per stepParamsGFLOP/stepms/stepQ/V netsPeak GPU (MiB)
ALBUM2.43M3.821.66 ± 0.02277
LQL4.06M5.901.41 ± 0.014110

ALBUM attains the higher average, 43.4 against 40.9. However, each method leads on five of the ten domains, so the LP formulation does not provide a uniform gain in success rate over the TD formulation of LQL. Instead, the LP formulation attains comparable performance without a target network or a critic ensemble. Both methods were measured with the procedure of Appendix L.3 on a single RTX A5000, so the time and memory values are not comparable to those in Appendix L.3. LQL requires more parameters, more FLOPs, and more peak memory than ALBUM, while its time per step is shorter.

To isolate what the LP formulation adds beyond the shared KK-step lower bound, we also removed the target network and the second critic from LQL, which gives the same single-critic structure without a target network as ALBUM. The BC coefficient and all other settings are unchanged from the comparison above. We report the default tasks of the three environments used for the ablations of Section 6.1, with 8 seeds at 1M steps.

TaskLQL: 2 critics + targetLQL: 1 critic, no targetALBUM: 1 critic, no target
antmaze-large (task 1)85.2 ± 4.485.0 ± 5.992.5 ± 3.1
scene (task 2)71.0 ± 4.518.8 ± 11.193.2 ± 3.2
puzzle-3x3 (task 4)92.8 ± 5.37.2 ± 9.493.0 ± 7.9

Without the target network and the second critic, LQL keeps its success rate on antmaze-large but drops from 71.0 to 18.8 on scene and from 92.8 to 7.2 on puzzle-3x3. With the same structure, ALBUM attains 93.2 and 93.0 on these tasks. The TD formulation of LQL therefore relies on the target network and the critic ensemble on these tasks, whereas the LP formulation trains without them. This result is consistent with Section 6.1, where removing the same devices degrades ReBRAC and FQL. We view this comparison as the most direct evidence of what distinguishes ALBUM from TD-based methods, including LQL, beyond the shared KK-step lower bound.

We thank the reviewer again for pointing out this related work. We will cite LQL and state the major differences in the multi-step values paragraph of the related work section.

[references] Other cited works are listed in the references of the paper.

(Abraham et al., 2026): Abraham, A. A., Shi, L. X., & Finn, C. (2026). Long-Horizon Q-Learning: Accurate Value Learning via n-Step Inequalities. arXiv preprint arXiv:2605.05812.

(Bertsekas, 2015, Section 1.5): Bertsekas, D. P., & Scientific, A. (2015). Convex optimization algorithms (Vol. 1). Belmont: Athena Scientific.

AuthorsOpenReview ↗

Response to Reviewer Tdrx (3/4)

Answer W2. We thank the reviewer for the comment. In short, no run diverges where the condition of Proposition 5.1 holds, the penalty coefficient affects the success rate rather than the stability of the update, and ALBUM tunes a single hyperparameter per environment.

(i) Divergence at λK=0\lambda_K = 0. We apologize that our presentation of the experiment at λK=0\lambda_K = 0 may have suggested that ALBUM is prone to divergence. This experiment deliberately violates the condition of Proposition 5.1, to test whether the update diverges where the condition predicts. In the sweep of (iii) below, every setting satisfies the condition, and no run diverges. Raising λB\lambda_B until the condition holds again also bounds the values at λK=0\lambda_K = 0 (Appendix L.6). In the revision, Section 6.1 and Appendix L.4 state this intent explicitly.

(ii) Complexity. ALBUM tunes a single hyperparameter per environment. The five critic coefficients and KK are fixed across all 10 domains, and only αBC\alpha_{\mathrm{BC}} of the actor is selected from {0.001,0.003,0.01,0.03,0.1,0.3}\lbrace 0.001, 0.003, 0.01, 0.03, 0.1, 0.3\rbrace on the default task of each environment. This is the same number as FQL and fewer than ReBRAC. The submission did not state this grid, and the revised caption of Table 5 states it. The critic coefficients are not arbitrary. ωQ\omega_Q and ωV\omega_V are the weights of the LBLP objective, the λ\lambda coefficients are its penalty coefficients, and equation 7 states the balance that they require.

(iii) Stability of the update and sensitivity of the success rate. To separate the stability of the update from the sensitivity of the success rate, we ran a finer sweep than that of Appendix L.8, with λB=λE∈{0.25,0.5,1,1.5,2,2.5,3,4,5,6,10}\lambda_B = \lambda_E \in \lbrace 0.25, 0.5, 1, 1.5, 2, 2.5, 3, 4, 5, 6, 10\rbrace over 4 seeds on the three tasks of Table 2, with 0.25 on antmaze-large only.

ALBUM, λB=λE\lambda_B = \lambda_E: 0.250.511.522.5345610Reference: FQLReference: ReBRAC
antmaze-large83.283.395.292.392.791.391.284.061.265.31.879.396.8
scene–36.374.095.292.594.793.782.241.012.215.083.764.0
puzzle-3x3–58.538.851.885.080.235.30.00.00.00.013.54.0
max⁡∣Q∣ (1−γ)/∣rmin⁡∣\max\vert Q\vert \thinspace (1-\gamma)/\vert r_{\min}\vert1.0000.9980.9980.9990.9990.9990.9990.9991.0000.9991.0021.1191.189
Diverged runs0/40/120/120/120/120/120/120/120/120/120/120/120/12

The first three rows report the success rate (%). The fourth row reports the largest magnitude of the critic values during training, relative to ∣rmin⁡∣/(1−γ)|r_{\min}|/(1-\gamma), and the last row counts the runs that produce NaN or inf or whose largest magnitude exceeds 1.2 ∣rmin⁡∣/(1−γ)1.2\thinspace |r_{\min}|/(1-\gamma). All entries are from diagnostic runs with 4 seeds, so they differ from the results with 8 seeds in the paper.

Stability. Equation 7 excludes the settings that diverge along the common shift before training. Within the region where equation 7 holds, no run diverges, and the largest magnitude of the critic values stays within 0.2% of ∣rmin⁡∣/(1−γ)|r_{\min}|/(1-\gamma). The critics of FQL and ReBRAC exceed this magnitude by 12% and 19%, despite their target networks and critic ensembles.

Sensitivity of the success rate. The penalty coefficient therefore affects mainly the success rate rather than the stability of the update, since the success rate depends on how the critic orders the actions rather than on the magnitude of its values. This sensitivity is an expected property of exact penalty methods, for which a coefficient that is too small can leave the penalty function unbounded below, and a coefficient that is too large slows the minimization because the iterates follow the constraints too closely (Byrd and Nocedal, 1998). In ALBUM, the KK-step term keeps equation 7 satisfied for small λB\lambda_B, so the first failure does not occur in the sweep and appears only when equation 7 is violated at λK=0\lambda_K = 0. The decrease at large coefficients is consistent with the second failure. We acknowledge that the success rate is sensitive to the penalty coefficient on puzzle-3x3, where the range near the default is narrow. However, the default coefficients are fixed across all 10 domains and do not need to be retuned per environment. In the revision, Appendix L.8 reports this sweep.

[reference]

(Byrd and Nocedal, 1998): Byrd, R. H., & Nocedal, J. (1998, January). Active set and interior methods for nonlinear optimization. In Proceedings of the International Congress of Mathematicians 1998 (pp. 667-676).

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We sincerely thank the reviewers for their careful reading and constructive comments. The main claim of the submission concerns how the critic is trained. ALBUM imposes in-sample Bellman optimality through inequality constraints instead of regression onto bootstrapped targets. ALBUM therefore trains a single critic without target networks or critic ensembles, and matches the average performance of FQL with the fewest parameters and the least peak GPU memory among the measured methods. During the discussion period, we tested this claim directly. With the same KK-step lower bound, removing the target network and the second critic from LQL keeps its success rate on antmaze-large but lowers it from 71.0 to 18.8 on scene and from 92.8 to 7.2 on puzzle-3x3, whereas ALBUM attains 93.2 and 93.0 with the same structure. On Baird's counterexample, TD(0), Q-learning, and clipped double Q-learning diverge or reach large errors without a target network, whereas relaxed LBLP and ALBUM remain bounded at every tested step size.

Revision. Revised text is marked in blue. Our answers refer to the numbering of the submission unless they state the revision. The new Table 3 shifts the numbers of the subsequent tables by one in the revision. The main changes are as follows.

  • Section 5 states the precise relation of the implemented update to the relaxed LBLP in place of direct implementation, and the abstract, introduction, and conclusion use the same wording.

  • Section 6.1 states why ALBUM requires neither target networks nor critic ensembles, and the purpose of the experiment at λK=0\lambda_K = 0.

  • The new Table 3 and Appendix L.7 combine the ALBUM critic with the one-step policy of FQL and the Best-of-NN policy of IFQL on all ten domains with 8 seeds. This experiment replaces the untuned experiment on three domains in Appendix L.7 of the submission.

  • The new Appendix L.1.1 reports Baird's counterexample.

  • Appendix L.8 reports a finer sweep of the penalty coefficients, with the largest magnitude of the critic values and the number of diverged runs.

  • Appendix L.11 states the reference level for the fraction of satisfied constraints, and compares the constraint residual and the learned values of ALBUM with those of FQL and ReBRAC.

  • The new Appendix L.12 analyzes how stochastic transitions enter ALBUM and reports experiments under action noise.

  • A new Limitations paragraph in the Conclusion states which results require deterministic dynamics.

  • The related work section cites LQL and states the major differences, and the caption of Table 5 states the tuning grid of the behavioral cloning coefficient.

AuthorsOpenReview ↗

Rebuttal summary

We sincerely appreciate the time and effort that the reviewers and the AC spent assessing our paper. We summarize how each concern is addressed.

Reviewer Tdrx

  • W1. Novelty with respect to LQL

We answered that ALBUM and LQL share a similar K-step penalty, but LQL is a TD method and ALBUM is an LP method. ALBUM uses a linear hinge for a theoretical reason, exact penalty theory, and a separate state-value function VV, and its solution is the in-sample optimal QD∗Q^*_D rather than QπQ^\pi of the current actor. Following the reviewer's request, we added a comparison with LQL on all ten domains, and an ablation that removes the target network and the critic ensemble from LQL. Without these devices, LQL drops from 71.0 to 18.8 on scene and from 92.8 to 7.2 on puzzle-3x3, whereas ALBUM attains 93.2 and 93.0.

  • W2. Fragility of the optimization and complexity

We answered that the divergence at λK=0\lambda_K=0 comes from an experiment that deliberately violates the condition of Proposition 5.1, and that no run diverges where equation 7 holds in a finer sweep. We separated the stability of the update from the sensitivity of the success rate, and explained the sensitivity as an expected property of exact penalty methods. As for LQL, ALBUM tunes only the behavioral cloning coefficient per environment. Counted in the same way, ALBUM fixes one more hyperparameter than LQL, and the same number in practice because λB\lambda_B and λE\lambda_E share a single value.

  • W3. Empirical gains and transfer to generative-policy methods

We answered that ALBUM attains a success rate comparable to FQL with about half the parameters, the least peak GPU memory, and no target network or critic ensemble, and we interpreted this combination as the empirical gain of ALBUM. Following the reviewer's request, we extended the transfer experiment to FQL and IFQL on all ten domains. The ALBUM critic raises the average of IFQL from 30.3 to 43.9 and does not improve the average of FQL. On the same ALBUM critic, Best-of-N exceeds the Gaussian policy by more than one standard deviation on five of ten domains. QC-FQL was not included because action chunking changes the action space of the critic.

Reviewer fggU

Reviewer fggU has confirmed the positive assessment after our response.

  • W1, Q1. Relation of the implemented update to the relaxed LBLP

We agreed that ALBUM does not descend the scalar objective of the relaxed LBLP. The revision replaces the claim of direct implementation in the main text with a precise statement.

  • W2, Q2. Stochastic dynamics

Appendix A already distinguished the claims that require deterministic dynamics. We moved this statement into a Limitations paragraph in the main text, and measured the robustness of ALBUM and ReBRAC under stochastic transitions in Appendix L.12.

Reviewer KDbs

  • W1, Q1. Relation of the theory to the implemented algorithm

We acknowledged that our theory does not establish convergence of the neural update, as for other deep off-policy methods. However, our response identified in a table what the theory explains. Appendix L.11 lacked a reference level for the fraction of satisfied constraints. Under deterministic dynamics and continuous states, every one-step constraint is active at the LBLP solution, so even an accurate critic satisfies about 50% of them. We added this reference level and compared the constraint residual of ALBUM with the TD error of FQL and ReBRAC. We explained that learned values above the dataset return can arise from generalization across neighboring trajectories, and the TD critics also exceed this return on scene and puzzle-3x3. For Q1, the K-step hinge does not improve IQL, and the LQL ablation in W1 of Reviewer Tdrx applies.

  • W2, Q2. Deadly triad and Baird's example

On Baird's counterexample, relaxed LBLP and ALBUM remain stable at every tested step size, whereas the TD-based methods diverge in several settings. We clarified that our claim is that ALBUM does not require target networks or critic ensembles, not a general resolution of the deadly triad.

  • W3, Q3. Stochastic transitions

Appendix A of the submission already stated the claims that rely on deterministic dynamics. Our response organized these claims into a table, and the revision moves them into a Limitations paragraph in the main text. We also explained that every environment of OGBench has deterministic dynamics, so the assumption does not weaken the justification of ALBUM in the setting of our experiments. Following the reviewer's request, we analyzed the hinge bias and measured the robustness of ALBUM and ReBRAC under stochastic transitions in Appendix L.12.

  • Q4. Sensitivity and selection of the coefficients

We answered that Proposition 5.1 concerns the stability of the update rather than the success rate. Equation 7 restricts the search before training, and the critic coefficients and the horizon are fixed across all ten domains.

We would be grateful to hear any remaining concern.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.