acceptodds
Under review as a conference paper at ICLR 2027

Horizon-Scalable Q-Learning

11 views

Abstract

Learning accurate off-policy state-action value functions (Q-functions) from offline datasets becomes harder as the task horizon grows. One-step temporal-difference (TD) learning bootstraps after every transition, so small per-step errors accumulate along the horizon, and early states can be badly misvalued even when the TD loss is small. Multi-step TD shortens this chain by following the dataset trajectory for several steps; however, since the behavior policy is often suboptimal, every followed action adds a bias, and no fixed number of steps suits a whole trajectory. In this paper, we propose Horizon-Scalable Q-Learning (HQL), a simple method that enables long-horizon value learning by automatically balancing between one-step and multi-step TD. HQL extends a target through a dataset action only when the critic values it at least as highly as the best of several actions sampled from the policy, and otherwise falls back to a one-step target. We show that this rule is a principled off-policy correction that needs no action densities, which makes HQL compatible with expressive flow and diffusion policies. Experiments show that HQL avoids the failures of both one-step and multi-step TD on a combination-lock task across horizons, and outperforms prior methods on long-horizon OGBench tasks, especially on the most challenging ones.

Primary area:
reinforcement learning
Keywords:
value learning, horizon scalability, off-policy RL

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_eoSTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses accurate off-policy Q-learning over long horizons. It motivates Horizon-Scalable Q-Learning (HQL) through the trade-off between accumulated one-step TD errors and bias from suboptimal actions in multi-step continuations. HQL adapts the backup length by comparing each logged action's critic value with the best of several actions sampled from a behavior model. When the logged action passes this comparison, downstream value corrections propagate through it; otherwise, the target uses a one-step bootstrap. The gate requires action samples and critic evaluations, making it compatible with sampling-based policies without evaluating action densities. The method also extends to action chunks. Experiments on a corrupted combination-lock task report lower value-estimation errors across horizons and corruption levels, while offline-to-online OGBench experiments report higher average success rates than the evaluated TD and long-horizon baselines.

Critical strengths
  1. HQL provides a simple critic-target modification that can use expressive action samplers without evaluating their densities. Algorithm 1 computes targets backward through a window and trains every position; Chunked-HQL separates decision stride from backup length. This makes the approach practically useful for continuous-control methods built around flow or other sampling-based policies. The QAM and VGF experiments provide further evidence that the critic modification can help beyond the main best-of-N policy.

  2. The experiments address both parts of the motivation: increasing horizon and suboptimal logged continuations. The combination-lock task permits direct measurement of Q-value error, while the main OGBench study uses five seeds and matches the number of critic targets per update. After 1M offline gradient updates and a further 1M online interaction steps, Table 1 reports mean offline/online success rates of 40%/74% for HQL, 34%/62% for LQL, and 21%/45% for Q-chunking; Chunked-HQL reaches 41%/81%. These gains support the practical value of the method in offline-to-online learning. The trace statistics also give a useful account of its behavior: 77% of targets remain one-step, indicating that most extensions are selective and short.

Critical weaknesses

1. The contribution of the sampled gate is insufficiently isolated from existing trace mechanisms and other changes to the backup.

Watkins' Q(λ)Q(\lambda) cuts traces at non-greedy actions [1], Retrace propagates weighted TD corrections [2], and Daley's Gated Q-learning attenuates non-greedy traces without importance ratios [3]. HQL's distinguishing mechanism is its finite-candidate value comparison for sampling-based continuous-action policies. Establishing the value of this mechanism requires an explicit comparison of the target operators and a controlled comparison with a suitable sampling-based greedy-action gate. The reported baselines cover fixed-step TD, LQL, and Q-chunking, leaving this direct comparison open.

The intended gate-only control also has an unresolved target definition. Direct expansion of Eq. (6), with c=1c=1 and two transitions ending at a zero-valued terminal state, gives

Y0=r0+γr1+γ(Vπ(s1)−Q(s1,a1)).Y_0 = r_0 + \gamma r_1 + \gamma\left(V^\pi(s_1)-Q(s_1,a_1)\right).

This differs from the raw two-step return by γ(Vπ(s1)−Q(s1,a1))\gamma\left(V^\pi(s_1)-Q(s_1,a_1)\right), whereas lines 221-223 and Section 4.3 identify the all-open recursion with an uncorrected return. The implemented target determines the interpretation of 16-window TD: using raw returns changes both gating and correction terms, while using Eq. (6) with c=1c=1 requires correcting the stated special case. Appendix A.6 additionally changes the actor/bootstrap sample count between soft and hard gates (16 versus 4), and Appendix A.7 varies gate width and action-search width together. These comparisons therefore combine changes whose individual effects need to be separated.

2. The finite-sample derivation uses a coefficient bound that its estimator can violate.

Section 4.2 invokes c(s,a)≤π(a∣s)/μ(a∣s)c(s,a)\leq \pi(a\mid s)/\mu(a\mid s), a sufficient coefficient condition for the contractive off-policy correction analyzed in [2]. It then replaces the exponential-tilt normalizer ZZ with a random estimate and calls the resulting clipped ratio the largest valid gate. A counterexample follows directly from Eq. (9): let mu assign probability 1/21/2 to each of two actions with critic values 0 and 1. For N=4N=4 and τ=5\tau=5, all candidates have value 0 with probability 1/161/16. On this event, the estimated gate for the lower-valued logged action is 1, whereas its true tilted-policy density ratio is 2/(1+e5)≈0.01342/(1+e^5)\approx 0.0134.

The estimated coefficient thus exceeds the stated bound with positive probability. The implementation uses a finite-best-of-N target policy, which differs from the exact exponential tilt in the derivation. The theoretical claim therefore requires an analysis of the coefficient actually used, its target policy, and any averaging or sample-independence assumptions. Fixed-point preservation and contraction should be stated and justified separately.

References

[1] Watkins, C. J. C. H. (1989). Learning from Delayed Rewards.

[2] Munos, R., Stepleton, T., Harutyunyan, A., and Bellemare, M. G. (2016). Safe and Efficient Off-Policy Reinforcement Learning.

[3] Daley, B. (2026). Gated Q-learning: Add Off-Policy Bias to Taste.

Questions for authors
  1. Please give the exact implemented recursions for HQL, 16-window TD, and the tau=0 soft-gate control. Does the control retain the V^pi-Q correction? This clarification determines whether it isolates gating or changes the backup operator as well.

  2. For which policy and in what sense is the finite-N gate valid: samplewise, or after averaging over candidate draws? Please address the counterexample above, distinguish fixed-point preservation from contraction, and specify any required independence between gate and bootstrap samples, including for QAM/VGF.

  3. Please explain the operator-level distinction from Watkins' Q(lambda), Retrace, and Daley's Gated Q-learning, cited as [1]-[3] in Critical Weaknesses. A targeted comparison should hold the candidate sampler, bootstrap, windows, and target budget fixed. Please identify an appropriate continuous-action adaptation or explain the specific obstacle to adapting these mechanisms.

  4. Which existing runs hold actor/bootstrap search width fixed while varying gate width or comparing soft and hard gates? Please distinguish conclusions supported by those controlled comparisons from conclusions about the complete configurations in Appendices A.6-A.7.

2 comments

We thank the reviewer for pointing to Watkins' Q(λ)Q(\lambda), Retrace and Gated Q-learning.

W1 / Q3. Distinction from existing trace mechanisms. All four methods set the trace coefficient of a multi-step backup. They differ in what the coefficient needs from the target policy, which is why the first three do not apply directly to a flow or best-of-NN policy with continuous actions.

trace coefficient c(s,a)c(s,a)needsfixed point
Watkins' Q(λ)Q(\lambda)λ 1[a∈arg⁡max⁡a′Q(s,a′)]\lambda\,\mathbb 1[a\in\arg\max_{a'}Q(s,a')]exact argmax over a discrete action setQ∗Q^*
Gated Q(λ)Q(\lambda) (Daley, 2026)λ\lambda if aa is greedy, else λχ\lambda\chiexact argmax over a discrete action set, gate χ\chiQπmixQ^{\pi_{\rm mix}}, πmix=μλ+(1−c(s))πgreedy\pi_{\rm mix}=\mu\lambda+(1-c(s))\pi_{\rm greedy} (biased for χ>0\chi>0)
Retrace (Munos et al., 2016)λmin⁡{1,π(a∣s)/μ(a∣s)}\lambda\min\lbrace 1,\pi(a\vert s)/\mu(a\vert s)\rbracedensities of π\pi and μ\muQπQ^\pi
HQL1[Qˉ(s,a)≥max⁡m≤NQˉ(s,a(m))]\mathbb 1[\bar Q(s,a)\ge\max_{m\le N}\bar Q(s,a^{(m)})], a(m)∼μa^{(m)}\sim\musamples of μ\mu and the criticQπMQ^{\pi_M} for N≥M−1N\ge M-1 (Proposition)
  • Watkins' Q(λ)Q(\lambda). A continuous logged action is almost surely not the exact argmax, so the trace is cut at every step and the target reduces to 1-step TD. Moreover, a flow policy cannot compute this argmax, and behavior-regularized methods do not target it.
  • Gated Q(λ)Q(\lambda). With continuous actions, every logged action is non-greedy, so the coefficient is the constant λχ\lambda\chi, independent of action quality, and the method reduces to Peng's Q(λχ)Q(\lambda\chi). HQL supplies the state-action-dependent λ(s,a)\lambda(s,a) that Daley's framework leaves open, in a form that works for continuous actions.
  • Retrace. It needs π/μ\pi/\mu, but log⁡μ\log\mu of a flow model requires integrating the velocity divergence along the ODE for every candidate, and the best-of-NN policy has no closed-form density.

W1 (cont.) / Q1. Exact recursions; does the τ=0\tau=0 control keep the V−QV-Q term? With oj=rj+γVˉ(sj+1)o_j=r_j+\gamma\bar V(s_{j+1}), the implemented recursions are

  • HQL: Yj=oj+γcj+1(Yj+1−Qˉ(sj+1,aj+1))Y_j=o_j+\gamma c_{j+1}\big(Y_{j+1}-\bar Q(s_{j+1},a_{j+1})\big), with the gate of Eq. (8);
  • 16-window TD: Gj=rj+γGj+1G_j=r_j+\gamma G_{j+1}, bootstrapping Vˉ\bar V only at the window end;
  • τ=0\tau=0 soft gate (Appendix A.6): c≡1c\equiv1, so Yj=oj+γ(Yj+1−Qˉ(sj+1,aj+1))Y_j=o_j+\gamma\big(Y_{j+1}-\bar Q(s_{j+1},a_{j+1})\big).

W2 / Q2. Validity of the finite-NN gate. See General response (1): the gate is valid in expectation for the best-of-MM policy when N≥M−1N\ge M-1.

Q4. Which runs isolate gate width or the soft/hard choice? See the table in our reply to Reviewer YXmF (Q4).

We sincerely hope the reviewer could consider raising the score if no major concerns are left, that means a lot to us. We hope you all the best!

Reviewer_jwVFOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Horizon-Scalable Q-Learning (HQL) aims to accelerate value propagation while controlling off-policy error in multi-step TD learning. It introduces a critic-dependent gate that determines whether to continue a multi-step backup or truncate it to a one-step target. The gate remains open when the estimated Q-value of the action in the sampled trajectory is at least as high as those of the policy-generated candidate actions. The method is evaluated on a combination-lock benchmark and offline-to-online OGBench tasks.

Critical strengths

None

Critical weaknesses
  1. The reliability of critic-dependent gating is insufficiently established.

The gating decision relies on the critic’s ability to rank trajectory actions against policy-generated candidates. An inaccurate critic may truncate a useful trace or retain a trace that introduces off-policy error. Since the same critic both estimates values and determines which corrections to propagate, errors in its predictions may also affect the reliability of subsequent updates.

Although the paper motivates the gating rule, it does not sufficiently characterize when the learned gate makes reliable decisions in practice. Direct diagnostics of erroneous trace retention and truncation, particularly under inaccurate critic estimates, would help establish when gating improves value propagation.

  1. Robustness to weak initialization and different critic training settings remains unclear.

The OGBench experiments use offline pretraining before online interaction. These results do not establish whether HQL remains effective when the critic is poorly initialized or the initial dataset provides limited coverage. In settings requiring substantial exploration, inaccurate value estimates could make gating less reliable precisely when effective value propagation is most important.

The robustness of the method to critic architecture and optimization choices is also insufficiently explored. Evaluations across critic widths and depths, learning rates, and target-network update rates would help determine whether the reported benefits depend on a particular critic configuration. Although the authors acknowledge the dependence on critic accuracy, the conditions under which gating remains beneficial are not yet clearly established.

Questions for authors
  1. Could the authors evaluate HQL with online training from scratch or reduced offline dataset coverage, including a sparse-reward setting requiring substantial exploration? How does gating behave when the critic is initially inaccurate?

  2. Could the authors evaluate sensitivity to critic width and depth, learning rate, and target-network update rate, using the same configurations for HQL and its TD baselines?

3 comments

We thank the reviewer and address the two reliability concerns below.

W1. When is the critic-dependent gate reliable? On the combination lock, where Q∗Q* is known, we track the ranking accuracy of the learned critic against Q∗Q^* (5 seeds, mean ± std):

updates100k300k500k1M1.5M2M2.5M3M3.5M4M4.5M5M
ranking accuracy (%)72±880±675±1381±985±780±1786±1591±1395±589±989±1390±14

W2 / Q1. Weak initialization, limited coverage, sparse reward. We re-initialize the critic and target critic after the offline phase and start online training from the replay buffer. The actor is the frozen pretrained BC proposal, so improvement comes only from best-of-NN selection under the new critic.

offline → online, critic from scratchcube-doublecube-triplescenehmaze-medantmaze-gianthmaze-giant
2-step TD18±2 → 90±20±0 → 1±185±2 → 99±147±3 → 55±126±3 → 81±110±5 → 8±9
4-step TD11±0 → 84±30±0 → 9±880±1 → 100±040±1 → 47±322±1 → 62±410±9 → 4±1
HQL72±1 → 96±114±4 → 69±1189±2 → 100±056±3 → 66±320±1 → 82±28±5 → 11±3

Mean ± std over 2 seeds. Offline: mean of the last three offline evaluations, before the reset; online: end of training. HQL still achieves the best average both offline and online.

Q2. Sensitivity to critic width/depth, learning rate and target update rate. We vary one factor at a time around the default critic (4×512, lr 3×10−43\times10^{-4}, target rate 0.005) for both HQL and 2-step TD: width {256,1024}\lbrace 256, 1024\rbrace, depth {2,6}\lbrace 2, 6\rbrace, lr {10−4,10−3}\lbrace 10^{-4}, 10^{-3}\rbrace and target rate {0.001,0.02}\lbrace 0.001, 0.02\rbrace, on cube-double, humanoidmaze-medium and antmaze-giant (five tasks each):

critic settingHQL: cube-doublehmaze-medantmaze-giantmean2-step TD: cube-doublehmaze-medantmaze-giantmean
default (4×512, lr 3e-4, target 0.005)8788888876939086
width 2569078848473858380
width 10249190888978988386
depth 2917105443801446
depth 68786928876839083
lr 1e-49092598077949087
lr 1e-3diverged7391–diverged8975–
target rate 0.0018285657770927178
target rate 0.028888928980959389

Thank you for the additional experiments. The combination-lock diagnostics help quantify critic ranking accuracy. However, my central concern remains: how should HQL handle gating decisions when the critic is inaccurate?

I therefore maintain my rating. I encourage the authors to better address the dependence on inaccurate critics and develop a clearer justification for a future conference submission.

Thank you for your response. We respectfully disagree with this concern, as we believe it stems from a misunderstanding of HQL’s role. HQL aims to improve critic learning rather than rely on an additional critic. In other words, HQL helps learn a more accurate critic. Our long-horizon experiments, where critic estimation is typically getting larger errors than in shorter-horizon tasks, demonstrate that HQL substantially improves critic accuracy and, consequently, policy performance. The key mechanism is HQL's design, i.e., dynamically adjust the horizon length based on the quality of the logged actions, balancing off-policy bias against error accumulation over longer horizons. Also, in the rebuttal we have added setting that the critic is learning from scratch (which is very inaccurate at the begining), under both settings across different hard, long-horizon tasks, HQL is always the best. Actually, under the extreme case where every critic value is totally wrong, then basically all TD or actor-critic based methods will fail as they depend on either the critic’s value estimates or its gradients.

Reviewer_YXmFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses long-horizon value learning in offline and offline-to-online reinforcement learning. Through a Bellman-residual decomposition on deterministic chains, it illustrates how small local TD errors can accumulate into substantial value errors, while raw multi-step returns can introduce bias from suboptimal behavior actions. This analysis motivates choosing backup lengths on a per-transition basis.

The proposed Horizon-Scalable Q-Learning (HQL) selectively propagates downstream TD corrections. A binary gate continues a trace only when the target critic ranks the logged action at least as highly as the best of N candidates sampled from a behavior model. The method requires sampling and value comparisons, but not explicit policy densities. It computes targets at every position in a trajectory window and extends to action chunks through Chunked-HQL.

Evaluation includes combination-lock tasks with analytically available optimal Q-values and six state-based OGBench configurations. The main OGBench comparison uses five tasks per configuration, five seeds, and a matched budget of 512 critic targets per update. Reported mean online success rates are approximately 74% for HQL, 49% for 8-step TD, 62% for LQL, and 45% for Q-chunking; Chunked-HQL achieves 81%. Additional experiments combine HQL with QAM/VGF and examine gate and window settings.

Critical strengths
  1. A simple and practically useful design for implicit policies. The gate uses only behavior-model samples and critic evaluations, avoiding explicit action densities. This makes it straightforward to integrate with flow-based samplers and best-of-N policy extraction. A low-intrusion critic modification with this compatibility is a meaningful practical contribution.

  2. Substantial empirical evidence beyond a toy example. The combination-lock experiment varies both horizon and data corruption and provides ground-truth Q-values. The main OGBench experiments cover multiple environments, tasks, and seeds, with explicit offline and online phases. The reported improvements over TD, LQL, and Q-chunking provide a credible positive signal supporting the method’s practical value.

Critical weaknesses
  1. The finite-sample normalization does not preserve the claimed importance-ratio bound. In Section 4.2, using the true normalizer Z gives c = min(1, exp(τQ)/Z) ≤ π/μ. Equation (9), however, substitutes a sample estimate Zhat, which need not upper-bound Z. Thus, the resulting clipped estimate is not necessarily a valid gate under the paper’s stated criterion. For example, consider two equally probable behavior actions l and h with Q(l)=0 and Q(h)=1. The true tilted-policy ratio for l is 2/(1+exp(τ)), which is less than one for τ>0. If all N proposals happen to be l, an event of probability 2^(-N), then Zhat=1 and the sampled gate equals one, exceeding the true ratio. For fixed finite N, this mismatch persists as τ tends to infinity. The sample-level limit from Equation (9) to Equation (7) can be correct without inheriting validity for the true tilted policy. The actual best-of-N policy may admit a different expected-operator justification, but that argument is not supplied. Fixed-point preservation should also be distinguished from contraction or convergence guarantees: violating a sufficient ratio bound does not by itself prove that Qπ ceases to be a fixed point. This is consequential because the density-free off-policy-correction interpretation is a central contribution.
  2. The gate-removal baseline is not algebraically equivalent to the raw multi-step return, and other ablations are confounded. Setting c=1 in Equation (6) and expanding two steps produces the raw two-step return plus γ[Vbarπ(s_(t+1)) − Qbar(s_(t+1),a_(t+1))]. This internal advantage-correction term generally does not vanish. Consequently, the statement that c≡1 recovers the uncorrected multi-step return is not generally correct. If 16-window TD uses raw n-step returns, its comparison with HQL changes both gating and the internal correction. If it instead uses Equation (6) with all gates open, it can be a clean gating ablation, but its stated definition needs correction. Without the implementation, I cannot determine which is used. Moreover, Appendix A.6 compares a soft gate using 16 proposals with a hard gate using four, changes the actor’s sampling width accordingly, and reports only one seed. Appendix A.7 also changes actor width and gate width together. These comparisons do not isolate the claimed effects. Given the predominantly short HQL traces, short-n and trace-length-matched random-gating controls would further strengthen mechanism attribution.
Questions for authors

The following are clarification and targeted validation requests that could materially change my recommendation, rather than suggestions for future work.

  1. Can you justify the expected backup operator for the actual finite-N best-of-N policy, rather than the true exponential tilt? Please specify the assumptions on sampling independence, Q-value ties, and behavior-model approximation, and distinguish fixed-point preservation from contraction guarantees. What corresponding claims hold for QAM/VGF?
  2. Does 16-window TD implement raw n-step returns or Equation (6) with c=1? Please provide target-construction pseudocode and reconcile the internal advantage term with the claimed special case. If the implementation changes more than the gate, can you provide a matched gate-only comparison?
  3. Under the same actor, networks, and target budget, how does HQL compare with n=2/4 TD and a random gate matched to its trace-length distribution? This would help distinguish action-quality selection from simply using shorter backups.
  4. Can you compare soft and hard gates at the same proposal count and actor setting with multiple seeds, and vary gate width while holding actor width fixed? Please also specify the actual L and H used for Chunked-HQL and whether its chunk size matches Q-chunking.
  5. How should the effective horizon in the residual analysis be interpreted for the discounted OGBench experiments? Appendix A.1 discusses discount-induced saturation, but the connection between that quantity, maximum episode length, and the empirical scalability claim remains unclear.
2 comments

We thank the reviewer for the detailed reading and precise requests.

W1 / Q1. Expected operator for the finite-NN best-of-NN policy. See General response (1). In addition:

  • Sampling independence. Gate candidates are drawn fresh at every position with a random key separate from the bootstrap's, so the expectation factorizes across positions; independence at the same state is not required.
  • Ties. The bound holds with uniform tie-breaking in πM\pi_M; the gate uses ≥\ge, as in Eq. (8) and our code.
  • Behavior-model approximation. The guarantee is for the learned μ^\hat\mu that generates the candidates; its gap to the data-generating μ\mu is a stated limitation.
  • Soft gate, Eq. (9). log⁡Z^\widehat{\log Z} excludes the logged action, in the paper and in our code. With w=exp⁡(τQ~)w=\exp(\tau\tilde Q) and S=∑m≤Nw(a(m))S=\sum_{m\le N}w(a^{(m)}), min⁡{1,Nw(a)/S}≤(N+1)w(a)/(w(a)+S)\min\lbrace 1,Nw(a)/S\rbrace \le(N+1)w(a)/(w(a)+S), whose expectation is the density ratio of the sampling-importance-resampling policy πN+1SIR\pi_{N+1}^{\rm SIR} (draw N+1N{+}1 behavior actions, return one with probability ∝w\propto w). The soft gate is thus a valid trace for πN+1SIR\pi_{N+1}^{\rm SIR} and recovers the hard gate (M=N+1M=N+1) as τ→∞\tau\to\infty. In the reviewers' example (N=4N=4, τ=5\tau=5), the low-value action's expected soft gate is 0.077, above its tilt ratio (0.013) but below its π5SIR\pi_{5}^{\rm SIR} ratio (0.080). Our soft-gate runs bootstrap with best-of-MM, which this argument does not cover; we state this in the revision.
  • Fixed point vs. contraction. Both follow from (ii). We drop the claim that violating the ratio bound removes QπQ^\pi as a fixed point.

W2 / Q2. Is 16-window TD raw nn-step or Eq. (6) with c=1c=1? It is the raw return; the gate-only control is in General response (2). When the gate is open, the logged action beats the best of NN candidates and Vˉ\bar V is the best of M=NM=N candidates, so the internal term γ[Vˉ(st+1)−Qˉ(st+1,at+1)]\gamma[\bar V(s_{t+1})-\bar Q(s_{t+1},a_{t+1})] stays close to zero, and HQL is close to a raw nn-step return cut where the logged action loses. Running this form directly, Yj=rj+γ[(1−cj+1)Vˉ(sj+1)+cj+1Yj+1]Y_j=r_j+\gamma[(1-c_{j+1})\bar V(s_{j+1})+c_{j+1}Y_{j+1}], which drops only the internal term, performs similarly:

cube-doublecube-triplescenehmaze-medantmaze-gianthmaze-giantmean
HQL without the internal term82±245±699±090±187±126±772
HQL87±254±4100±086±385±429±373

Mean ± std over 3 seeds.

Q3. Short nn and trace-length-matched random gate. See General response (3).

Q4. Soft vs. hard gate at matched width; gate width at fixed actor width; Chunked-HQL. With the actor and the bootstrap fixed to best-of-4, only the gate changes:

gate (actor = bootstrap = best-of-4)cube-doublecube-triplescenehmaze-medantmaze-gianthmaze-giant
hard, N=4N=4 (HQL)87±254±4100±086±385±429±3
hard, N=16N=1683±146±799±187±073±224±5
soft, τ=3\tau=3, N=4N=480±141±599±172±154±119±8
soft, τ=3\tau=3, N=16N=1679±240±199±272±344±124±9

Chunked-HQL. Chunked-HQL and QC both use L=8L=8 in Table 1; on OCBench, all methods use 25-step chunks.

Q5. Effective horizon for discounted OGBench. The residual analysis is undiscounted, and Figure 3 tests exactly this regime (γ=1\gamma=1, HH from 512 to 4096). With γ<1\gamma<1, the error ϵt=∑kγket+k\epsilon_t=\sum_k\gamma^ke_{t+k} has variance growing linearly in the distance TT to a state of known value and saturating at T≈1/(1−γ)T\approx1/(1-\gamma) (Appendix A.1), so the analysis applies with Heff=min⁡{T,1/(1−γ)}H_{\rm eff}=\min\lbrace T,1/(1-\gamma)\rbrace. On our tasks, TT is 500–4000 steps while 1/(1−γ)1/(1-\gamma) is 100–200, so value is still chained through 100–200 one-step backups, which HQL shortens.

Saturation caps the error but not its effect. With reward −1-1 per step, delaying the goal by kk steps from distance TT changes the value by γT(1−γk)/(1−γ)\gamma^T(1-\gamma^k)/(1-\gamma), so action gaps decay as γT\gamma^T while the error stays at σ/1−γ2\sigma/\sqrt{1-\gamma^2}. The error-to-gap ratio, which decides whether best-of-NN picks the right action, keeps growing with TT. This matches where HQL helps most over one-step TD (antmaze-giant 85 vs. 26, humanoidmaze-giant 29 vs. 0, cube-triple 54 vs. 16).

We sincerely hope the reviewer could consider raising the score if no major concerns are left, that means a lot to us. We hope you all the best.

Comments on the paper

1 comment

We thank all reviewers for their careful reviews. The shared concerns are (1) whether the finite-NN gate is a valid trace coefficient, (2) whether the gate-removal baseline isolates the gate, and (3) whether the gain comes from quality-based selection or from shorter backups. We address them with new theory and new controlled experiments.

(1) Validity of the finite-NN gate. We agree that log⁡Z^\widehat{\log Z} in Eq. (9) does not upper-bound the exponential-tilt ratio sample by sample; the counterexample of Reviewers eoST and YXmF is correct. However, HQL evaluates the best-of-MM policy πM\pi_M of its bootstrap, not the exact tilt, so we analyze the operator in expectation over the gate's candidates (new Appendix, with proof).

Proposition. Fix the ranking critic Q~\tilde Q. Let πM\pi_M return the Q~\tilde Q-best of MM i.i.d. behavior samples (ties broken uniformly), let the gate ci=1[Q~(si,ai)≥max⁡m≤NQ~(si,ai(m))]c_i=\mathbb 1[\tilde Q(s_i,a_i)\ge\max_{m\le N}\tilde Q(s_i,a_i^{(m)})] use fresh candidates at every position, and let the bootstrap be an unbiased estimate of EπMQˉ\mathbb E_{\pi_M}\bar Q. In the tabular setting, (i) the expected HQL operator is Retrace for πM\pi_M with traces cˉ(s,a)=Fs(a)N\bar c(s,a)=F_s(a)^N, where Fs(a)=Pa′∼μ(Q~(s,a′)≤Q~(s,a))F_s(a)=P_{a'\sim\mu}(\tilde Q(s,a')\le\tilde Q(s,a)); (ii) if N≥M−1N\ge M-1, then 0≤cˉ≤min⁡{1,πM/μ}0\le\bar c\le\min\{1,\pi_M/\mu\}, so QπMQ^{\pi_M} is the unique fixed point and the operator is a γ\gamma-contraction (Munos et al., 2016, Thm. 1); (iii) for continuous values, N≥M−1N\ge M-1 is also necessary.

Proof sketch. Unrolled, Eq. (6) is Yt=Qˉt+∑kγk(∏j=1kct+j)(rt+k+γV^t+k+1−Qˉt+k)Y_t=\bar Q_t+\sum_k\gamma^k(\prod_{j=1}^k c_{t+j})(r_{t+k}+\gamma\hat V_{t+k+1}-\bar Q_{t+k}). The gates and the bootstrap in each term come from distinct positions, so the expectation replaces cic_i by FNF^N and V^\hat V by EπMQˉ\mathbb E_{\pi_M}\bar Q, which is Retrace. (ii) πM/μ=MFM−1\pi_M/\mu=MF^{M-1} without ties and (FM−F−M)/(F−F−)≥FM−1(F^M-F_-^M)/(F-F_-)\ge F^{M-1} with ties (convexity of xMx^M), so FN≤FM−1≤πM/μF^N\le F^{M-1}\le\pi_M/\mu for N≥M−1N\ge M-1. (iii) FN−M+1≤MF^{N-M+1}\le M fails as F→0F\to0 when N<M−1N<M-1.

Hence HQL's default N=MN=M is a valid gate for the policy it evaluates.

(2) Gate-only control. We agree that Eq. (6) with c≡1c\equiv1 is not the raw nn-step return, and correct lines 221–223 and Section 4.3; the paper's 16-window TD uses raw returns. We add Eq. (6) with c≡1c\equiv1, which differs from HQL only in the gate. The following result shows the Gate-only control is indeed the key of HQL.

cube-doublecube-triplescenehmaze-medantmaze-gianthmaze-giantmean
16-window TD (raw)710932838940
Eq. (6), c≡1c\equiv177±13±298±130±856±70±044
HQL87±254±4100±086±385±429±373

(3) Selection vs. shorter backups. Following Reviewer YXmF, we compare HQL with fixed nn-step backups and random traces:

cube-doublecube-triplescenehmaze-medantmaze-gianthmaze-giant
1-step TD76169571260
2-step TD75±10±098±172±268±312±3
4-step TD67±30±097±168±376±124±1
8-step TD63096675911
Random trace, c∼U(0,1)c\sim U(0,1)69±318±790±570±173±59±2
HQL87±254±4100±086±385±429±3

No fixed backup length works across tasks. The best nn is 1 on cube-double and cube-triple, 2 on scene and humanoidmaze-medium, and 4 on the giant mazes, and every n≥2n\ge2 scores 0 on cube-triple, where HQL reaches 54. Without tuning nn, HQL outperforms the best fixed nn by 18 points (73 vs. 55) and the per-task best nn by 13 (73 vs. 60). The gain comes from quality-based selection, not shorter backups (random traces: 55 vs. 73), which is what makes multi-step backups both usable and beneficial.

(4) Longer horizons on OCBench. We additionally evaluate on OCBench (Park et al., 2026): eight 50 Hz manipulation tasks with episodes of 1000–5000 steps and γ=0.9999\gamma=0.9999, i.e., an effective horizon of 10410^4 steps, versus 100–200 on OGBench. Following OCBench's own RL setup (25-step action chunks), Chunked-HQL still performs the best, with a mean success rate of 82%, against 53% for 100-step TD and 46% for 200-step TD, the backup used by OCBench's baselines. See the paper for details.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.