- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper addresses accurate off-policy Q-learning over long horizons. It motivates Horizon-Scalable Q-Learning (HQL) through the trade-off between accumulated one-step TD errors and bias from suboptimal actions in multi-step continuations. HQL adapts the backup length by comparing each logged action's critic value with the best of several actions sampled from a behavior model. When the logged action passes this comparison, downstream value corrections propagate through it; otherwise, the target uses a one-step bootstrap. The gate requires action samples and critic evaluations, making it compatible with sampling-based policies without evaluating action densities. The method also extends to action chunks. Experiments on a corrupted combination-lock task report lower value-estimation errors across horizons and corruption levels, while offline-to-online OGBench experiments report higher average success rates than the evaluated TD and long-horizon baselines.
Critical strengths
-
HQL provides a simple critic-target modification that can use expressive action samplers without evaluating their densities. Algorithm 1 computes targets backward through a window and trains every position; Chunked-HQL separates decision stride from backup length. This makes the approach practically useful for continuous-control methods built around flow or other sampling-based policies. The QAM and VGF experiments provide further evidence that the critic modification can help beyond the main best-of-N policy.
-
The experiments address both parts of the motivation: increasing horizon and suboptimal logged continuations. The combination-lock task permits direct measurement of Q-value error, while the main OGBench study uses five seeds and matches the number of critic targets per update. After 1M offline gradient updates and a further 1M online interaction steps, Table 1 reports mean offline/online success rates of 40%/74% for HQL, 34%/62% for LQL, and 21%/45% for Q-chunking; Chunked-HQL reaches 41%/81%. These gains support the practical value of the method in offline-to-online learning. The trace statistics also give a useful account of its behavior: 77% of targets remain one-step, indicating that most extensions are selective and short.
Critical weaknesses
1. The contribution of the sampled gate is insufficiently isolated from existing trace mechanisms and other changes to the backup.
Watkins' cuts traces at non-greedy actions [1], Retrace propagates weighted TD corrections [2], and Daley's Gated Q-learning attenuates non-greedy traces without importance ratios [3]. HQL's distinguishing mechanism is its finite-candidate value comparison for sampling-based continuous-action policies. Establishing the value of this mechanism requires an explicit comparison of the target operators and a controlled comparison with a suitable sampling-based greedy-action gate. The reported baselines cover fixed-step TD, LQL, and Q-chunking, leaving this direct comparison open.
The intended gate-only control also has an unresolved target definition. Direct expansion of Eq. (6), with and two transitions ending at a zero-valued terminal state, gives
This differs from the raw two-step return by , whereas lines 221-223 and Section 4.3 identify the all-open recursion with an uncorrected return. The implemented target determines the interpretation of 16-window TD: using raw returns changes both gating and correction terms, while using Eq. (6) with requires correcting the stated special case. Appendix A.6 additionally changes the actor/bootstrap sample count between soft and hard gates (16 versus 4), and Appendix A.7 varies gate width and action-search width together. These comparisons therefore combine changes whose individual effects need to be separated.
2. The finite-sample derivation uses a coefficient bound that its estimator can violate.
Section 4.2 invokes , a sufficient coefficient condition for the contractive off-policy correction analyzed in [2]. It then replaces the exponential-tilt normalizer with a random estimate and calls the resulting clipped ratio the largest valid gate. A counterexample follows directly from Eq. (9): let mu assign probability to each of two actions with critic values 0 and 1. For and , all candidates have value 0 with probability . On this event, the estimated gate for the lower-valued logged action is 1, whereas its true tilted-policy density ratio is .
The estimated coefficient thus exceeds the stated bound with positive probability. The implementation uses a finite-best-of-N target policy, which differs from the exact exponential tilt in the derivation. The theoretical claim therefore requires an analysis of the coefficient actually used, its target policy, and any averaging or sample-independence assumptions. Fixed-point preservation and contraction should be stated and justified separately.
References
[1] Watkins, C. J. C. H. (1989). Learning from Delayed Rewards.
[2] Munos, R., Stepleton, T., Harutyunyan, A., and Bellemare, M. G. (2016). Safe and Efficient Off-Policy Reinforcement Learning.
[3] Daley, B. (2026). Gated Q-learning: Add Off-Policy Bias to Taste.
Questions for authors
-
Please give the exact implemented recursions for HQL, 16-window TD, and the tau=0 soft-gate control. Does the control retain the V^pi-Q correction? This clarification determines whether it isolates gating or changes the backup operator as well.
-
For which policy and in what sense is the finite-N gate valid: samplewise, or after averaging over candidate draws? Please address the counterexample above, distinguish fixed-point preservation from contraction, and specify any required independence between gate and bootstrap samples, including for QAM/VGF.
-
Please explain the operator-level distinction from Watkins' Q(lambda), Retrace, and Daley's Gated Q-learning, cited as [1]-[3] in Critical Weaknesses. A targeted comparison should hold the candidate sampler, bootstrap, windows, and target budget fixed. Please identify an appropriate continuous-action adaptation or explain the specific obstacle to adapting these mechanisms.
-
Which existing runs hold actor/bootstrap search width fixed while varying gate width or comparing soft and hard gates? Please distinguish conclusions supported by those controlled comparisons from conclusions about the complete configurations in Appendices A.6-A.7.
2 comments
We sincerely hope the reviewer could consider raising the score if no major concerns are left, that means a lot to us. We hope you all the best!
We thank the reviewer for pointing to Watkins' Q(λ), Retrace and Gated Q-learning.
W1 / Q3. Distinction from existing trace mechanisms. All four methods set the trace coefficient of a multi-step backup. They differ in what the coefficient needs from the target policy, which is why the first three do not apply directly to a flow or best-of-N policy with continuous actions.
W1 (cont.) / Q1. Exact recursions; does the τ=0 control keep the V−Q term? With oj=rj+γVˉ(sj+1), the implemented recursions are
W2 / Q2. Validity of the finite-N gate. See General response (1): the gate is valid in expectation for the best-of-M policy when N≥M−1.
Q4. Which runs isolate gate width or the soft/hard choice? See the table in our reply to Reviewer YXmF (Q4).