acceptodds
Under review as a conference paper at ICLR 2027

Flow Q-trace: Multi-step Critic Learning with Density-Ratio-Based OOD Action Handling

Abstract

In this paper, we consider multi-step critic learning for offline reinforcement learning (RL), moving beyond the widely-used one-step temporal difference methods. Although multi-step learning can offer more accurate value estimation, it has been underutilized in offline RL settings due to two challenges: the difficulty of correcting multi-step off-policyness via importance sampling, and value overestimation caused by bootstrapping from out-of-distribution (OOD) actions. We address these challenges by introducing a conditional density ratio estimator and develop Q-trace, an offline multi-step critic learning algorithm with several desirable properties. In our framework, the learned density ratio estimator is employed not only for off-policyness correction but also for effective OOD action handling through critic penalization and rejection sampling, exploiting its capacity to identify OOD actions in a pointwise manner. By integrating Q-trace with expressive flow-based policies and our density-ratio-based OOD handling, we propose Flow Q-trace (FQT), which demonstrates strong performance across diverse offline RL tasks.

Primary area:
reinforcement learning
Keywords:
offline reinforcement learning, multi-step critic learning, flow policies

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_MVPpOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work presents Flow Q‑trace, an offline multi‑step critic RL algorithm leveraging a learned conditional density ratio for off‑policy correction and point‑wise out‑of‑distribution action mitigation combined with expressive flow‑based policies.

Critical strengths
  1. This paper extends multi-step critic learning to offline DRL.
  2. The effectiveness of the proposed method is verified through experiments.
Critical weaknesses
  1. Motivation is unclear: For offline DRL, many methods already exist to address the Q-learning evaluation problem, so why must multi-step critic learning be adopted to solve Q-value evaluation? In other words, what obvious shortcomings do other types of methods have compared with multi-step critic learning? Meanwhile, the online DRL community has also developed a large number of methods to address inaccurate Q-learning evaluation. Compared with the authors' multi-step critic learning, what obvious shortcomings do these methods have when directly integrated into offline DRL?
  2. The experiments are insufficient: there are too few baselines, and the method is not validated on D4RL, a widely used benchmark, nor on low-quality offline datasets.
Questions for authors
  1. For offline DRL, many methods already exist to address the Q-learning evaluation problem, so why must multi-step critic learning be adopted to solve Q-value evaluation? In other words, what obvious shortcomings do other types of methods have compared with multi-step critic learning?
  2. The online DRL community has also developed a large number of methods to address inaccurate Q-learning evaluation. Compared with the authors' multi-step critic learning, what obvious shortcomings do these methods have when directly integrated into offline DRL?
  3. The method is not validated on the widely used D4RL, so it is unclear how the proposed method performs on D4RL (MuJoCo, Adroit, etc.).
  4. Although this paper uses a fixed sub-trajectory length of l=5l=5, it does not conduct a complete ablation on different sub-trajectory lengths ll, so it is unclear how to select the optimal backtracking length, and the sensitivity of the hyperparameter ll is insufficiently studied.
  5. The algorithm's performance on suboptimal and mixed-quality datasets is not tested. When the dataset contains a large number of suboptimal trajectories, how does multi-step Retrace + density ratio estimation perform? Will severe drift occur?
  6. Four baselines are too few. It is recommended to add more existing SOTA methods in offline DRL that address inaccurate Q-learning evaluation, as well as several methods from online DRL that address inaccurate Q-learning evaluation and are directly integrated into the offline setting for comparison, bringing the total number of baselines to around 10.
  7. This paper does not plot the curves of Q-values over training iterations, nor does it provide direct metrics (such as the MSE error between predicted Q-values and true Q-values) to directly and quantitatively demonstrate the improvement in Q-value estimation accuracy.
8 comments
AuthorsOpenReview ↗

Authors' response to reviewer MVPp (1/3)

We thank the reviewer for the constructive review. All changes and additions are marked in blue in the revised paper.

R1. Motivation for multi-step critic learning

Most offline RL methods focus on how to handle OOD actions and the resulting value overestimation, for example through behavior regularization or conservative value estimation, while keeping the one-step TD target. In addition to OOD actions, we also consider the critic errors on in-distribution state-action pairs, on which the one-step TD target relies through bootstrapping at every step. Multi-step critic learning can reduce the effect of these errors on the target by also using the rewards along the trajectory in the dataset, and we focus on Retrace(λ\lambda), a representative multi-step method with off-policy correction.

In the revised paper, we newly introduce two theoretical results that compare this multi-step target with one-step TD learning (Propositions 1 and 2). They are in Section 3.1 and their proofs are in Appendix C.2. These results suggest that, if we can learn an actor that does not sample OOD actions, our Retrace(λ\lambda) target with estimated ratios can be better than one-step TD learning.

Propositions 1 and 2 are briefly summarized below.

Proposition 1 (page 5). Let ρ^≥0\hat\rho\ge0 be an estimate of ρ=π/β\rho=\pi/\beta on supp(β)\mathrm{supp}(\beta), and let R^\hat{\mathcal R} be the Retrace(λ\lambda) operator (eq. (9) on page 4) with the estimated trace coefficient c^=λmin⁡(1,ρ^)\hat c=\lambda\min(1,\hat\rho).

  • (i) QπQ^\pi is a fixed point of R^\hat{\mathcal R} for every ρ^\hat\rho.
  • (ii) If c^≤ρ\hat c\le\rho on supp(β)\mathrm{supp}(\beta), then R^\hat{\mathcal R} is a γ\gamma-contraction in the sup norm with unique fixed point QπQ^\pi. For λ>0\lambda>0, the condition holds if ρ^≤ρ/λ\hat\rho\le\rho/\lambda.

Proposition 2 (page 6). Under the condition of Proposition 1(ii), let Rℓ^\hat{\mathcal R_\ell} be the depth-ℓ\ell Retrace(λ\lambda) target with c^\hat c, where ℓ=1\ell=1 is the one-step Bellman operator, and let εID\varepsilon_{\mathrm{ID}} and εOOD\varepsilon_{\mathrm{OOD}} bound the critic error ∣Q−Qπ∣|Q-Q^\pi| on in-distribution and OOD pairs, respectively. Then for any (s,a)(s,a) and ℓ≥1\ell\ge1,

∣Rℓ^Q(s,a)−Qπ(s,a)∣≤w^ID,ℓ(s,a)εID+w^OOD,ℓ(s,a)εOOD,|\hat{\mathcal R_\ell} Q(s,a)-Q^\pi(s,a)|\le\hat w_{\mathrm{ID},\ell}(s,a) \varepsilon_{\mathrm{ID}}+\hat w_{\mathrm{OOD},\ell}(s,a) \varepsilon_{\mathrm{OOD}},

where the ID weight w^ID,ℓ\hat w_{\mathrm{ID},\ell} is nonincreasing in the depth ℓ\ell and the OOD weight w^OOD,ℓ\hat w_{\mathrm{OOD},\ell} is nondecreasing.

In theory, the one-step TD target also has QπQ^\pi as its fixed point. In practice, however, the critic used for bootstrapping has function approximation errors, so the target computed from it is also inaccurate. Proposition 2 bounds how much these critic errors can affect the target. It shows that the multi-step target can reduce the worst-case contribution of in-distribution critic errors compared with one-step TD (ℓ=1\ell=1), while the contribution of OOD critic errors does not decrease with depth. We note that this is a statement on the worst-case bound, and it does not mean that the multi-step target is always better than one-step TD. If the learned policy generates no OOD actions, the OOD weight vanishes, and the smaller ID weight gives a tighter worst-case bound than one-step TD. It further suggests that multi-step critic learning can be more effective when combined with OOD action handling, which appears consistent with Table 2 (page 9): the multi-step target adds 1 point without OOD handling (53 to 54) and 4 points with POA and DRS (58 to 62).

AuthorsOpenReview ↗

Authors' response to reviewer MVPp (2/3)

R2. Ablation on the sub-trajectory length ll

In the revised paper, we newly introduce an ablation on the sub-trajectory length ll. It is in Figure 6(f) of Appendix D.3.

Here ll is the practical sub-trajectory length of FQT, i.e., the multi-step target is computed from a sub-trajectory (st,at,rt,st+1,…,rt+l−1,st+l)(s_t,a_t,r_t,s_{t+1},\ldots,r_{t+l-1},s_{t+l}) sampled from the dataset (Algorithm 1 on page 8). We vary ll with POA, DRS, and all other hyperparameters fixed, so that l=1l=1 corresponds to one-step critic learning (FQL+POA&DRS). Success rate (%), averaged over 5 singletasks and 8 seeds. "Final" is the average over the last three evaluation epochs (800K, 900K, and 1M gradient steps), as in Table 2, and "500K" is the evaluation at 500K gradient steps.

Task categoryEvaluationl=1l=1l=2l=2l=3l=3l=5l=5 (default)l=10l=10
humanoidmaze-large500K5 ±\pm 413 ±\pm 512 ±\pm 716 ±\pm 719 ±\pm 9
Final19 ±\pm 1022 ±\pm 1026 ±\pm 1028 ±\pm 1030 ±\pm 10
cube-double500K28 ±\pm 734 ±\pm 738 ±\pm 244 ±\pm 646 ±\pm 3
Final41 ±\pm 346 ±\pm 445 ±\pm 449 ±\pm 448 ±\pm 2

With POA and DRS fixed, a longer sub-trajectory tends to make learning faster. At 500K gradient steps, the success rate tends to increase with ll, from 5 (l=1l=1) to 16 (l=5l=5) and 19 (l=10l=10) on humanoidmaze-large, and from 28 to 44 and 46 on cube-double. The final performance also tends to increase with ll, from 19 (l=1l=1) to 28 (l=5l=5) on humanoidmaze-large and from 41 to 49 on cube-double, and it tends to be comparable for l≥5l\ge5.

These results appear consistent with Proposition 2 in R1: a longer sub-trajectory can reduce the worst-case contribution of in-distribution critic errors, and with OOD actions handled by POA and DRS, this tends to lead to faster learning and higher performance.

AuthorsOpenReview ↗

Authors' response to reviewer MVPp (3/3)

R3. Additional baselines and dataset quality

Regarding the number of baselines, we also report the results of FQT under the discount-factor setting of the original FQL paper [1] (Table 2 on page 9). In this setting, we use the same 50 OGBench tasks, the same discount factors, and the same evaluation protocol as [1] (the average over the last three evaluation epochs, 8 seeds), so our results can be compared with the offline RL baselines reported in [1].

The comparison is summarized below. The baseline results are taken from Table 2 of [1], except FQL, for which we report our reproduction (Table 2 on page 9). The baselines include Gaussian-policy methods (BC, IQL [2], ReBRAC [3]), diffusion-policy methods (IDQL [4], SRPO [5], CAC [6]), and flow-policy methods (FAWAC, FBRAC, and IFQL, which are flow-based baselines from [1], and FQL [1]). Success rate (%), averaged over 5 singletasks of each category.

Task categoryBCIQLReBRACIDQLSRPOCACFAWACFBRACIFQLFQLFQT
antmaze-large1153812111336602878 ±\pm 281 ±\pm 2
antmaze-giant042600004310 ±\pm 523 ±\pm 3
humanoidmaze-medium23322115319386054 ±\pm 654 ±\pm 15
humanoidmaze-large12210002114 ±\pm 128 ±\pm 10
antsoccer-arena180121212163360 ±\pm 258 ±\pm 3
cube-single5839195808581797996 ±\pm 198 ±\pm 1
cube-double271215265151430 ±\pm 349 ±\pm 4
scene5284146204030453058 ±\pm 359 ±\pm 1
puzzle-3x329211018196141929 ±\pm 2100 ±\pm 0
puzzle-4x407142910151132519 ±\pm 138 ±\pm 3
Average323312314251629304459

The "Average" row is the mean over the 10 task categories. Since we follow the evaluation protocol of [1], other methods evaluated under the same protocol can also be compared with our results.

Regarding suboptimal datasets, in OGBench the five singletasks of each task category share the same dataset, with the rewards relabeled for each singletask (Appendix E.1). The dataset consists of trajectories that are not collected for a specific singletask, so for a given singletask, most of the dataset may be unrelated to the task or suboptimal for it.

[1] Seohong Park, Qiyang Li, and Sergey Levine. "Flow Q-Learning." ICML 2025.

[2] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. "Offline Reinforcement Learning with Implicit Q-Learning." ICLR 2022.

[3] Denis Tarasov, et al. "Revisiting the Minimalist Approach to Offline Reinforcement Learning." NeurIPS 2023.

[4] Philippe Hansen-Estruch, et al. "IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies." arXiv 2023.

[5] Huayu Chen, et al. "Score Regularized Policy Optimization through Diffusion Behavior." ICLR 2024.

[6] Zihan Ding and Chi Jin. "Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learning." ICLR 2024.

Reviewer_rEEgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Flow Q-trace (FQT) for multi-step critic learning in offline reinforcement learning. The method combines flow-based policies with density-ratio estimation for off-policy correction and out-of-distribution (OOD) action handling. Experiments on OGBench show improved overall performance compared with the evaluated baselines.

Critical strengths
  • The use of a conditional density-ratio estimator to obtain importance sampling ratios directly from samples is interesting, as it enables multi-step off-policy correction without requiring explicit evaluation of the behavior or target policy densities.
Critical weaknesses
  • Although the authors state that their focus is on multi-step critic learning for offline RL, the reported results provide limited evidence of its standalone benefit. In Table 2, FQL and QT-FQL show similar performance across all OGBench task categories, with overlapping mean ±\pm standard deviation ranges. The additional components, POA (Penalizing OOD Actions) and DRS (Density-ratio-based Rejection Sampling), appear to play a more important role in improving performance. The authors also acknowledge the benefits of POA and DRS and the modest improvement from multi-step critic learning alone in lines 465–466 and 468–471, respectively. These observations raise concerns about the strength of the paper’s empirical contribution specifically to multi-step critic learning in offline RL.
Questions for authors
  • The proposed FQT combines Q-trace with FQL, POA, and DRS. One consideration is that FQL employs a behavior-regularized actor, while POA encourages conservative value estimates by penalizing overestimated values for OOD actions. The latter follows a similar motivation to CQL [1] and subsequent work. Given the similar performance of FQL and QT-FQL, the reviewer wonders whether behavior regularization might limit the additional benefit of density-ratio-based off-policy correction, and whether combining it with POA could further limit this benefit. To investigate this, the reviewer believes the authors should also evaluate their approach using Gaussian policies rather than only flow-based policies.

  • Although learning a conditional density-ratio estimator requires an additional network, the reviewer acknowledges that the reported memory and inference overhead is limited. However, the overall method introduces several additional hyperparameters. POA requires an OOD identification threshold ζ\zeta and a penalization coefficient η\eta, while the trace-decay parameter λ\lambda must also be considered. Table 6 further illustrates the additional hyperparameter choices involved.

  • The ablation studies in Section D.3 suggest that the method is sensitive to several hyperparameters. The current analysis varies one hyperparameter at a time while keeping the others fixed. The reviewer suspects that varying multiple hyperparameters simultaneously could substantially change the results. In other words, the reviewer is concerned that the method may be sensitive to the joint choice of hyperparameters.

  • The authors choose Retrace among the multi-step operators used in online RL, but many other operators also use importance sampling, including α\alpha-Retrace [2] and C-trace [2], leaving several alternatives worth considering. There are also algorithms such as Peng’s Q(λ\lambda) that do not require importance sampling ratios. Without comparisons with these alternative multi-step operators, the empirical justification for choosing Retrace is unclear. Furthermore, the recent CPQL [3] paper discusses how Peng’s Q(λ\lambda) induces implicit behavior regularization in offline RL, which can help mitigate Q-value overestimation. Since FQT also incorporates POA to address overestimation, comparisons with these related algorithms are essential.

Minor

  • The experimental comparison is too limited, with the offline RL baselines restricted to FQL, FAC, and QC-FQL.

[1] Kumar, Aviral, et al. “Conservative Q-Learning for Offline Reinforcement Learning.” Advances in Neural Information Processing Systems 33 (2020): 1179–1191.

[2] Rowland, Mark, Will Dabney, and Rémi Munos. “Adaptive Trade-Offs in Off-Policy Learning.” International Conference on Artificial Intelligence and Statistics. PMLR, 2020.

[3] Byeongchan Kim and Min-hwan Oh. “Peng’s Q(λ\lambda) for Conservative Value Estimation in Offline Reinforcement Learning.” In The Fourteenth International Conference on Learning Representations, 2026.

5 comments
AuthorsOpenReview ↗

Authors' response to reviewer rEEg (1/3)

We thank the reviewer for the constructive review. All changes and additions are marked in blue in the revised paper.

R1. Contribution of multi-step critic learning

In the revised paper, we newly introduce two theoretical results on the multi-step target with estimated ratios (Propositions 1 and 2) and a new ablation on the sub-trajectory length ll. Propositions 1 and 2 are in Section 3.1 and their proofs are in Appendix C.2. The ablation is in Figure 6(f) of Appendix D.3.

These results suggest that, if we can learn an actor that does not sample OOD actions, our Retrace(λ\lambda) target with estimated ratios can be better than one-step TD learning. This may explain why, as the reviewer observed in Table 2, the gain of the multi-step target appears mainly together with POA and DRS.

Propositions 1 and 2 and the ablation are briefly summarized below.

Proposition 1 (page 5). Let ρ^≥0\hat\rho\ge0 be an estimate of ρ=π/β\rho=\pi/\beta on supp(β)\mathrm{supp}(\beta), and let R^\hat{\mathcal R} be the Retrace(λ\lambda) operator (eq. (9) on page 4) with the estimated trace coefficient c^=λmin⁡(1,ρ^)\hat c=\lambda\min(1,\hat\rho).

  • (i) QπQ^\pi is a fixed point of R^\hat{\mathcal R} for every ρ^\hat\rho.
  • (ii) If c^≤ρ\hat c\le\rho on supp(β)\mathrm{supp}(\beta), then R^\hat{\mathcal R} is a γ\gamma-contraction in the sup norm with unique fixed point QπQ^\pi. For λ>0\lambda>0, the condition holds if ρ^≤ρ/λ\hat\rho\le\rho/\lambda.

Proposition 2 (page 6). Under the condition of Proposition 1(ii), let Rℓ^\hat{\mathcal R_\ell} be the depth-ℓ\ell Retrace(λ\lambda) target with c^\hat c, where ℓ=1\ell=1 is the one-step Bellman operator, and let εID\varepsilon_{\mathrm{ID}} and εOOD\varepsilon_{\mathrm{OOD}} bound the critic error ∣Q−Qπ∣|Q-Q^\pi| on in-distribution and OOD pairs, respectively. Then for any (s,a)(s,a) and ℓ≥1\ell\ge1,

∣Rℓ^Q(s,a)−Qπ(s,a)∣≤w^ID,ℓ(s,a)εID+w^OOD,ℓ(s,a)εOOD,|\hat{\mathcal R_\ell} Q(s,a)-Q^\pi(s,a)|\le\hat w_{\mathrm{ID},\ell}(s,a) \varepsilon_{\mathrm{ID}}+\hat w_{\mathrm{OOD},\ell}(s,a) \varepsilon_{\mathrm{OOD}},

where the ID weight w^ID,ℓ\hat w_{\mathrm{ID},\ell} is nonincreasing in the depth ℓ\ell and the OOD weight w^OOD,ℓ\hat w_{\mathrm{OOD},\ell} is nondecreasing.

Proposition 2 shows that the multi-step target can reduce the worst-case contribution of in-distribution critic errors compared with one-step TD, while the contribution of OOD critic errors does not decrease with depth. Regarding the question of whether behavior regularization and POA limit the benefit of the density-ratio-based correction, this result suggests the opposite direction: handling OOD actions may be what allows the multi-step target to be beneficial. This appears consistent with Table 2 (page 9), where the multi-step target adds 1 point without OOD handling (53 to 54) and 4 points with POA and DRS (58 to 62).

Ablation on ll (Figure 6(f)). Here ll is the practical sub-trajectory length of FQT, i.e., the multi-step target is computed from a sub-trajectory (st,at,rt,st+1,…,rt+l−1,st+l)(s_t,a_t,r_t,s_{t+1},\ldots,r_{t+l-1},s_{t+l}) sampled from the dataset (Algorithm 1 on page 8). We vary ll with POA, DRS, and all other hyperparameters fixed, so that l=1l=1 corresponds to one-step critic learning (FQL+POA&DRS). Success rate (%), averaged over 5 singletasks and 8 seeds. "Final" is the average over the last three evaluation epochs (800K, 900K, and 1M gradient steps), as in Table 2, and "500K" is the evaluation at 500K gradient steps.

Task categoryEvaluationl=1l=1l=2l=2l=3l=3l=5l=5 (default)l=10l=10
humanoidmaze-large500K5 ±\pm 413 ±\pm 512 ±\pm 716 ±\pm 719 ±\pm 9
Final19 ±\pm 1022 ±\pm 1026 ±\pm 1028 ±\pm 1030 ±\pm 10
cube-double500K28 ±\pm 734 ±\pm 738 ±\pm 244 ±\pm 646 ±\pm 3
Final41 ±\pm 346 ±\pm 445 ±\pm 449 ±\pm 448 ±\pm 2

One-step critic learning (l=1l=1) tends to be the slowest and lowest-performing setting, and longer sub-trajectories tend to accelerate learning. This may support the contribution of the multi-step target itself.

AuthorsOpenReview ↗

Authors' response to reviewer rEEg (2/3)

R2. Joint hyperparameter sensitivity

In the revised paper, we newly introduce a joint hyperparameter sensitivity analysis. It is in Figure 7 of Appendix D.3.1.

We vary each pair of (λ,η,ζ)(\lambda,\eta,\zeta) jointly on a 3×33\times3 grid, with the remaining one fixed to its default value, on the default tasks of humanoidmaze-large (task1) and cube-double (task2). The results are briefly summarized below. Success rate (%), averaged over the last three evaluation epochs and 8 seeds. The default configuration, marked in bold, is (λ,η,ζ)=(0.7,1,0.99)(\lambda,\eta,\zeta)=(0.7,1,0.99) for humanoidmaze-large and (0.7,0.1,0.99)(0.7,0.1,0.99) for cube-double.

Results over η×ζ\eta\times\zeta, with λ=0.7\lambda=0.7:

humanoidmaze-largecube-double
ζ=0.9\zeta=0.9ζ=0.99\zeta=0.99ζ=0.9999\zeta=0.9999ζ=0.9\zeta=0.9ζ=0.99\zeta=0.99ζ=0.9999\zeta=0.9999
η=0.1\eta=0.113 ±\pm 127 ±\pm 83 ±\pm 353 ±\pm 1064 ±\pm 648 ±\pm 9
η=1\eta=129 ±\pm 732 ±\pm 2112 ±\pm 1428 ±\pm 545 ±\pm 446 ±\pm 8
η=10\eta=1011 ±\pm 121 ±\pm 12 ±\pm 412 ±\pm 99 ±\pm 511 ±\pm 7

Results over λ×η\lambda\times\eta, with ζ=0.99\zeta=0.99:

humanoidmaze-largecube-double
η=0.1\eta=0.1η=1\eta=1η=10\eta=10η=0.1\eta=0.1η=1\eta=1η=10\eta=10
λ=0.3\lambda=0.38 ±\pm 424 ±\pm 235 ±\pm 752 ±\pm 745 ±\pm 88 ±\pm 3
λ=0.7\lambda=0.77 ±\pm 832 ±\pm 211 ±\pm 164 ±\pm 645 ±\pm 49 ±\pm 5
λ=0.9\lambda=0.99 ±\pm 845 ±\pm 114 ±\pm 661 ±\pm 743 ±\pm 811 ±\pm 8

Results over λ×ζ\lambda\times\zeta, with η=1\eta=1 for humanoidmaze-large and η=0.1\eta=0.1 for cube-double:

humanoidmaze-largecube-double
ζ=0.9\zeta=0.9ζ=0.99\zeta=0.99ζ=0.9999\zeta=0.9999ζ=0.9\zeta=0.9ζ=0.99\zeta=0.99ζ=0.9999\zeta=0.9999
λ=0.3\lambda=0.336 ±\pm 1124 ±\pm 238 ±\pm 1053 ±\pm 452 ±\pm 743 ±\pm 5
λ=0.7\lambda=0.729 ±\pm 732 ±\pm 2112 ±\pm 1453 ±\pm 1064 ±\pm 648 ±\pm 9
λ=0.9\lambda=0.935 ±\pm 845 ±\pm 1113 ±\pm 759 ±\pm 461 ±\pm 751 ±\pm 8

The joint results show trends similar to the one-at-a-time ablations (Figure 6 in Appendix D.3) on both tasks. The penalization coefficient η\eta tends to be the dominant factor, and its best value (η=1\eta=1 on humanoidmaze-large and η=0.1\eta=0.1 on cube-double) remains the best for every tested λ\lambda and ζ\zeta. An overly large threshold ζ=0.9999\zeta=0.9999 tends to lower performance for every λ\lambda, and λ\lambda has a comparatively smaller effect. These results also indicate that a better-performing configuration than our default may exist. However, we use POA and DRS as add-on components to multi-step critic learning. Our aim was to examine whether they are effective when added, rather than to find the best configuration. To avoid excessive hyperparameter tuning, FQT reuses (α,λ)(\alpha,\lambda) selected for QT-FQL with ζ=0.99\zeta=0.99 fixed, and we search η∈{0.1,1,10}\eta\in\lbrace 0.1,1,10\rbrace (as in Appendix E.3).

AuthorsOpenReview ↗

Authors' response to reviewer rEEg (3/3)

R3. Comparison with other multi-step critic targets

In the revised paper, we newly introduce a comparison of multi-step critic targets, including Peng's Q(λ)Q(\lambda). It is in Table 5 of Appendix D.4, and the relation to CPQL [3] is discussed in Appendix B.

All variants use the same flow-based policy extraction of FQL, without POA or DRS, and differ in the critic target. PQL-FQL (Peng's Q(λ)Q(\lambda) [1]) and HQL-FQL (Harutyunyan's Q(λ)Q(\lambda) [2], c=λc=\lambda) do not use importance-sampling-based correction, whereas QT-FQL uses c=λmin⁡(1,ρ^)c=\lambda\min(1,\hat\rho) with ρ^\hat\rho estimated by Γw\Gamma_w. Success rate (%), averaged over 5 singletasks and 8 seeds.

Task categoryλ\lambdaPQL-FQLHQL-FQLQT-FQL
humanoidmaze-large0.16 ±\pm 25 ±\pm 23 ±\pm 2
0.39 ±\pm 35 ±\pm 24 ±\pm 2
0.512 ±\pm 44 ±\pm 24 ±\pm 2
0.715 ±\pm 34 ±\pm 15 ±\pm 2
0.916 ±\pm 31 ±\pm 14 ±\pm 2
cube-double0.123 ±\pm 227 ±\pm 327 ±\pm 4
0.39 ±\pm 127 ±\pm 227 ±\pm 4
0.53 ±\pm 127 ±\pm 229 ±\pm 4
0.74 ±\pm 123 ±\pm 330 ±\pm 3
0.94 ±\pm 120 ±\pm 230 ±\pm 3

HQL-FQL and QT-FQL differ in whether the estimated ratio is used. QT-FQL remains relatively stable across λ\lambda, whereas HQL-FQL tends to degrade at large λ\lambda, which suggests that the ratio estimated by Γw\Gamma_w may provide a useful correction.

PQL-FQL tends to improve with larger λ\lambda on humanoidmaze-large, while its performance is lower for λ≥0.3\lambda\ge0.3 on cube-double. As the reviewer mentioned for CPQL, this may be related to the implicit behavior regularization of Peng's Q(λ)Q(\lambda). It may also be connected to Table 2 (page 9), where POA is particularly beneficial on humanoidmaze-large (from 5 to 13 for QT-FQL): a behavior-biased fixed point and explicit penalization of OOD actions might address a similar issue in this task category. Our method takes the latter approach while aiming to keep the fixed point QπQ^\pi, and with POA and DRS, FQT reaches 28 on humanoidmaze-large and 49 on cube-double (Table 2). We also note that CPQL is built on Gaussian policies while we focus on flow-based actors, so these results may be viewed as an indirect comparison with a Peng's Q(λ)Q(\lambda)-based target under an expressive flow policy.

Regarding α\alpha-Retrace and C-trace [4], α\alpha-Retrace applies Retrace to the mixture policy πα=απ+(1−α)β\pi_\alpha=\alpha\pi+(1-\alpha)\beta, so that the trace coefficient becomes (1−α)+αmin⁡(1,π/β)(1-\alpha)+\alpha\min(1,\pi/\beta), and C-trace adapts α\alpha to reach a target contraction rate. Roughly speaking, these operators interpolate between Retrace (α=1\alpha=1) and a behavior-biased multi-step target in the spirit of Peng's Q(λ)Q(\lambda). Since the ratio π/β\pi/\beta is estimated by Γw\Gamma_w in our framework, these operators could be combined with our density ratio estimator. Given the above results of PQL-FQL and QT-FQL, we think this is a promising future direction.

[1] Jing Peng and Ronald J. Williams. "Incremental Multi-Step Q-Learning." Machine Learning Proceedings 1994.

[2] Anna Harutyunyan, et al. "Q(λ\lambda) with Off-Policy Corrections." ALT 2016.

[3] Byeongchan Kim and Min-hwan Oh. "Peng's Q(λ\lambda) for Conservative Value Estimation in Offline Reinforcement Learning." ICLR 2026.

[4] Mark Rowland, Will Dabney, and Remi Munos. "Adaptive Trade-Offs in Off-Policy Learning." AISTATS 2020.

Reviewer_RzPMOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Flow Q-Trace (FQT), an offline reinforcement learning algorithm that integrates multi-step critic learning with density-ratio-based out-of-distribution (OOD) action handling. The method builds on Retrace(λ) and estimates the target-to-behavior policy density ratio using a binary classifier trained on policy and dataset actions. The estimated density ratio is further used for critic penalization (POA) and rejection sampling (DRS). The complete method combines these components with the flow-based actor from FQL.

Experiments on 50 OGBench tasks demonstrate improvements over several existing offline RL methods.

While the empirical results are promising, I have substantial concerns regarding the validity of the theoretical guarantees, the reliability of the density-ratio-based OOD identification, and the extent to which multi-step critic learning contributes to the reported improvements.

Critical strengths

The paper addresses a relevant problem: enabling multi-step critic learning in offline RL without requiring explicit behavior-policy likelihoods.

Using a shared density-ratio estimator for off-policy correction and OOD handling is a coherent design choice.

The experimental evaluation is relatively extensive, covering 50 OGBench tasks with eight random seeds. The authors also provide comparisons with strong baselines and component-level ablations.

Critical weaknesses
  1. The theoretical support-constraint guarantee appears insufficiently justified.

My main concern is Theorem 3, which claims that a policy trained using the proposed critic penalty and behavior-regularized actor objective is guaranteed to assign zero probability to actions outside the support of the behavior policy.

In Appendix C.3, the proof of Theorem 2 rewrites an expectation under the target policy as an expectation under the behavior policy weighted by π(a|s)/β(a|s). However, this change of measure is not valid when β(a|s) = 0 and π(a|s) > 0, precisely the OOD-action case considered in Theorem 3.

The subsequent proof substitutes β(a|s) = 0 into an expression involving division by β(a|s), concluding that the penalized Q-value becomes negative infinity.

This argument is problematic. For an action with zero behavior-policy density but positive target-policy density, the original critic objective contains a linear penalty without a corresponding squared Bellman-error term. Its infimum can be unbounded below, rather than yielding a well-defined finite critic solution to which the derived operator applies.

Furthermore, the analysis assumes exact density ratios and idealized optimization, whereas the practical implementation uses a learned classifier, finite neural networks, and a threshold-based approximation.

Consequently, I do not think the current analysis establishes the claimed support-constrained policy guarantee for the proposed practical algorithm. This is a significant concern because the guarantee is presented as a central theoretical contribution.

  1. The density-ratio estimator is not sufficiently validated as a reliable OOD detector.

The proposed method uses

Γ(s,a) = π(a|s) / [π(a|s) + β(a|s)]

to estimate importance weights and identify OOD actions.

However, the OOD decision is based on a threshold on a relative density ratio, rather than on the absolute density or support of the behavior policy.

For example, with the default threshold ζ = 0.99, an action with β(a|s) > 0 can still be classified as OOD whenever π(a|s)/β(a|s) is sufficiently large. Conversely, a low-density action need not be rejected when the two policies assign similarly low densities.

Therefore, the criterion measures relative policy-distribution mismatch and should not automatically be interpreted as exact identification of actions outside the dataset support.

The empirical analysis in Appendix D.2 reports the fractions of policy and dataset actions classified as OOD, but it does not directly establish the accuracy or calibration of this classifier against known behavior densities or support labels.

Since the same estimator is used for importance correction, critic penalization, and rejection sampling, its errors can affect multiple components simultaneously. A more direct evaluation of estimation accuracy and robustness seems necessary.

  1. The contribution of multi-step critic learning to the final performance is relatively modest.

The paper emphasizes moving beyond one-step critic learning as a central contribution. However, the component analysis in Table 2 suggests that much of the improvement arises from the proposed OOD-handling mechanisms.

Under the tuned discount-factor setting, the average scores are:

FQL: 53

QT-FQL (multi-step alone): 54

FQL + POA + DRS: 58

FQT (multi-step + POA + DRS): 62

Thus, introducing the multi-step target without OOD handling improves the average score by only one point. Even after incorporating POA and DRS, the additional improvement attributable to multi-step learning is four points.

These gains are useful, but they are considerably smaller than the overall improvement relative to the original one-step baseline.

Given that the paper's primary motivation is improved multi-step value estimation, I would expect more direct evidence demonstrating the benefits of the proposed critic target, beyond the combined performance gains of multiple components.

In particular, the paper does not sufficiently establish whether the improvements correspond to more accurate value estimates, lower bootstrapping bias, or merely better downstream policy performance.

  1. Limited analysis of density-ratio estimation errors in multi-step learning.

The favorable properties of Retrace(λ) are established for importance weights satisfying the appropriate conditions relative to the true target and behavior policies.

In the proposed method, however, these weights are replaced by estimates from a learned classifier that is updated jointly with the policy.

Estimation errors may lead to incorrect trace coefficients, potentially affecting the contraction properties and fixed-point behavior relied upon in the theoretical motivation.

The paper does not provide a theoretical error analysis or sufficient empirical diagnostics for this approximation.

Although Appendix D.2 analyzes cumulative trace weights, it does not quantify their estimation error relative to the true density ratios or establish how much effective multi-step information is retained during training.

This leaves an important gap between the idealized Retrace formulation and its practical offline implementation.

  1. The empirical validation is concentrated on one benchmark suite.

Despite evaluating 50 tasks, all experiments use state-based singletask variants of OGBench.

The results establish competitiveness in this benchmark, but do not fully demonstrate robustness across different offline data distributions, behavior-policy coverage levels, or settings where density-ratio estimation is particularly challenging.

The additional training and inference costs introduced by the classifier, multi-step targets, and candidate-action filtering also warrant consideration when interpreting the relatively modest performance differences between the strongest variants.

Questions for authors

How is the change-of-measure argument in Appendix C.3 justified when β(a|s) = 0 but π(a|s) > 0? Does Theorem 3 require additional assumptions or a different proof?

Can the authors provide a quantitative evaluation of density-ratio estimation accuracy and OOD detection calibration, ideally in controlled settings where ground-truth densities are available?

Can the authors directly measure critic estimation error or bootstrapping bias to demonstrate the advantages of Q-trace over one-step TD learning?

How sensitive is the method to density-ratio estimation errors, especially when the behavior and target policies have substantially different action distributions?

What is the practical effective backup depth of Q-trace as training progresses, and how does this relate to the observed performance gains?

5 comments
AuthorsOpenReview ↗

Authors' response to reviewer RzPM (1/3)

We thank the reviewer for the careful review. All changes and additions are marked in blue in the revised paper.

R1. Support-constraint analysis

We thank the reviewer for pointing out the problem in Theorems 2 and 3. We revised both theorems so that no change of measure is used, and refined their conclusions accordingly. The revised Theorems 2 and 3 are in Section 3.2, and Lemma 4, which supports the assumption of Theorem 3, is in Appendix C.3. All proofs are in Appendix C.3.

Theorems 2 and 3 and Lemma 4 are briefly summarized below.

Theorem 2 (page 7). In the tabular setting, consider the proposed critic objective (eq. (14) on page 6 of the revised paper) with the exact estimator Γ∗\Gamma^*, η>0\eta>0, and ζ∈(0,1]\zeta\in(0,1]. Let dDd^{\mathcal D} be the dataset state distribution. At every state-action pair (s,a)(s,a) with dD(s)>0d^{\mathcal D}(s)>0:

  • (i) If β(a∣s)>0\beta(a|s)>0 (in-distribution), the unique minimizing value of Q(s,a)Q(s,a) is Rλπ,βQ(s,a)−ηπ(a∣s)2β(a∣s)1[Γ∗(s,a)≥ζ]\mathcal R_\lambda^{\pi,\beta}Q(s,a)-\frac{\eta \pi(a|s)}{2\beta(a|s)}\mathbf{1}[\Gamma^*(s,a)\ge\zeta].
  • (ii) If β(a∣s)=0<π(a∣s)\beta(a|s)=0<\pi(a|s) (OOD), a critic gradient step of size ν\nu decreases Q(s,a)Q(s,a) by νdD(s)ηπ(a∣s)\nu d^{\mathcal D}(s) \eta \pi(a|s).
  • (iii) If β(a∣s)=π(a∣s)=0\beta(a|s)=\pi(a|s)=0, the objective is independent of Q(s,a)Q(s,a).

Theorem 3 (page 7). In the tabular setting of Theorem 2, at each round kk let the actor πk\pi_k be ϵ\epsilon-optimal for the behavior-regularized actor objective (eq. (3) on page 3) under the critic QkQ_k, and let Qk+1Q_{k+1} be obtained by one critic gradient step on eq. (14) under πk\pi_k. If the critic values Qk(s,a)Q_k(s,a) on in-distribution pairs are bounded (supported by Lemma 4), then

lim⁡k→∞∑a:β(a∣s)=0πk(a∣s)=0for all s with dD(s)>0.\lim_{k\to\infty}\sum_{a:\beta(a|s)=0}\pi_k(a|s)=0\qquad\text{for all }s\text{ with }d^{\mathcal D}(s)>0 .

Lemma 4 (page 25). The boundedness of in-distribution critic values holds for step size 0<ν≤12max⁡s,adD(s)β(a∣s)0<\nu\le\frac{1}{2\max_{s,a}d^{\mathcal D}(s)\beta(a|s)}.

Theorem 3 shows that, in the tabular setting, the actor's probability of proposing OOD actions tends to zero at dataset states under the proposed critic objective. In practice, POA discourages OOD action proposals during training, but the learned actor may still propose OOD actions. We therefore additionally use density-ratio-based rejection sampling (DRS) at execution time.

R2. Density-ratio estimator as an OOD detector

We newly added Lemma 3 in the revised paper, which gives a sufficient condition under which the learned estimator Γw\Gamma_w identifies OOD actions. Lemma 3 and its proof are in Appendix C.1, and it is referred to in Section 3.3. Lemma 3 is briefly summarized below.

Lemma 3 (page 20). Let Γ∗\Gamma^* be the optimal estimator and ζ∈(0,1]\zeta\in(0,1] the OOD threshold used by POA and DRS. For an OOD state-action pair (s,a)(s,a) with β(a∣s)=0\beta(a|s)=0 and π(a∣s)>0\pi(a|s)>0, if

∣Γw(s,a)−Γ∗(s,a)∣<1−ζ,\big|\Gamma_w(s,a)-\Gamma^*(s,a)\big|<1-\zeta,

then Γw(s,a)≥ζ\Gamma_w(s,a)\ge\zeta, i.e., every OOD action proposed by π\pi is identified.

Lemma 3 gives the role of the margin 1−ζ1-\zeta: with a threshold ζ<1\zeta<1, OOD actions are identified even when the learned estimator Γw\Gamma_w is not exact, as long as its error is smaller than the margin.

We agree with the example in Weakness 2: with such a margin, some in-distribution pairs, with β(a∣s)≤1−ζζπ(a∣s)\beta(a|s)\le\frac{1-\zeta}{\zeta}\pi(a|s), can also be identified as OOD. However, an overly large threshold (ζ=0.9999\zeta=0.9999), i.e., a very small margin, tends to degrade performance, particularly on humanoidmaze-large, as shown in Figure 6(d) of Appendix D.3.

We believe that not missing OOD actions matters more for multi-step critic learning than occasionally penalizing such in-distribution actions, and R3 below discusses a possible reason.

AuthorsOpenReview ↗

Authors' response to reviewer RzPM (2/3)

R3. Contribution of multi-step critic learning

In the revised paper, we newly introduce two theoretical results on the multi-step target with estimated ratios (Propositions 1 and 2) and a new ablation on the sub-trajectory length ll. Propositions 1 and 2 are in Section 3.1 and their proofs are in Appendix C.2. The ablation is in Figure 6(f) of Appendix D.3.

These results suggest that, if we can learn an actor that does not sample OOD actions, our Retrace(λ\lambda) target with estimated ratios can be better than one-step TD learning.

Propositions 1 and 2 and the ablation are briefly summarized below.

Proposition 1 (page 5). Let ρ^≥0\hat\rho\ge0 be an estimate of ρ=π/β\rho=\pi/\beta on supp(β)\mathrm{supp}(\beta), and let R^\hat{\mathcal R} be the Retrace(λ\lambda) operator (eq. (9) on page 4) with the estimated trace coefficient c^=λmin⁡(1,ρ^)\hat c=\lambda\min(1,\hat\rho).

  • (i) QπQ^\pi is a fixed point of R^\hat{\mathcal R} for every ρ^\hat\rho.
  • (ii) If c^≤ρ\hat c\le\rho on supp(β)\mathrm{supp}(\beta), then R^\hat{\mathcal R} is a γ\gamma-contraction in the sup norm with unique fixed point QπQ^\pi. For λ>0\lambda>0, the condition holds if ρ^≤ρ/λ\hat\rho\le\rho/\lambda.

Proposition 2 (page 6). Under the condition of Proposition 1(ii), let Rℓ^\hat{\mathcal R_\ell} be the depth-ℓ\ell Retrace(λ\lambda) target with c^\hat c, where ℓ=1\ell=1 is the one-step Bellman operator, and let εID\varepsilon_{\mathrm{ID}} and εOOD\varepsilon_{\mathrm{OOD}} bound the critic error ∣Q−Qπ∣|Q-Q^\pi| on in-distribution and OOD pairs, respectively. Then for any (s,a)(s,a) and ℓ≥1\ell\ge1,

∣Rℓ^Q(s,a)−Qπ(s,a)∣≤w^ID,ℓ(s,a)εID+w^OOD,ℓ(s,a)εOOD,|\hat{\mathcal R_\ell} Q(s,a)-Q^\pi(s,a)|\le\hat w_{\mathrm{ID},\ell}(s,a) \varepsilon_{\mathrm{ID}}+\hat w_{\mathrm{OOD},\ell}(s,a) \varepsilon_{\mathrm{OOD}},

where the ID weight w^ID,ℓ\hat w_{\mathrm{ID},\ell} is nonincreasing in the depth ℓ\ell and the OOD weight w^OOD,ℓ\hat w_{\mathrm{OOD},\ell} is nondecreasing.

Proposition 1 gives the condition under which the multi-step target with the estimated ratio remains a valid target for QπQ^\pi. Under this condition, Proposition 2 shows that the multi-step target can reduce the worst-case contribution of in-distribution critic errors compared with one-step TD, while the contribution of OOD critic errors does not decrease with depth. This suggests that multi-step critic learning can be more effective when combined with OOD action handling, which appears consistent with Table 2 (page 9): the multi-step target adds 1 point without OOD handling (53 to 54) and 4 points with POA and DRS (58 to 62).

Ablation on ll (Figure 6(f)). Here ll is the practical sub-trajectory length of FQT, i.e., the multi-step target is computed from a sub-trajectory (st,at,rt,st+1,…,rt+l−1,st+l)(s_t,a_t,r_t,s_{t+1},\ldots,r_{t+l-1},s_{t+l}) sampled from the dataset (Algorithm 1 on page 8). We vary ll with POA, DRS, and all other hyperparameters fixed, so that l=1l=1 corresponds to one-step critic learning (FQL+POA&DRS). Success rate (%), averaged over 5 singletasks and 8 seeds. "Final" is the average over the last three evaluation epochs (800K, 900K, and 1M gradient steps), as in Table 2, and "500K" is the evaluation at 500K gradient steps.

Task categoryEvaluationl=1l=1l=2l=2l=3l=3l=5l=5 (default)l=10l=10
humanoidmaze-large500K5 ±\pm 413 ±\pm 512 ±\pm 716 ±\pm 719 ±\pm 9
Final19 ±\pm 1022 ±\pm 1026 ±\pm 1028 ±\pm 1030 ±\pm 10
cube-double500K28 ±\pm 734 ±\pm 738 ±\pm 244 ±\pm 646 ±\pm 3
Final41 ±\pm 346 ±\pm 445 ±\pm 449 ±\pm 448 ±\pm 2

One-step critic learning (l=1l=1) tends to be the slowest and lowest-performing setting, and longer sub-trajectories tend to accelerate learning. This may support the contribution of the multi-step target itself.

AuthorsOpenReview ↗

Authors' response to reviewer RzPM (3/3)

R4. Empirical diagnostics of the estimated density ratio in the multi-step target

Regarding the effect of density-ratio estimation errors on the multi-step target, Propositions 1 and 2 in R3 show that QπQ^\pi remains a fixed point for any estimate, and that the contraction and the error bound hold under the condition

λmin⁡(1,ρ^)≤ρon supp(β),ρ^=Γw1−Γw,ρ=πβ.\lambda\min(1,\hat\rho)\le\rho\quad\text{on }\mathrm{supp}(\beta),\qquad\hat\rho=\frac{\Gamma_w}{1-\Gamma_w},\quad\rho=\frac{\pi}{\beta}.

Thus the ratio estimated by Γw\Gamma_w does not need to be exact, as long as the clipped trace coefficient does not exceed the true ratio. We note that this is a sufficient condition that we identified, and weaker conditions may exist.

As empirical diagnostics, we report how much multi-step information is retained with the estimated ratio (Appendix D.2) and newly introduce a comparison of multi-step critic targets (Table 5 in Appendix D.4).

Effective backup depth (Figure 5(a), Appendix D.2). The table reports the ratio-dependent trace weight Rk=∏i=1kmin⁡(1,ρ^t+i)R_k=\prod_{i=1}^{k}\min(1,\hat\rho_{t+i}) of FQT at backup depth kk, as the mean over the sampled sub-trajectories, averaged over 40 runs.

Task categoryStepR1R_1R2R_2R3R_3R4R_4
humanoidmaze-large100K0.2700.0870.0310.011
1M0.2100.0480.0110.003
cube-double100K0.3900.1680.0750.035
1M0.2230.0580.0170.005

We emphasize that these are mean values. As discussed in Appendix D.2, Retrace(λ\lambda) can adjust the backup depth adaptively for each sub-trajectory: the maximum of R4R_4 over the sampled sub-trajectories remains 0.2500.250 (humanoidmaze-large) and 0.4070.407 (cube-double) at 1M steps, which suggests that some sub-trajectories retain substantial multi-step weights while the others use shorter backups. Consistently, in the ablation on ll in R3, a longer sub-trajectory tends to accelerate learning, with a comparable or higher final performance.

Comparison of multi-step critic targets (Table 5, Appendix D.4). All variants use the same flow-based policy extraction of FQL, without POA or DRS, and differ in the critic target. PQL-FQL (Peng's Q(λ)Q(\lambda) [1]) and HQL-FQL (Harutyunyan's Q(λ)Q(\lambda) [2], c=λc=\lambda) do not use importance-sampling-based correction, whereas QT-FQL uses c=λmin⁡(1,ρ^)c=\lambda\min(1,\hat\rho) with ρ^\hat\rho estimated by Γw\Gamma_w.

Task categoryλ\lambdaPQL-FQLHQL-FQLQT-FQL
humanoidmaze-large0.16 ±\pm 25 ±\pm 23 ±\pm 2
0.39 ±\pm 35 ±\pm 24 ±\pm 2
0.512 ±\pm 44 ±\pm 24 ±\pm 2
0.715 ±\pm 34 ±\pm 15 ±\pm 2
0.916 ±\pm 31 ±\pm 14 ±\pm 2
cube-double0.123 ±\pm 227 ±\pm 327 ±\pm 4
0.39 ±\pm 127 ±\pm 227 ±\pm 4
0.53 ±\pm 127 ±\pm 229 ±\pm 4
0.74 ±\pm 123 ±\pm 330 ±\pm 3
0.94 ±\pm 120 ±\pm 230 ±\pm 3

HQL-FQL and QT-FQL differ in whether the estimated ratio is used, and QT-FQL remains relatively stable across λ\lambda whereas HQL-FQL tends to degrade at large λ\lambda, which suggests that the ratio estimated by Γw\Gamma_w may provide a useful correction. PQL-FQL tends to improve with larger λ\lambda on humanoidmaze-large, which may be related to the implicit behavior regularization of Peng's Q(λ)Q(\lambda) discussed for CPQL [3] (Appendix B), but its performance is lower for λ≥0.3\lambda\ge0.3 on cube-double. It may also be connected to Table 2 (page 9), where POA is particularly beneficial on humanoidmaze-large (from 5 to 13 for QT-FQL): a behavior-biased fixed point and explicit penalization of OOD actions might address a similar issue in this task category. Our method takes the latter approach while aiming to keep the fixed point QπQ^\pi, and with POA and DRS, FQT reaches 28 on humanoidmaze-large and 49 on cube-double (Table 2). Since CPQL is built on Gaussian policies while we focus on flow-based actors, these results may also be viewed as an indirect comparison with a Peng's Q(λ)Q(\lambda)-based target under an expressive flow policy.

[1] Jing Peng and Ronald J. Williams. "Incremental Multi-Step Q-Learning." Machine Learning Proceedings 1994.

[2] Anna Harutyunyan, et al. "Q(λ\lambda) with Off-Policy Corrections." ALT 2016.

[3] Byeongchan Kim and Min-hwan Oh. "Peng's Q(λ\lambda) for Conservative Value Estimation in Offline Reinforcement Learning." ICLR 2026.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions and Responses

Dear Area Chair and Reviewers,

We thank the reviewers and the Area Chair for their time and thoughtful consideration of our work, and for their constructive feedback. We would like to summarize our additional contributions and our responses to the reviewers’ concerns. Changes in the revised manuscript are highlighted in blue.

  1. Analysis of multi-step learning with estimated-ratios. We added Proposition 1 (Section 3.1, p. 5), which establishes fixed-point preservation and gives a sufficient contraction condition for Retrace(λ\lambda) with estimated ratios, and Proposition 2 (Section 3.1, p. 6), which separates the worst-case contributions of in-distribution and out-of-distribution critic errors as backup depth increases. Proofs are in Appendix C.2.

  2. Revised support-constraint analysis and OOD-identification guarantee. We revised Theorems 2 and 3 (Section 3.2, p. 7; proofs in Appendix C.3) without using a change of measure across unsupported actions, under explicit tabular assumptions. Lemma 4 (Appendix C.3, p. 25) supports the boundedness assumption for in-distribution critic values in Theorem 3. We also added Lemma 3 (Appendix C.1, p. 20), giving a sufficient estimator-error condition for identifying OOD actions.

  3. Controlled backup-length ablation. In Figure 6(f) (Appendix D.3, p. 35), we vary l∈l\in{1,2,3,5,101,2,3,5,10} while holding POA, DRS, and other hyperparameters fixed. On humanoidmaze-large and cube-double, longer backups generally accelerate learning and improve final performance over the one-step setting.

  4. Multi-step target comparisons and estimator diagnostics. Table 5 and Figure 8 (Appendix D.4, pp. 37–38) compare Peng’s Q(λ)Q(\lambda), Harutyunyan’s Q(λ)Q(\lambda), and QT-FQL using the same flow-policy extraction, without POA or DRS. Our responses discuss the estimated trace weights and OOD flagging rates in Figures 5(a) and 5(b) (Appendix D.2, p. 34). Our follow-up response to rEEg further discusses the role of density-ratio correction under behavior regularization.

  5. Joint hyperparameter sensitivity. Figure 7 (Appendix D.3.1, p. 36) adds pairwise 3×33\times3 grids over (λ,η,ζ)(\lambda,\eta,\zeta) on two representative tasks, examining their interactions.

  6. Broader OGBench comparisons. We provided results under the original FQL discount-factor settings in Table 2 (p. 9), with task-level results in Table 4 (Appendix D, p. 30). Our response to MVPp, R3, compares FQT with ten Gaussian-, diffusion-, and flow-policy baselines across 50 tasks.

  7. Additional D4RL evaluation. Our follow-up responses to RzPM (R5) and MVPp (R4) report QT-FQL, +POA, +DRS, and FQT on nine MuJoCo and two Adroit tasks, alongside published FQL, CQL, and CPQL baselines where applicable. Our four variants are trained for 1M gradient steps and evaluated at the final step over four seeds.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.