- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work presents Flow Q‑trace, an offline multi‑step critic RL algorithm leveraging a learned conditional density ratio for off‑policy correction and point‑wise out‑of‑distribution action mitigation combined with expressive flow‑based policies.
Critical strengths
- This paper extends multi-step critic learning to offline DRL.
- The effectiveness of the proposed method is verified through experiments.
Critical weaknesses
- Motivation is unclear: For offline DRL, many methods already exist to address the Q-learning evaluation problem, so why must multi-step critic learning be adopted to solve Q-value evaluation? In other words, what obvious shortcomings do other types of methods have compared with multi-step critic learning? Meanwhile, the online DRL community has also developed a large number of methods to address inaccurate Q-learning evaluation. Compared with the authors' multi-step critic learning, what obvious shortcomings do these methods have when directly integrated into offline DRL?
- The experiments are insufficient: there are too few baselines, and the method is not validated on D4RL, a widely used benchmark, nor on low-quality offline datasets.
Questions for authors
- For offline DRL, many methods already exist to address the Q-learning evaluation problem, so why must multi-step critic learning be adopted to solve Q-value evaluation? In other words, what obvious shortcomings do other types of methods have compared with multi-step critic learning?
- The online DRL community has also developed a large number of methods to address inaccurate Q-learning evaluation. Compared with the authors' multi-step critic learning, what obvious shortcomings do these methods have when directly integrated into offline DRL?
- The method is not validated on the widely used D4RL, so it is unclear how the proposed method performs on D4RL (MuJoCo, Adroit, etc.).
- Although this paper uses a fixed sub-trajectory length of , it does not conduct a complete ablation on different sub-trajectory lengths , so it is unclear how to select the optimal backtracking length, and the sensitivity of the hyperparameter is insufficiently studied.
- The algorithm's performance on suboptimal and mixed-quality datasets is not tested. When the dataset contains a large number of suboptimal trajectories, how does multi-step Retrace + density ratio estimation perform? Will severe drift occur?
- Four baselines are too few. It is recommended to add more existing SOTA methods in offline DRL that address inaccurate Q-learning evaluation, as well as several methods from online DRL that address inaccurate Q-learning evaluation and are directly integrated into the offline setting for comparison, bringing the total number of baselines to around 10.
- This paper does not plot the curves of Q-values over training iterations, nor does it provide direct metrics (such as the MSE error between predicted Q-values and true Q-values) to directly and quantitatively demonstrate the improvement in Q-value estimation accuracy.
8 comments
Authors' response to reviewer MVPp (2/3)
R2. Ablation on the sub-trajectory length
In the revised paper, we newly introduce an ablation on the sub-trajectory length . It is in Figure 6(f) of Appendix D.3.
Here is the practical sub-trajectory length of FQT, i.e., the multi-step target is computed from a sub-trajectory sampled from the dataset (Algorithm 1 on page 8). We vary with POA, DRS, and all other hyperparameters fixed, so that corresponds to one-step critic learning (FQL+POA&DRS). Success rate (%), averaged over 5 singletasks and 8 seeds. "Final" is the average over the last three evaluation epochs (800K, 900K, and 1M gradient steps), as in Table 2, and "500K" is the evaluation at 500K gradient steps.
| Task category | Evaluation | (default) | ||||
|---|---|---|---|---|---|---|
| humanoidmaze-large | 500K | 5 4 | 13 5 | 12 7 | 16 7 | 19 9 |
| Final | 19 10 | 22 10 | 26 10 | 28 10 | 30 10 | |
| cube-double | 500K | 28 7 | 34 7 | 38 2 | 44 6 | 46 3 |
| Final | 41 3 | 46 4 | 45 4 | 49 4 | 48 2 |
With POA and DRS fixed, a longer sub-trajectory tends to make learning faster. At 500K gradient steps, the success rate tends to increase with , from 5 () to 16 () and 19 () on humanoidmaze-large, and from 28 to 44 and 46 on cube-double. The final performance also tends to increase with , from 19 () to 28 () on humanoidmaze-large and from 41 to 49 on cube-double, and it tends to be comparable for .
These results appear consistent with Proposition 2 in R1: a longer sub-trajectory can reduce the worst-case contribution of in-distribution critic errors, and with OOD actions handled by POA and DRS, this tends to lead to faster learning and higher performance.
Authors' response to reviewer MVPp (3/3)
R3. Additional baselines and dataset quality
Regarding the number of baselines, we also report the results of FQT under the discount-factor setting of the original FQL paper [1] (Table 2 on page 9). In this setting, we use the same 50 OGBench tasks, the same discount factors, and the same evaluation protocol as [1] (the average over the last three evaluation epochs, 8 seeds), so our results can be compared with the offline RL baselines reported in [1].
The comparison is summarized below. The baseline results are taken from Table 2 of [1], except FQL, for which we report our reproduction (Table 2 on page 9). The baselines include Gaussian-policy methods (BC, IQL [2], ReBRAC [3]), diffusion-policy methods (IDQL [4], SRPO [5], CAC [6]), and flow-policy methods (FAWAC, FBRAC, and IFQL, which are flow-based baselines from [1], and FQL [1]). Success rate (%), averaged over 5 singletasks of each category.
| Task category | BC | IQL | ReBRAC | IDQL | SRPO | CAC | FAWAC | FBRAC | IFQL | FQL | FQT |
|---|---|---|---|---|---|---|---|---|---|---|---|
| antmaze-large | 11 | 53 | 81 | 21 | 11 | 33 | 6 | 60 | 28 | 78 2 | 81 2 |
| antmaze-giant | 0 | 4 | 26 | 0 | 0 | 0 | 0 | 4 | 3 | 10 5 | 23 3 |
| humanoidmaze-medium | 2 | 33 | 22 | 1 | 1 | 53 | 19 | 38 | 60 | 54 6 | 54 15 |
| humanoidmaze-large | 1 | 2 | 2 | 1 | 0 | 0 | 0 | 2 | 11 | 4 1 | 28 10 |
| antsoccer-arena | 1 | 8 | 0 | 12 | 1 | 2 | 12 | 16 | 33 | 60 2 | 58 3 |
| cube-single | 5 | 83 | 91 | 95 | 80 | 85 | 81 | 79 | 79 | 96 1 | 98 1 |
| cube-double | 2 | 7 | 12 | 15 | 2 | 6 | 5 | 15 | 14 | 30 3 | 49 4 |
| scene | 5 | 28 | 41 | 46 | 20 | 40 | 30 | 45 | 30 | 58 3 | 59 1 |
| puzzle-3x3 | 2 | 9 | 21 | 10 | 18 | 19 | 6 | 14 | 19 | 29 2 | 100 0 |
| puzzle-4x4 | 0 | 7 | 14 | 29 | 10 | 15 | 1 | 13 | 25 | 19 1 | 38 3 |
| Average | 3 | 23 | 31 | 23 | 14 | 25 | 16 | 29 | 30 | 44 | 59 |
The "Average" row is the mean over the 10 task categories. Since we follow the evaluation protocol of [1], other methods evaluated under the same protocol can also be compared with our results.
Regarding suboptimal datasets, in OGBench the five singletasks of each task category share the same dataset, with the rewards relabeled for each singletask (Appendix E.1). The dataset consists of trajectories that are not collected for a specific singletask, so for a given singletask, most of the dataset may be unrelated to the task or suboptimal for it.
[1] Seohong Park, Qiyang Li, and Sergey Levine. "Flow Q-Learning." ICML 2025.
[2] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. "Offline Reinforcement Learning with Implicit Q-Learning." ICLR 2022.
[3] Denis Tarasov, et al. "Revisiting the Minimalist Approach to Offline Reinforcement Learning." NeurIPS 2023.
[4] Philippe Hansen-Estruch, et al. "IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies." arXiv 2023.
[5] Huayu Chen, et al. "Score Regularized Policy Optimization through Diffusion Behavior." ICLR 2024.
[6] Zihan Ding and Chi Jin. "Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learning." ICLR 2024.
Authors' response to reviewer MVPp (1/3)
We thank the reviewer for the constructive review. All changes and additions are marked in blue in the revised paper.
R1. Motivation for multi-step critic learning
Most offline RL methods focus on how to handle OOD actions and the resulting value overestimation, for example through behavior regularization or conservative value estimation, while keeping the one-step TD target. In addition to OOD actions, we also consider the critic errors on in-distribution state-action pairs, on which the one-step TD target relies through bootstrapping at every step. Multi-step critic learning can reduce the effect of these errors on the target by also using the rewards along the trajectory in the dataset, and we focus on Retrace(λ), a representative multi-step method with off-policy correction.
In the revised paper, we newly introduce two theoretical results that compare this multi-step target with one-step TD learning (Propositions 1 and 2). They are in Section 3.1 and their proofs are in Appendix C.2. These results suggest that, if we can learn an actor that does not sample OOD actions, our Retrace(λ) target with estimated ratios can be better than one-step TD learning.
Propositions 1 and 2 are briefly summarized below.
Proposition 1 (page 5). Let ρ^≥0 be an estimate of ρ=π/β on supp(β), and let R^ be the Retrace(λ) operator (eq. (9) on page 4) with the estimated trace coefficient c^=λmin(1,ρ^).
Proposition 2 (page 6). Under the condition of Proposition 1(ii), let Rℓ^ be the depth-ℓ Retrace(λ) target with c^, where ℓ=1 is the one-step Bellman operator, and let εID and εOOD bound the critic error ∣Q−Qπ∣ on in-distribution and OOD pairs, respectively. Then for any (s,a) and ℓ≥1,
∣Rℓ^Q(s,a)−Qπ(s,a)∣≤w^ID,ℓ(s,a)εID+w^OOD,ℓ(s,a)εOOD,
where the ID weight w^ID,ℓ is nonincreasing in the depth ℓ and the OOD weight w^OOD,ℓ is nondecreasing.
In theory, the one-step TD target also has Qπ as its fixed point. In practice, however, the critic used for bootstrapping has function approximation errors, so the target computed from it is also inaccurate. Proposition 2 bounds how much these critic errors can affect the target. It shows that the multi-step target can reduce the worst-case contribution of in-distribution critic errors compared with one-step TD (ℓ=1), while the contribution of OOD critic errors does not decrease with depth. We note that this is a statement on the worst-case bound, and it does not mean that the multi-step target is always better than one-step TD. If the learned policy generates no OOD actions, the OOD weight vanishes, and the smaller ID weight gives a tighter worst-case bound than one-step TD. It further suggests that multi-step critic learning can be more effective when combined with OOD action handling, which appears consistent with Table 2 (page 9): the multi-step target adds 1 point without OOD handling (53 to 54) and 4 points with POA and DRS (58 to 62).