acceptodds
Under review as a conference paper at ICLR 2027

Flow Matching Loss for Offline Flow Policy Reinforcement Learning

Abstract

Flow models have demonstrated substantial promise for robotics control tasks. Usually, they are trained using behavior cloning, and a key challenge in training these policies is adapting traditional offline RL algorithms to flow policies. In particular, modern offline RL algorithms include a deterministic Q-gradient, which requires backpropagating through the iterative decoding process. This step undermines one of the key advantages of flow policies, which is that every decoding step is trained using a strong supervised signal derived directly from the final output. We propose the following solution: rather than target the action in the offline dataset, we first use Q-guided inference-time policy optimization to identify a better action, and then directly train the flow policy to target that action using the standard flow matching objective. We implement this idea in the context of Reversal Q-learning (RQL), a state-of-the-art offline RL algorithm for flow policies, to obtain Q-gradient RQL (QGRQL). Across multiple tasks from the OGBench locomotion and manipulation environments, we show that QGRQL outperforms existing state-of-the-art methods. Our results demonstrate the promise of better aligning offline RL algorithms with flow matching objectives.

Primary area:
reinforcement learning
Keywords:
flow policies, offline reinforcement learning, q learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_xf65OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Q-Gradient Reversal Q-Learning (QGRQL), an offline RL method that modifies the policy update in Reversal Q-Learning. Starting from dataset actions, the method performs projected gradient ascent using the learned value function and uses the resulting actions as flow-matching targets. The actor objective also retains a value-maximization term, while value learning uses reverse-flow trajectories associated with the original dataset actions. The motivation is that the value function can identify better actions before the policy learns to generate them. Experiments on 50 OGBench tasks report an average success rate of 67.3%, compared with 64.1% for RQL.

Critical strengths

The idea of improving the actions used as flow-matching targets is intuitive and addresses a relevant limitation of behavior-based supervision in offline RL. The diagnostic experiment showing that value-guided action optimization can improve the current policy provides some motivation for this direction.

Critical weaknesses

The experimental comparisons are too limited to establish the method’s position relative to existing approaches. Only RQL and floq are included, while relevant methods such as FQL and an offline instantiation of DSRL are missing. Although floq builds on FQL, its inclusion does not replace a direct comparison with FQL under the same experimental conditions. Moreover, Table 1 states that floq is trained for fewer updates than in its original evaluation and that configurations are transferred to tasks with different reward settings. This makes its poor performance on several environments difficult to interpret. Covering 50 tasks provides useful breadth, but does not compensate for the narrow baseline selection. The overall improvement over RQL is relatively modest, and the use of best-checkpoint results over three seeds, together with single-seed ablations, leaves uncertainty about how consistently this improvement holds. The additional cost of ten action-optimization steps per update is also not reported.

I find the motivation for the general idea clearer than the justification for the actual algorithm. Improving an action and then fitting a flow policy to it is straightforward, but QGRQL combines projected action optimization, an expanded-state value function, separate trajectories for actor and critic training, and an additional value-maximization term in the actor objective. The paper does not sufficiently explain which of these choices are necessary or what each contributes. In particular, Eq. (12) changes both the flow-matching targets and the intermediate states used for value maximization, so the comparison with RQL does not isolate the proposed target-improvement mechanism. The ablations on projection radius and optimization start time do not resolve this issue. A simpler method that only fits optimized actions would be an important comparison. Furthermore, Figure 1 studies optimization of policy-generated actions, whereas the proposed algorithm optimizes dataset actions. This provides suggestive motivation, but does not directly establish the reliability of the targets used during training. As presented, the method feels like a plausible collection of modifications whose necessity and interaction remain insufficiently understood.

Questions for authors

(a) Could you provide comparisons with FQL and an offline implementation of DSRL using matched datasets, reward definitions, action chunking, and training budgets? These comparisons would help determine whether the gains extend beyond the particular RQL and floq configurations evaluated here.

(b) How does QGRQL perform when the explicit value-maximization term in Eq. (12) is removed and the policy is trained solely by flow matching on optimized actions? Conversely, what happens when the dataset actions are left unchanged but the actor uses the same intermediate-state construction as QGRQL? These controls would help identify the source of the improvement.

(c) Which flow timestep ff is used when evaluating ∇xV(s,x,f)\nabla_x V(s,x,f) in Eq. (10)? Since VV evaluates intermediate flow states, please clarify why its gradient at the chosen timestep is appropriate for optimizing a completed dataset action.

(d) What is the training-time overhead relative to RQL, and does the improvement persist under an equal wall-clock budget? It would also help to report final-checkpoint performance and repeat the central ablations across multiple seeds, given that the current results use peak performance and single-seed ablations.

3 comments
AuthorsOpenReview ↗

Rebuttal (1/2)

Thank you for your feedback and the detailed review!

Updated Results

We opted to report the best performance possible in our original submission since it is unclear to us which performance metric RQL reports (it is unspecified in their paper), and we chose the metric that would report the highest RQL performance to better understand our improvement over RQL. Based on reviewer feedback, we have updated the results in the paper to instead report the mean success rate on the last 3 checkpoints to standardize our reporting with earlier work like FQL. Our results exhibit the same improvement over RQL, with a +3.3 point difference when averaging success across all environments.

We have also added Q-Flow, QAM, and FQL as baselines, and we show that mean QGRQL performance across all environments is better than all of them.

Additional Ablations

We have updated our paper with multiple ablations to isolate the impact of flow supervision, regularization, and selecting optimal actions using the value function, and compared them to QGRQL and RQL (Appendix C, Table 6). Due to the short rebuttal time period, we report our results on a single seed across all the non-100M environments. We compare against the following ablations:

  1. Best-of-N samples N=32N=32 actions from the trained RQL policy at inference time and selects the highest-value action, as determined by the RQL value function.
  2. Policy Optimization directly seeks to utilize an optimized policy action target instead of an optimized dataset action target in QGRQL’s training loss. At each iteration, this method replaces adataa_\textrm{data} with an action sampled from the flow policy (denoted as apolicya_\textrm{policy}) in Equation 10. It then uses apolicya_\textrm{policy} to compute the optimized policy action aPOLICY-OPEXa_\textrm{POLICY-OPEX}, and recomputes xOPEXfx^f_\textrm{OPEX} using aPOLICY-OPEXa_\textrm{POLICY-OPEX} in Equation 11. It then continues executing QGRQL with the modified action.
  3. No Regularization: sets R=∞R=\infty in Equation 10 while executing QG-RQL, removing any regularization with respect to the original dataset action.
  4. No Value Maximization: removes the first term from Equation 12 and only performs flow-matching with respect to the optimized dataset action aOPEXa_\textrm{OPEX}.

Our results show that QGRQL’s mean performance is better than all of the above ablations.

  1. We observe severe degradation in the policy optimization setting (-57.7 percentage points on average, with several environments at 0% success), and attribute this to the low quality of the actions sampled from the policy during training, especially during early iterations.
  2. We observe an average decrease of 5.2 percentage points compared to QGRQL (and 3.6 points compared to RQL) across environments in the Best-of-N setting, suggesting that inference time usage of the value function to select the best action provides limited benefits compared to utilizing the value function for action optimization during training.
  3. We observe sizable degradation across environments compared to QGRQL in the no regularization setting (-12.3 percentage points on average). We believe regularization is essential during Q-gradient optimization, as unregularized gradient ascent may deviate significantly from the dataset action and its associated rewards.
  4. We observe severe degradation in the no value maximization setting (-52.5 percentage points on average) when we simply perform flow matching with the optimized dataset action. Our experiments suggest that value maximization with the optimized dataset action is the main reason why we observe performance improvements in QGRQL.

We are running additional experiments to identify the impact of leaving the dataset actions unchanged while performing value maximization. In the subset of environments we have evaluated so far, we do not observe any sizable decreases in performance compared to QGRQL.

We also hope that the Policy Optimization ablation addresses your concern about our motivation - while we observe the value-function/policy lag in Figure 1 on optimizing policy actions, we chose to use the value function to optimize dataset actions during training because we believed that regularizing with respect to the dataset action provides a reasonable “floor” during early training that prevented flow matching to extraordinarily poor actions. On the other hand, when we train using the Q-gradient optimized policy actions as a target, we expect actions sampled from the policy during the initial training iterations to be of significantly lower quality. Even with a strong value function, optimizing these sampled policy actions produces poor action targets, and as our results show, severely destabilizes training to the point that the final learned policy does not achieve nonzero success for multiple environments.

AuthorsOpenReview ↗

Rebuttal (2/2)

Wallclock Time

In our experiments, we observe high variance across environments on wallclock time, which can be attributed to a number of factors such as machine state and code optimizations. Even with perfect isolation, any differences in performance with RQL would be due to gradient ascent in calculating Q-gradient optimized actions and recalculating xOPEXfx^f_{\textrm{OPEX}} in τ~′\widetilde \tau'. We believe these runtimes would be sensitive to the environment implementation and underlying compute used, optimizing which is beyond the scope of our work. We believe that it is non-standard to discuss wallclock time when measuring improvements in algorithmic performance, and runtime is not reported in the original papers for several of our baselines (e.g. Q-Flow, RQL, QAM). We instead fix the number of gradient steps at 1M across all our baselines for our final reported results, and provide performance vs gradient step plots in Figure 3, to provide an accurate comparison of different methods under different compute budgets.

Training floq for 1M steps

While we acknowledge that the original floq paper reported results at 2M steps, we fix the total number of steps at 1M for maintaining consistency in number of training steps across all our baselines, several of which train for 1M steps.

Equation 10

We set f=Ff=F during the action optimization process, and have updated the manuscript to reflect this change.

Thank you for the response and the additional experiments. I appreciate the new comparisons with FQL, QAM, and Q-Flow, as well as the change from best-checkpoint performance to the mean of the last three checkpoints. These additions address important parts of my original concerns. However, my central concerns about the attribution of the improvement, the computational cost, and the presentation remain.

1. The contribution of the proposed action optimization remains insufficiently isolated.

The “No Value Maximization” ablation directly addresses one of my questions and provides useful evidence that flow matching on optimized actions alone is insufficient in the current setup. However, RQL already includes an explicit value-maximization term. The substantial degradation after removing this term therefore does not establish the incremental benefit of QGRQL’s optimized flow-matching targets over RQL.

The preliminary result that leaving the dataset actions unchanged causes no sizable performance decrease makes the remaining control particularly important. This setting needs to be precisely defined: are only the flow-matching targets restored to the original dataset actions, while QGRQL’s intermediate-state construction is retained? Are optimized actions still used in constructing those states? A completed comparison against both RQL and full QGRQL, with other settings controlled and results across multiple seeds, is needed to distinguish the benefit of changing the supervision targets from that of changing the states used for value maximization. The current results do not yet justify a strong conclusion about which proposed modification accounts for the improvement.

2. The additional computation and the robustness of the improvement remain unclear.

Comparing methods at a fixed number of outer updates is informative, but it does not answer the question about computational overhead or performance under a matched time budget. QGRQL introduces additional action-gradient optimization steps within each update, so the same number of outer updates does not imply the same computational cost. Hardware and implementation variability can be controlled through measurements on matched hardware and configurations. Describing wall-clock evaluation as non-standard does not address this concern.

At minimum, reporting the complete update time for QGRQL and RQL, including the internal action-optimization steps, would allow readers to assess the cost associated with the reported improvement. Representative comparisons under a common time budget would further strengthen the evidence.

The revised checkpoint metric is a meaningful improvement, but it does not resolve the uncertainty in the central ablations, which still use a single seed. This is particularly relevant when interpreting the reported average gain of 3.3 percentage points and the preliminary observation of little difference in the unchanged-action control.

3. The writing and presentation require substantial revision.

I also remain concerned about the clarity and organization of the manuscript. The exposition needs to distinguish more clearly between components inherited from RQL and the proposed modifications, explain how the actor and critic trajectory constructions relate to the training objectives, and make the purpose of each modification explicit. The interpretation of the ablations should also distinguish evidence that a component is important within the full method from evidence that it explains the improvement over the baseline.

In my assessment, the current writing and presentation do not yet meet the standard of clarity and precision expected at a leading AI conference such as ICLR. The algorithmic explanation and the connection between the experimental evidence and the contribution claims need substantial revision.

For these reasons, I maintain my recommendation to reject the paper in its current form. I encourage the authors to revise the exposition substantially, complete the central controlled comparisons, and quantify the additional computational cost before resubmission.

Reviewer_PSkTOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes QGRQL, an offline RL method for training flow policies. It builds on Reversal Q-Learning (RQL).

The method changes only RQL's policy update. For each dataset action, it takes 10 steps of gradient ascent on the learned value function. Each step is projected onto a small L2 ball around the original action. The policy is then trained with RQL's policy loss, but its flow-matching term targets this improved action instead of the dataset action. Value learning is the same as in RQL.

The authors' motivation is that early in training, the value function can already find better actions than the policy.

They evaluate on 50 OGBench tasks across 10 environments (antmaze, humanoidmaze, cube, puzzle, scene). They compare against two baselines, RQL and floq. QGRQL gets an average success rate of 67.3%, vs. 64.1% for RQL and 37.9% for floq. The gains are mostly on manipulation tasks (scene, puzzle-4x4, cube-double, cube-quadruple). Results are reported as the best checkpoint over 1M gradient steps, averaged over 3 seeds. Ablations on the ball radius R and on when the action optimization starts use a single seed.

Critical strengths
  1. The idea is very simple. It adds one step to RQL: improve the dataset action with a few gradient steps on the value function, then use it as the flow-matching target. It is easy to implement and easy to reuse in other flow-policy methods.
  2. The gains are good on hard manipulation tasks. QGRQL improves over RQL on puzzle-4x4-100M (from 66.4 to 74.5), cube-double (from 37.1 to 50.5), scene-sparse (from 94.3 to 99.6), and cube-quadruple-100M (from 58.8 to 62.7). These are long-horizon tasks where existing methods struggle.
Critical weaknesses
  1. The evaluation protocol is unreliable. This is the main reason for my recommendation. This paper reports the best checkpoint with 3 seeds. Its baselines do not report results this way. floq trains for 2M steps and reports the average of its last three evaluations (1.96M, 1.98M and 2M) over 3 seeds. Its paper does not state this, but its released runs show it. From my reading of RQL, it is not clear what protocol it uses. FQL, which floq builds on, follows the OGBench protocol: it averages the 800K, 900K and 1M checkpoints over 8 seeds and argues against reporting the best checkpoint. I find best-checkpoint reporting unreliable. RL results are noisy across checkpoints and seeds, so picking the best one inflates the numbers. That is why averaging the last few checkpoints over many seeds matters. The inflation shows up here: the RQL paper reports an average of 56 on these 50 tasks, while this paper reports 64.1 for RQL. The gain over RQL is only 3.2 points, and with this protocol I cannot tell if it is real or noise.

  2. The contribution is incremental and the baselines are weak. The method is a small change to RQL. Using value gradients to improve actions and then training the policy toward them is close to PA-RL and EXPO. The paper compares against only RQL and floq. RQL is a recent arXiv paper that has not been peer reviewed. floq was not run under its own setup: this paper trains it for 1M steps instead of its 2M, scores it by best checkpoint instead of its last-three average, and runs it on sparse-reward tasks it was not tuned for. Established baselines like QAM and ReBRAC are missing. On locomotion the gains are flat: antmaze-large 93.1 vs 94.0, antmaze-giant 39.6 vs 43.2, humanoidmaze-medium 97.6 vs 95.7, humanoidmaze-large 46.4 vs 46.7.

Questions for authors

Note to authors that I will start from rejection because I am very confused about the eval protocols. I am open to changing my score if the authors convince me that their eval protocol is unbiased, and do make sense such that they can be compared against prior work.

I would like to see the following things:

  1. Could you please do some sample efficiency comparison between the methods? I think this is very important to try because the paper mentions that the value function learns faster than the policy. This could mean that the method is sample efficient.
  2. Could the authors also share some discussion on wall clock times of the suggested method?
  3. IMPORTANTLY: could you explicitly clarify the eval protocols used in the original floq, and RQL, and the one you implemented in this work? What is the rationale of reporting the best checkpoint number? I would argue this clearly inflates your numbers. From my reading, floq reports the mean sr on terminal last 3 checkpoints. But this paper reports best number. This is highly highly unreliable comparison. If the authors succeed in fully convincing me and defending their design choice here, i remain open to changing my score.
  4. It would be helpful if the authors also provide link to their anonymous source code. I wanted to check some experimental details directly from the codebase but could not do it because the paper provides none.
4 comments
AuthorsOpenReview ↗

Rebuttal (1/2)

Thank you for your feedback, and for recognizing that our contributions improve RQL’s performance on harder tasks!

Updated Results

We opted to report the best performance possible in our original submission since it is unclear to us which performance metric RQL reports (it is unspecified in their paper), and we chose the metric that would report the highest RQL performance to better understand our improvement over RQL. Based on reviewer feedback, we have updated the results in the paper to instead report the mean success rate on the last 3 checkpoints to standardize our reporting with earlier work like FQL. Our results exhibit the same improvement over RQL, with a +3.3 point difference when averaging success across all environments.

We have also added Q-Flow, QAM, and FQL as baselines, and we show that mean QGRQL performance across all environments is better than all of them. Additionally, we have linked our anonymized source code in the updated paper (line 53).

Strength of Our Contribution

We respectfully disagree that our contribution is incremental with respect to the linked references. We cite both EXPO [1] and PA-RL [2] in our paper, and also clarify how we differ in lines 256-263. Value-gradient based policy action optimization is well known in previous literature (even being described in [3] as OPEX). However, EXPO and PA-RL are offline-to-online learning algorithms which use value function gradients to optimize actions sampled from a learned policy, similar to OPEX in [3]. In our work, we do not use the value function gradients to optimize policy actions, but instead use the gradients to optimize dataset actions in an offline setting. We believe that we are the first to show that value functions can effectively optimize dataset actions during offline RL training to provide a better learning target than the original offline dataset itself.

While our approach builds on the ideas in PA-RL, there are some key differences arising from the fact that we are in the purely offline setting and working with flow policies. First, optimizing policy actions as in PA-RL works poorly in this setting since policies learn slowly. Our approach optimizes dataset actions to address this issue. Second, we leverage projected gradient ascent to avoid the distribution shift challenge in offline RL. Third, we integrate our approach with RQL, which incorporates the flow computation into the MDP to improve performance.

Additional Ablations

We have updated our paper with multiple ablations to isolate the impact of flow supervision, regularization, and selecting optimal actions using the value function, and compared them to QGRQL and RQL (Appendix C, Table 6). Due to the short rebuttal time period, we report our results on a single seed across all the non-100M environments. We compare against the following ablations:

  1. Best-of-N samples N=32N=32 actions from the trained RQL policy at inference time and selects the highest-value action, as determined by the RQL value function.
  2. Policy Optimization directly seeks to utilize an optimized policy action target (similar to optimized actions in EXPO and PA-RL) instead of an optimized dataset action target in QGRQL’s training loss.
  3. No Regularization: sets R=∞R=\infty in Equation 10 while executing QG-RQL, removing any regularization with respect to the original dataset action.
  4. No Value Maximization: removes the first term from Equation 12 and only performs flow-matching with respect to the optimized dataset action aOPEXa_\textrm{OPEX}.

Our results show that QGRQL’s mean performance is better than all of the above ablations.

  1. We observe severe degradation in the policy optimization setting (-57.7 percentage points on average, with several environments at 0% success), and attribute this to the low quality of the actions sampled from the policy during training, especially during early iterations.
  2. We observe an average decrease of 5.2 percentage points compared to QGRQL (and 3.6 points compared to RQL) across environments in the Best-of-N setting, suggesting that inference time usage of the value function to select the best action provides limited benefits compared to utilizing the value function for action optimization during training.
  3. We observe sizable degradation across environments compared to QGRQL in the no regularization setting (-12.3 percentage points on average). We believe regularization is essential during Q-gradient optimization, as unregularized gradient ascent may deviate significantly from the dataset action and its associated rewards.
  4. We observe severe degradation in the no value maximization setting (-52.5 percentage points on average) when we simply perform flow matching with the optimized dataset action. Our experiments suggest that value maximization with the optimized dataset action is the main reason why we observe performance improvements in QGRQL.
AuthorsOpenReview ↗

Rebuttal (2/2)

We are currently running additional experiments with respect to sample efficiency which we hope to complete during the rebuttal period.

Wallclock Time

In our experiments, we observe high variance across environments on wallclock time, which can be attributed to a number of factors such as machine state and code optimizations. Even with perfect isolation, any differences in performance with RQL would be due to gradient ascent in calculating Q-gradient optimized actions and recalculating xOPEXfx^f_{\textrm{OPEX}} in τ~′\widetilde \tau'. We believe these runtimes would be sensitive to the environment implementation and underlying compute used, optimizing which is beyond the scope of our work. We believe that it is non-standard to discuss wallclock time when measuring improvements in algorithmic performance, and runtime is not reported in the original papers for several of our baselines (e.g. Q-Flow, RQL, QAM). We instead fix the number of gradient steps at 1M across all our baselines for our final reported results, and provide performance vs gradient step plots in Figure 3, to provide an accurate comparison of different methods under different compute budgets.

References

  1. Dong et al. (2026). Expo: Stable reinforcement learning with expressive policies.
  2. Mark et al. (2025). Policy-agnostic RL: Offline RL and online RL fine-tuning of any class and backbone.
  3. Park et al. (2024). Is value learning really the main bottleneck in offline rl?
Reviewer_PSkTOpenReview ↗

reviewer Response

Thank you for the additional experiments. My concerns about the checkpoint evaluation protocol have been partially addressed. I am therefore updating my score from 1 to 2. However, the ablations are still based on a single seed. For an RL paper, I think it is important to report results across multiple seeds. I understand the limited rebuttal timeline, but I do not think it justifies reporting single-seed result.

Thank you for your feedback - we acknowledge your concern, and we shall do our best to add multiple seed results within the remaining rebuttal period. We wanted to point out that we have updated the paper with a single-seed result from our sample efficiency experiment. We evaluate performance after training on subsampled datasets with 300k transitions and 600k transitions for our non-100M environments, and show that QGRQL improves over RQL on average by 6.2 points for 300k transitions and by 6.1 points for 600k transitions. Please refer to Table 7 in Appendix C for the full results table for this experiment.

Reviewer_RXofOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an offline RL method built on Reversal Q-Learning (RQL), called Q-Gradient Reversal Q-Learning (QGRQL), which addresses the mismatch between flow-matching supervision and Q-gradient-based policy extraction (direct value maximization) by replacing the dataset action in the flow-matching loss with a target action obtained through projected gradient ascent on the learned value function within an L2L_2 ball around the dataset action. The method is evaluated on OGBench locomotion and manipulation tasks, where it shows improvements over RQL and floq on average.

Critical strengths

The core idea of adapting policy extraction to the flow-matching supervision signal is conceptually well motivated. Constructing the target action through Q-gradient ascent and then training the policy to match it with a flow-matching objective can, in principle, lead to more effective policy optimization than direct value maximization. This is a sound and useful direction in the offline setting, where bootstrapped value estimates can destabilize policy optimization and direct Q-value maximization is prone to exploiting value overestimation on out-of-distribution actions.

Critical weaknesses
  1. Policy optimization objective with aOPEXa_{\text{OPEX}}: My main concern with this paper is the policy optimization objective itself. The paper motivates QGRQL by claiming flow-matching supervision is preferable to existing policy extraction methods in an offline setting, yet the proposed loss (Eq.12) retains the Q-gradient term, and no experiment in the main text isolates the flow-matching term with aOPEXa_{\text{OPEX}}. As a result, it is unclear whether the benefit of QGRQL comes from the direct value maximization at xOPEXfx_{\text{OPEX}}^f (the first term in Eq.12) or from the flow supervision toward aOPEXa_{\text{OPEX}}.

  2. Hyperparameter choices: This follows from the first weakness. Eq.12 includes the hyperparameter α\alpha, but the value used in the experiments is not reported anywhere in the paper. Moreover, the method has two main hyperparameters, (1) RR for aOPEXa_{\text{OPEX}} construction, and (2) α\alpha in Eq.12, which potentially makes it more sensitive to tuning.

  3. Limited empirical gains and baselines: The empirical evaluation includes only two baselines, RQL and floq, and the suitability of floq as a baseline is questionable. Floq’s primary contribution, increasing the expressivity of the value function, is orthogonal to the central theme of this paper, namely improving policy optimization for flow policies. A more informative evaluation would compare QGRQL against flow-based methods that primarily address policy extraction. Furthermore, the improvement of QGRQL over its direct counterpart, RQL, is marginal: the average success rate increases by only about 3 percentage points across the evaluated benchmark.

Questions for authors

see Weaknesses please

4 comments
AuthorsOpenReview ↗

Rebuttal (1/1)

Thank you for your feedback, and for recognizing that our contributions are well-motivated!

Ablating Contribution of Direct Value Maximization

We have updated our paper with multiple ablations to isolate the impact of flow supervision, regularization, and selecting optimal actions using the value function, and compared them to QGRQL and RQL (Appendix C, Table 6). Due to the short rebuttal time period, we report our results on a single seed across all the non-100M environments. We compare against the following ablations:

  1. No Value Maximization: removes the first term from Equation 12 and only performs flow-matching with respect to the optimized dataset action aOPEXa_\textrm{OPEX}.
  2. No Regularization: sets R=∞R=\infty in Equation 10 while executing QGRQL, removing any regularization with respect to the original dataset action.
  3. Policy Optimization directly seeks to utilize an optimized policy action target instead of an optimized dataset action target in QGRQL’s training loss.
  4. Best-of-N samples N=32N=32 actions from the trained RQL policy at inference time and selects the highest-value action, as determined by the RQL value function.

Our results show that QGRQL’s mean performance is better than all of the above ablations.

  1. We observe severe degradation in the no value maximization setting (-52.5 percentage points on average) when we simply perform flow matching with the optimized dataset action. Our experiments suggest that value maximization with the optimized dataset action is the main reason why we observe performance improvements in QGRQL.
  2. We observe sizable degradation across environments compared to QGRQL in the no regularization setting (-12.3 percentage points on average). Despite needing an additional hyperparameter, we believe regularization is essential during Q-gradient optimization, as unregularized gradient ascent may deviate significantly from the dataset action and its associated rewards.
  3. We observe severe degradation in the policy optimization setting (-57.7 percentage points on average, with several environments at 0% success), and attribute this to the low quality of the actions sampled from the policy during training, especially during early iterations.
  4. We observe an average decrease of 5.2 percentage points compared to QGRQL (and 3.6 points compared to RQL) across environments in the Best-of-N setting, suggesting that inference time usage of the value function to select the best action provides limited benefits compared to utilizing the value function for action optimization during training.

Hyperparameter Choices

We have reported our hyperparameter choices for RR in Appendix B.2, and for α\alpha, in Table 4 in Appendix B.1 in the updated submission. We would also like to highlight that α\alpha is a pre-existing hyperparameter in RQL [1, Equation 15] and not introduced by our method. We use the same values for α\alpha in our environments as RQL does.

While we acknowledge that performance may be impacted by poor choices for RR, we believe that introducing a regularization hyperparameter is a standard choice in machine learning algorithms and does not necessarily mean that our method is likely to be extremely sensitive to tuning. Additionally, our ablation in Section 5.3 demonstrates some performance sensitivity to RR, but sizable deviations in performance are observed only for large values of RR (e.g. R=0.4R=0.4), and R=∞R=\infty (new ablation reported in Appendix C as No Regularization).

Limited Empirical Gains and Baselines

We reported floq results since it consistently outperformed FQL in its reported paper results, and we wished to demonstrate that simple action target optimization can achieve better performance than expending compute on the Q-function as done in floq.

We have added results from Q-Flow, QAM, and FQL to our paper, and show that QGRQL outperforms all of the above algorithms in terms of mean performance. We have also updated the reporting method to report the mean success rate on the terminal 3 checkpoints similar to prior work. Our updated results for QGRQL retain the same mean performance improvement over RQL of 3.3 percentage points.

We respectfully disagree that the improvement of QGRQL over RQL is necessarily marginal: some environments where both methods score highly are saturated with over 90% success (e.g. humanoidmaze-medium and puzzle-3x3-sparse). On the other hand, on some manipulation environments and environments with large datasets, we outperform RQL by large margins (e.g. cube-double and puzzle-4x4-100M-sparse). While we have reported the average improvement in performance in the paper, we believe that the strength of the paper’s contributions lies in its ability to improve offline flow policy algorithm performance on harder environments (as pointed out by reviewer PSkT).

References

  1. Oberai et al. (2026). Reversal Q-Learning.

Thanks for the response and additional explanations, which indeed help in better understanding the proposed method. However, based on the rebuttal, I still have a major concern regarding the core principle of the paper. Given that the improved performance is mostly attributed to the value maximization on the value-ascended action aOPEXa_{\text{OPEX}}, the main motivation for utilizing a flow-matching-style supervised loss is not well justified. I therefore suspect that the original policy extraction used in RQL with a properly tuned behavioral regularization coefficient would be sufficient to match the performance of QGRQL, and its results are indeed within one or two standard deviations of QGRQL's reported average success rate. I would like the authors to further clarify this point. What justifies the use of supervised flow-matching to aOPEXa_{\text{OPEX}} in QGRQL? If so, why would policy optimization on value-ascended action be beneficial, given that it comes with a higher risk of value overexploitation?

Thank you for your comment. We would like to clarify that we attribute our performance improvement over RQL to value maximization using the intermediate state construction with the optimized dataset action (i.e. in Equation 12, we use xOPEXfx^f_\textrm{OPEX} instead of RQL's xfx^f. RQL's xfx^f computes the intermediate state with respect to the dataset action adataa_\textrm{data}, whereas we compute xOPEXfx^f_\textrm{OPEX} with respect to the optimized dataset action target aOPEXa_\textrm{OPEX}). We use the same behavioral regularization coefficient that RQL does, which they had tuned for each environment (as reported in RQL's Appendix C.2). We thus do not believe that QGRQL's performance would be recoverable simply by tuning the hyperparameter α\alpha. Consequently, given the performance gap, we do not believe that we are overexploiting the value function while performing value maximization. We would also like to point out that despite average performance being small, we do outperform RQL in harder manipulation environments by large margins when averaged across multiple seeds (e.g. by 15.9 points on average on cube-double\texttt{cube-double} and 9.8 points on average on scene-sparse\texttt{scene-sparse}, well over 2 standard deviations.

The purpose of using a flow matching loss in our offline learning algorithm is to perform behavior cloning to the dataset actions as regularization during policy training (similar to our baselines FQL and RQL). In our work, we believe that the value function can identify better actions during training, so QGRQL's approach here is to simply flow match to a better action when available, given that we believe that the Q-function learns to identify high-quality actions earlier in training (as discussed in Section 4.1). We do not believe that our "no value maximization" ablation necessarily shows that flow matching is unnecessary during training, given that the role of the flow matching term has mostly been to serve as a regularizer in previous work. Training a policy using flow matching alone (as in the no value maximization setting) is akin to behavior cloning from a mixed dataset; in this case, we expect that the flow-matching-only method severely underperforms value learning methods due to the presence of suboptimal trajectories in the training data. Our conclusion from the no value maximization ablation is simply that OPEX action optimization does not overcome this degradation in performance caused by flow matching to lower-quality trajectories. Our main paper results, on the other hand, show that action optimization helps performance when we train QGRQL, which incorporates value learning alongside flow-matching regularization.

Thank you for the response. I agree that the flow-matching loss has primarily served as a behavioral regularizer in prior flow-based offline RL methods. However, the paper's motivation suggests a rather different role for this objective, and I do not find the empirical evidence sufficient to justify that particular motivation. Furthermore, while I acknowledge the improvements on some manipulation tasks, I still find the overall performance gains insufficiently substantial, particularly given that QGRQL is a relatively modest extension of RQL.

More importantly, I believe the underlying idea that policy extraction may lag behind improvements in value-function quality, which the authors refer to as value-learning-policy-extraction lag, is potentially applicable to a broader class of flow-based offline RL methods, rather than being specific to RQL. Exploring the proposed objective from this broader perspective, with a more carefully aligned motivation and empirical evidence that isolates and substantiates its benefits, could make the contribution considerably stronger, better motivated, and more broadly applicable. Overall, while I find the underlying idea potentially interesting, I believe the current contribution remains somewhat weak at this stage. Therefore, I would keep my current score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.