- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes Q-Gradient Reversal Q-Learning (QGRQL), an offline RL method that modifies the policy update in Reversal Q-Learning. Starting from dataset actions, the method performs projected gradient ascent using the learned value function and uses the resulting actions as flow-matching targets. The actor objective also retains a value-maximization term, while value learning uses reverse-flow trajectories associated with the original dataset actions. The motivation is that the value function can identify better actions before the policy learns to generate them. Experiments on 50 OGBench tasks report an average success rate of 67.3%, compared with 64.1% for RQL.
Critical strengths
The idea of improving the actions used as flow-matching targets is intuitive and addresses a relevant limitation of behavior-based supervision in offline RL. The diagnostic experiment showing that value-guided action optimization can improve the current policy provides some motivation for this direction.
Critical weaknesses
The experimental comparisons are too limited to establish the method’s position relative to existing approaches. Only RQL and floq are included, while relevant methods such as FQL and an offline instantiation of DSRL are missing. Although floq builds on FQL, its inclusion does not replace a direct comparison with FQL under the same experimental conditions. Moreover, Table 1 states that floq is trained for fewer updates than in its original evaluation and that configurations are transferred to tasks with different reward settings. This makes its poor performance on several environments difficult to interpret. Covering 50 tasks provides useful breadth, but does not compensate for the narrow baseline selection. The overall improvement over RQL is relatively modest, and the use of best-checkpoint results over three seeds, together with single-seed ablations, leaves uncertainty about how consistently this improvement holds. The additional cost of ten action-optimization steps per update is also not reported.
I find the motivation for the general idea clearer than the justification for the actual algorithm. Improving an action and then fitting a flow policy to it is straightforward, but QGRQL combines projected action optimization, an expanded-state value function, separate trajectories for actor and critic training, and an additional value-maximization term in the actor objective. The paper does not sufficiently explain which of these choices are necessary or what each contributes. In particular, Eq. (12) changes both the flow-matching targets and the intermediate states used for value maximization, so the comparison with RQL does not isolate the proposed target-improvement mechanism. The ablations on projection radius and optimization start time do not resolve this issue. A simpler method that only fits optimized actions would be an important comparison. Furthermore, Figure 1 studies optimization of policy-generated actions, whereas the proposed algorithm optimizes dataset actions. This provides suggestive motivation, but does not directly establish the reliability of the targets used during training. As presented, the method feels like a plausible collection of modifications whose necessity and interaction remain insufficiently understood.
Questions for authors
(a) Could you provide comparisons with FQL and an offline implementation of DSRL using matched datasets, reward definitions, action chunking, and training budgets? These comparisons would help determine whether the gains extend beyond the particular RQL and floq configurations evaluated here.
(b) How does QGRQL perform when the explicit value-maximization term in Eq. (12) is removed and the policy is trained solely by flow matching on optimized actions? Conversely, what happens when the dataset actions are left unchanged but the actor uses the same intermediate-state construction as QGRQL? These controls would help identify the source of the improvement.
(c) Which flow timestep is used when evaluating in Eq. (10)? Since evaluates intermediate flow states, please clarify why its gradient at the chosen timestep is appropriate for optimizing a completed dataset action.
(d) What is the training-time overhead relative to RQL, and does the improvement persist under an equal wall-clock budget? It would also help to report final-checkpoint performance and repeat the central ablations across multiple seeds, given that the current results use peak performance and single-seed ablations.
3 comments
Rebuttal (2/2)
Wallclock Time
In our experiments, we observe high variance across environments on wallclock time, which can be attributed to a number of factors such as machine state and code optimizations. Even with perfect isolation, any differences in performance with RQL would be due to gradient ascent in calculating Q-gradient optimized actions and recalculating in . We believe these runtimes would be sensitive to the environment implementation and underlying compute used, optimizing which is beyond the scope of our work. We believe that it is non-standard to discuss wallclock time when measuring improvements in algorithmic performance, and runtime is not reported in the original papers for several of our baselines (e.g. Q-Flow, RQL, QAM). We instead fix the number of gradient steps at 1M across all our baselines for our final reported results, and provide performance vs gradient step plots in Figure 3, to provide an accurate comparison of different methods under different compute budgets.
Training floq for 1M steps
While we acknowledge that the original floq paper reported results at 2M steps, we fix the total number of steps at 1M for maintaining consistency in number of training steps across all our baselines, several of which train for 1M steps.
Equation 10
We set during the action optimization process, and have updated the manuscript to reflect this change.
Thank you for the response and the additional experiments. I appreciate the new comparisons with FQL, QAM, and Q-Flow, as well as the change from best-checkpoint performance to the mean of the last three checkpoints. These additions address important parts of my original concerns. However, my central concerns about the attribution of the improvement, the computational cost, and the presentation remain.
1. The contribution of the proposed action optimization remains insufficiently isolated.
The “No Value Maximization” ablation directly addresses one of my questions and provides useful evidence that flow matching on optimized actions alone is insufficient in the current setup. However, RQL already includes an explicit value-maximization term. The substantial degradation after removing this term therefore does not establish the incremental benefit of QGRQL’s optimized flow-matching targets over RQL.
The preliminary result that leaving the dataset actions unchanged causes no sizable performance decrease makes the remaining control particularly important. This setting needs to be precisely defined: are only the flow-matching targets restored to the original dataset actions, while QGRQL’s intermediate-state construction is retained? Are optimized actions still used in constructing those states? A completed comparison against both RQL and full QGRQL, with other settings controlled and results across multiple seeds, is needed to distinguish the benefit of changing the supervision targets from that of changing the states used for value maximization. The current results do not yet justify a strong conclusion about which proposed modification accounts for the improvement.
2. The additional computation and the robustness of the improvement remain unclear.
Comparing methods at a fixed number of outer updates is informative, but it does not answer the question about computational overhead or performance under a matched time budget. QGRQL introduces additional action-gradient optimization steps within each update, so the same number of outer updates does not imply the same computational cost. Hardware and implementation variability can be controlled through measurements on matched hardware and configurations. Describing wall-clock evaluation as non-standard does not address this concern.
At minimum, reporting the complete update time for QGRQL and RQL, including the internal action-optimization steps, would allow readers to assess the cost associated with the reported improvement. Representative comparisons under a common time budget would further strengthen the evidence.
The revised checkpoint metric is a meaningful improvement, but it does not resolve the uncertainty in the central ablations, which still use a single seed. This is particularly relevant when interpreting the reported average gain of 3.3 percentage points and the preliminary observation of little difference in the unchanged-action control.
3. The writing and presentation require substantial revision.
I also remain concerned about the clarity and organization of the manuscript. The exposition needs to distinguish more clearly between components inherited from RQL and the proposed modifications, explain how the actor and critic trajectory constructions relate to the training objectives, and make the purpose of each modification explicit. The interpretation of the ablations should also distinguish evidence that a component is important within the full method from evidence that it explains the improvement over the baseline.
In my assessment, the current writing and presentation do not yet meet the standard of clarity and precision expected at a leading AI conference such as ICLR. The algorithmic explanation and the connection between the experimental evidence and the contribution claims need substantial revision.
For these reasons, I maintain my recommendation to reject the paper in its current form. I encourage the authors to revise the exposition substantially, complete the central controlled comparisons, and quantify the additional computational cost before resubmission.
Rebuttal (1/2)
Thank you for your feedback and the detailed review!
Updated Results
We opted to report the best performance possible in our original submission since it is unclear to us which performance metric RQL reports (it is unspecified in their paper), and we chose the metric that would report the highest RQL performance to better understand our improvement over RQL. Based on reviewer feedback, we have updated the results in the paper to instead report the mean success rate on the last 3 checkpoints to standardize our reporting with earlier work like FQL. Our results exhibit the same improvement over RQL, with a +3.3 point difference when averaging success across all environments.
We have also added Q-Flow, QAM, and FQL as baselines, and we show that mean QGRQL performance across all environments is better than all of them.
Additional Ablations
We have updated our paper with multiple ablations to isolate the impact of flow supervision, regularization, and selecting optimal actions using the value function, and compared them to QGRQL and RQL (Appendix C, Table 6). Due to the short rebuttal time period, we report our results on a single seed across all the non-100M environments. We compare against the following ablations:
Our results show that QGRQL’s mean performance is better than all of the above ablations.
We are running additional experiments to identify the impact of leaving the dataset actions unchanged while performing value maximization. In the subset of environments we have evaluated so far, we do not observe any sizable decreases in performance compared to QGRQL.
We also hope that the Policy Optimization ablation addresses your concern about our motivation - while we observe the value-function/policy lag in Figure 1 on optimizing policy actions, we chose to use the value function to optimize dataset actions during training because we believed that regularizing with respect to the dataset action provides a reasonable “floor” during early training that prevented flow matching to extraordinarily poor actions. On the other hand, when we train using the Q-gradient optimized policy actions as a target, we expect actions sampled from the policy during the initial training iterations to be of significantly lower quality. Even with a strong value function, optimizing these sampled policy actions produces poor action targets, and as our results show, severely destabilizes training to the point that the final learned policy does not achieve nonzero success for multiple environments.