- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Q-learning with Scalar Adjoint Matching (SQAM) for off-policy fine-tuning of pretrained flow policies. Motivated by the approximately diagonal structure of batch-averaged velocity Jacobians, it replaces the backward adjoint computation with a time-scaled critic gradient at the generated action, eliminating per-step vector–Jacobian products. SQAM combines this approximation with a value penalty at policy-generated actions and retains TRQAM’s trust-region mechanism by default. The method is evaluated on 50 OGBench tasks in offline and offline-to-online settings, with additional experiments fine-tuning a vision-language-action policy on three real-robot tasks. The paper also reports adjoint diagnostics, regularization ablations, and per-update runtime measurements.
Critical strengths
This paper replaces the per-step vector–Jacobian products with a time-scaled terminal critic gradient, simplifying the training procedure. The per-update timing experiments provide evidence of computational savings, making the approach potentially useful for fine-tuning expressive flow policies.
The OGBench evaluation covers 50 tasks with eight seeds, and the improvements are substantial on several challenging domains. The real-robot experiments provide additional evidence of applicability beyond small benchmark policies. The critic-regularization ablations are also useful for understanding when the scalar approximation succeeds or fails.
Critical weaknesses
(a) The contribution of the scalar approximation to the reported performance is not sufficiently isolated. SQAM changes several components relative to TRQAM: the adjoint computation, the critic objective, and the sampling procedure used to construct intermediate training states. Consequently, the main comparison does not establish how much of the performance improvement arises from the scalar approximation itself, rather than the value penalty or bridge-state sampling. An approximate update can outperform an exact adjoint when using an imperfect critic, but the mechanism underlying that advantage is not sufficiently established here.
Appendix F.2 makes a useful step by adding the value penalty to TRQAM. However, this comparison is limited to two tasks and uses method-specific KL budgets. Broader comparisons with matched KL budgets and comparable tuning, together with an evaluation separating adjoint approximation from sampling changes, would substantially strengthen the assessment of the methodological contribution.
(b) The evaluation does not yet establish the method’s practical advantage over relevant alternatives under comparable budgets. Computational savings are central to the motivation, but Figure 15 measures individual updates, while the main performance results are plotted against training steps. End-to-end training time and success rate under a fixed wall-clock budget remain unclear, particularly for the large pretrained policy setting.
The baseline coverage also requires clarification. QC-FQL is directly relevant to action-chunked OGBench learning, yet the paper reports only a baseline labeled FQL while stating that manipulation tasks use action chunks of length five. It is unclear whether this implementation includes the chunked critic and corresponding multi-step Bellman backup of QC-FQL. SAC Flow is cited but not experimentally evaluated, despite addressing stable off-policy training of flow policies. These gaps limit the conclusions that can be drawn about SQAM’s competitive performance and practical efficiency.
Questions for authors
(1) Does the reported FQL implementation include QC-FQL’s chunked critic and corresponding multi-step Bellman backup, or does action chunking apply only to policy outputs? If it implements QC-FQL, please identify the remaining implementation differences. Otherwise, can you provide a properly tuned QC-FQL comparison under matched data and training budgets? What motivated the exclusion of SAC Flow from the experimental comparison?
(2) Can you extend the comparison between TRQAM with the value penalty and SQAM to more tasks using matched KL budgets and comparable hyperparameter tuning? In particular, comparing exact and scalar adjoints on the same rollout states, followed by separately evaluating bridge-state sampling, would clarify which modifications account for the improvements.
(3) Can you report success rate versus wall-clock time and time to reach representative success thresholds for SQAM and TRQAM under matched hardware and training configurations?
(4) Please provide the code implementation of this paper.