acceptodds
Under review as a conference paper at ICLR 2027

Q-Learning with Scalar Adjoint Matching

Abstract

Flow policies capture rich and diverse action distributions, and fine-tuning them with off-policy RL to improve beyond the demonstrations has drawn growing interest. However, fine-tuning a flow policy against a learned value function is not trivial, because the policy generates its action over many flow steps. Adjoint matching offers a principled way to update the flow model itself by propagating value information from the final action back to each flow step, but it requires a vector–Jacobian product through the policy at every step, a cost that grows with the number of flow steps and the policy size. We observe that the batch-averaged velocity Jacobian of pretrained flow policies concentrates on its diagonal. Motivated by this finding, we derive a closed-form scalar adjoint that scales the value gradient at the final action by the flow time, eliminating the per-step vector–Jacobian products. We further find that controlling the critic's value at policy-generated actions is particularly important under the scalar adjoint. Based on these findings, we propose Q-learning with Scalar Adjoint Matching (SQAM), which combines the scalar adjoint with a value penalty at those actions. SQAM's gains concentrate on the four hardest OGBench domains, where its success rate exceeds that of the strongest baseline in each domain by 18 to 35 percentage points. To test whether SQAM extends to large pretrained policies, we also fine-tune a vision-language-action policy on a real bimanual robot. SQAM improves over supervised fine-tuning on all three tasks.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, flow matching, diffusion policy, offline-to-online reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Nky3OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Q-learning with Scalar Adjoint Matching (SQAM) for off-policy fine-tuning of pretrained flow policies. Motivated by the approximately diagonal structure of batch-averaged velocity Jacobians, it replaces the backward adjoint computation with a time-scaled critic gradient at the generated action, eliminating per-step vector–Jacobian products. SQAM combines this approximation with a value penalty at policy-generated actions and retains TRQAM’s trust-region mechanism by default. The method is evaluated on 50 OGBench tasks in offline and offline-to-online settings, with additional experiments fine-tuning a vision-language-action policy on three real-robot tasks. The paper also reports adjoint diagnostics, regularization ablations, and per-update runtime measurements.

Critical strengths

This paper replaces the per-step vector–Jacobian products with a time-scaled terminal critic gradient, simplifying the training procedure. The per-update timing experiments provide evidence of computational savings, making the approach potentially useful for fine-tuning expressive flow policies.

The OGBench evaluation covers 50 tasks with eight seeds, and the improvements are substantial on several challenging domains. The real-robot experiments provide additional evidence of applicability beyond small benchmark policies. The critic-regularization ablations are also useful for understanding when the scalar approximation succeeds or fails.

Critical weaknesses

(a) The contribution of the scalar approximation to the reported performance is not sufficiently isolated. SQAM changes several components relative to TRQAM: the adjoint computation, the critic objective, and the sampling procedure used to construct intermediate training states. Consequently, the main comparison does not establish how much of the performance improvement arises from the scalar approximation itself, rather than the value penalty or bridge-state sampling. An approximate update can outperform an exact adjoint when using an imperfect critic, but the mechanism underlying that advantage is not sufficiently established here.

Appendix F.2 makes a useful step by adding the value penalty to TRQAM. However, this comparison is limited to two tasks and uses method-specific KL budgets. Broader comparisons with matched KL budgets and comparable tuning, together with an evaluation separating adjoint approximation from sampling changes, would substantially strengthen the assessment of the methodological contribution.

(b) The evaluation does not yet establish the method’s practical advantage over relevant alternatives under comparable budgets. Computational savings are central to the motivation, but Figure 15 measures individual updates, while the main performance results are plotted against training steps. End-to-end training time and success rate under a fixed wall-clock budget remain unclear, particularly for the large pretrained policy setting.

The baseline coverage also requires clarification. QC-FQL is directly relevant to action-chunked OGBench learning, yet the paper reports only a baseline labeled FQL while stating that manipulation tasks use action chunks of length five. It is unclear whether this implementation includes the chunked critic and corresponding multi-step Bellman backup of QC-FQL. SAC Flow is cited but not experimentally evaluated, despite addressing stable off-policy training of flow policies. These gaps limit the conclusions that can be drawn about SQAM’s competitive performance and practical efficiency.

Questions for authors

(1) Does the reported FQL implementation include QC-FQL’s chunked critic and corresponding multi-step Bellman backup, or does action chunking apply only to policy outputs? If it implements QC-FQL, please identify the remaining implementation differences. Otherwise, can you provide a properly tuned QC-FQL comparison under matched data and training budgets? What motivated the exclusion of SAC Flow from the experimental comparison?

(2) Can you extend the comparison between TRQAM with the value penalty and SQAM to more tasks using matched KL budgets and comparable hyperparameter tuning? In particular, comparing exact and scalar adjoints on the same rollout states, followed by separately evaluating bridge-state sampling, would clarify which modifications account for the improvements.

(3) Can you report success rate versus wall-clock time and time to reach representative success thresholds for SQAM and TRQAM under matched hardware and training configurations?

(4) Please provide the code implementation of this paper.

Reviewer_qRBAOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper finds that the batch-averaged velocity Jacobian of a pretrained flow policy concentrates on its diagonal, eliminating the need for vector–Jacobian product adjoint matching performed in each step, and proposes a Q-learning algorithm that combines scalar concomitant with value penalties at these actions.

Critical strengths

None

Critical weaknesses
  1. This paper does not explain why its method is better than adjoint matching. It remains unclear how saving the computation of vector–Jacobian product adjoint matching translates into gain on the four hardest OGBench domains.
  2. In some domains, scalar adjoint alone underperforms. And it is unclear whether SQAM still offers advantages over other RL methods in this large pretrained policy setting.
Questions for authors

refer to weaknesses

Reviewer_cTsjOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SQAM, an efficient method for fine-tuning flow policies with off-policy RL. It replaces the per-step vector–Jacobian products in adjoint matching with a scalar approximation of the terminal critic gradient, and introduces a critic value penalty at policy-generated actions. The method is evaluated on 50 OGBench tasks and three real-robot manipulation tasks.

Critical strengths
  • A clear and useful computational contribution. Removing per-step VJPs addresses a practical bottleneck in adjoint matching, particularly for large pretrained policies. The paper reports measured update-time savings.
  • Useful diagnostic experiments. The paper compares exact and scalar adjoints and evaluates multiple critic regularizers, including applying the proposed value penalty to the exact-adjoint baseline.
Critical weaknesses
  • The Jacobian evidence does not fully justify the approximation. A batch-averaged Jacobian being approximately diagonal does not imply that individual Jacobians are diagonal, nor that their diagonal entries are approximately equal. Sample-wise diagnostics, especially errors in the critic-gradient direction, would better support the isotropic approximation.
Questions for authors
  • How accurate is the isotropic approximation at the individual-sample level, and does this accuracy persist during fine-tuning?
  • How does scalar adjoint matching perform when evaluated on the original trajectory states rather than bridge states?
  • How sensitive is performance to the specific time weighting (\tau), compared with constant or alternative scalar schedules?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.