acceptodds
Under review as a conference paper at ICLR 2027

Score-Calibrated Flow for Sampling from Unnormalized Densities with Applications to Generative Online Reinforcement Learning

Abstract

Diffusion and flow models provide expressive policy classes for online reinforcement learning (RL), enabling multimodal behaviors and improved performance. However, training these policies remains challenging: the critic specifies the desired policy as an unnormalized Boltzmann density but does not provide direct samples from it. Many existing methods rely on importance sampling to construct training signals, which can suffer from high variance, increasing computational cost and destabilizing training. In this paper, we propose Score-Calibrated Flow (SCF), a simple and efficient algorithm for training generative models to sample from unnormalized densities without importance sampling or backpropagation through the sampling trajectory. Our key idea is to learn the desired flow by enforcing self-consistency, bypassing the challenging estimation of the target posterior mean. By jointly exploiting the prescribed target score and the structure of flow matching, we establish these self-consistency requirements as score-calibrated optimality conditions, first for the terminal density and then for the trainable velocity field. We prove that their unique solutions are, respectively, the target density and the ideal flow model that conditional flow matching (CFM) would recover if target samples were available. We formulate the velocity condition as a fixed-point equation and exploit its conditional-expectation structure to construct a stop-gradient objective for enforcing it. The resulting training procedure retains the scalable sample–interpolate–regress structure of conditional flow matching despite the absence of target samples, using endpoints generated by the current flow. For online RL, the critic gradient supplies the target score at the generated actions, yielding a direct approach to actor training. Experiments on RL benchmarks demonstrate that SCF matches or improves upon state-of-the-art generative-policy baselines, while substantially reducing training time.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, diffusion, flow, sampling

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Qcx5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Score-Calibrated Flow (SCF) for sampling from unnormalized densities when target scores are available but target samples are not. The method uses self-consistency conditions and stop-gradient regression with self-generated samples. It is also applied to online RL with bounded actions. It is evaluated on an eight-component Gaussian mixture and 14 continuous-control tasks with several sampling and RL baselines. The results show promising performance on the tested tasks.

Critical strengths

The velocity-uniqueness result is a meaningful theoretical contribution. Under stated assumptions, the target flow is the only exact fixed point. The result does not require the candidate velocity field to be the ideal CFM velocity of its generated density. This gives a useful guarantee that exact self-consistency identifies the intended sampler.

Critical weaknesses

1. The experiments do not sufficiently support the practical reliability of the proposed sampling method.

Theorems 3.4–3.5 show that the target flow is the unique exact fixed point, but they do not guarantee that the proposed training procedure can reach this solution or that a small fixed-point residual leads to accurate sampling. The paper only evaluates standalone sampling on one symmetric 2D Gaussian mixture and does not test unequal mixture weights, recovery when the initial model misses some modes, or higher dimensions. In addition, the coefficient ablation in Appendix D changes several settings at the same time, so it is unclear which component improves training stability. These limitations make it difficult to assess whether the theoretical results lead to reliable sampling in practice.

2. There is a mismatch between the maximum-entropy RL formulation and the proposed RL algorithm.

Section 2.3 defines soft policy evaluation with an entropy term, but the critic update in Section 4 does not include this term. The paper does not explain this difference. As a result, even if SCF learns the Boltzmann distribution defined by the critic, this does not guarantee policy improvement under the stated maximum-entropy objective, since the missing entropy term can affect the critic values and action rankings. The RL algorithm as written uses a critic without entropy regularization, so its connection to the maximum-entropy objective is unclear. The paper also does not evaluate the effect of this difference. In addition, the RL inverse-temperature values and how they are selected are not reported, although they directly affect the target policy. These issues weaken the connection between the proposed formulation and the experimental results.

Questions for authors
  1. Section 2.3 includes an entropy term in policy evaluation, but the critic update in Section 4 does not. Which update is actually used in the experiments, and why do the two formulations differ?

  2. Theorems 3.4–3.5 establish uniqueness of the exact fixed point, but not convergence of the proposed training procedure. Is there theoretical or empirical evidence that the training updates approach this solution? How does the fixed-point residual, rather than the sampled regression loss, relate to sampling accuracy during training?

  3. The coefficient ablation in Appendix D changes both the regression coefficients and the time-sampling interval. Can the authors compare the default setting with λt=t2, ζt=0\lambda_t=t^2,\ \zeta_t=0 under the same training settings to isolate the effect of the conditional-velocity correction?

  4. The RL experiments report policy returns but do not directly test whether SCF learns the intended Boltzmann distribution. For a fixed critic, can the authors compare the learned action distribution with a numerical reference at representative states?

  5. Proposition 3.1 is closely related to the Target Score Identity in Target Score Matching (De Bortoli et al., 2024), which is not cited. Implicit Tilt Matching (Potaptchik et al., 2026) also uses a related fixed-point regression approach. How does SCF differ from these methods, and which parts of its formulation and training procedure are new?

  6. The sampling experiment only uses one symmetric 2D Gaussian mixture. How does SCF perform with unequal mixture weights or when some modes are poorly represented at initialization?

  7. What values of τ\tau are used in the RL experiments, and how are they selected?

  8. On DeepMind Control, SCF uses best-of-32 action selection, while QSM, QFlex, and DQS use a single action sample. What are SCF's results with n=1?

References:

  • De Bortoli et al. (2024). Target Score Matching. arXiv:2402.08667.
  • Potaptchik et al. (2026). Tilt Matching for Scalable Sampling and Fine-Tuning. ICML 2026.
8 comments
AuthorsOpenReview ↗

Response to Reviewer Qcx5 (1/6)

We sincerely thank the reviewer for the time and effort devoted to reviewing our paper and for the constructive comments. We address the questions below.

Q1: Clarification on the entropy term in the critic update.

A1: Thank you for this question. Section 2.3 provides the motivation for the Boltzmann policy. Targeting the Boltzmann distribution of the critic is widely adopted for generative policies (e.g., MaxEntDP, SDAC, DQS, and RFM), and it originates from maximum-entropy RL, notably SAC and its soft policy iteration. All experiments use the critic update in Section 4, that is, the standard Q-function without the entropy term. In practice, we find that dropping this term works well empirically, and we do so for three reasons.

  1. Computational cost. A flow policy has no closed-form density. Evaluating the action log-density requires integrating the divergence of the velocity field along the sampling ODE, which SCF does not need anywhere else. This adds substantial cost to every training step (see the table below).
  2. Numerical difficulty. Even with the additional compute, the log-density is hard to evaluate reliably in general. The divergence integral accumulates discretization error, stochastic trace estimators add variance to the critic target, and accurate likelihood estimation for ODE-based generative models often depends on carefully designed objectives and problem-specific treatment [1].
  3. Established practice. We follow RFM [2], which adopts the standard Q-function for flow policies and reports that a soft-Q variant slightly degrades performance. QSM [3] likewise matches the policy to the Boltzmann distribution of the standard Q-function rather than the soft Q-function.

To quantify the effect for SCF, we compare with a soft-Q variant on h1-run and h1-walk. The soft-Q variant adds the log-density term to the critic target, computed with the exact divergence. The table reports the final return (average over the last five evaluation checkpoints; mean ± standard deviation over five seeds) and the time per training step on h1-run.

Methodh1-runh1-walkTime per training step
SCF (standard Q)825 ± 5868 ± 101.12 ms
SCF (soft Q)771 ± 30782 ± 731.54 ms

The soft-Q variant reaches comparable but slightly lower returns, while each training step takes about 37% longer. The training curves are shown in this figure for h1-run and this figure for h1-walk. We will clarify the critic update and add this comparison in the revised paper.

References:

[1] K. Zheng, C. Lu, J. Chen, and J. Zhu. Improved Techniques for Maximum Likelihood Estimation for Diffusion ODEs. ICML, 2023.

[2] Z. Li, S. Tang, and N. Azizan. Reverse Flow Matching: A Unified Framework for Online Reinforcement Learning with Diffusion and Flow Policies. ICML, 2026.

[3] M. Psenka, A. Escontrela, P. Abbeel, and Y. Ma. Learning a Diffusion Model Policy from Rewards via Q-Score Matching. ICML, 2024.

AuthorsOpenReview ↗

Response to Reviewer Qcx5 (2/6)

Q2: Evidence that training approaches the fixed point, and how the fixed-point residual relates to sampling accuracy.

A2: Thank you for this question. Theorems 3.4–3.5 guarantee that the desired velocity field is the only solution of the fixed-point equation. The training objective is constructed to enforce this equation. The sampled target is an unbiased estimate of the operator output Iλ,ζv\mathcal I_{\lambda,\zeta}v (Section 3.3), so the expected discrepancy between target and prediction is exactly the fixed-point residual, which vanishes only at the desired velocity field. Self-consistency therefore cannot be met by a spurious solution. We complement this characterization with empirical evidence on how training approaches the fixed point.

For the eight-Gaussian experiment of the paper, we track four quantities along training: the forward and reverse KL divergences between the generated density ρv\rho^v and the target π\pi, computed from the exact model density, which is tractable in two dimensions; the sampled regression loss LL; and the fixed-point residual

R=12E∥(Iλ,ζv)t(Xt)−vt(Xt)∥2.R=\frac{1}{2}\mathbb{E}\Vert(\mathcal I_{\lambda,\zeta}v)_t(X_t)-v_t(X_t)\Vert^2.

The table reports the mean over the five seeds along training; the last column is the ratio between the initial and final values.

Updates010k20k50k100kReduction
KL(ρv∥π\rho^v \Vert \pi)242.80.150.00990.00673,600×
KL(π∥ρv\pi \Vert \rho^v)301.00.110.00800.00545,500×
Fixed-point residual RR1.71.80.0280.00160.00131,300×
Sampled loss LL9.36.54.23.94.02.3×
  1. Training approaches the fixed point. RR decreases by three orders of magnitude, and both KL divergences fall below 0.01.
  2. The residual tracks sampling accuracy, whereas the sampled loss plateaus. Over all snapshots along training, the Spearman rank correlation (a measure of how consistently two quantities rise and fall together) between RR and KL(ρv∥π\rho^v \Vert \pi) is 0.92. The sampled loss instead plateaus near 4 while RR and the KL keep decreasing, because L=R+VL=R+V, where VV is the conditional variance of the sampled target, which remains positive at the solution, as for the conditional flow matching (CFM) loss.

The KL divergences along training are shown in this figure, the residual and the sampled loss in this figure, and the generated density in this figure. We will add this study to the revised paper.

AuthorsOpenReview ↗

Response to Reviewer Qcx5 (3/6)

Q3: Isolating the effect of the conditional-velocity term in the coefficient ablation.

A3: Thank you for this suggestion. We ran the requested variant, λt=t2\lambda_t=t^2 and ζt=0\zeta_t=0, on h1-run with five seeds. It differs from the default only in ζt\zeta_t; all other training settings, including the uniform sampling of tt on (0,1)(0,1), are unchanged. The table reports the final return (average over the last five evaluation checkpoints; mean ± standard deviation over five seeds).

Coefficientsh1-run
Default825 ± 5
λt=t2\lambda_t=t^2, ζt=0\zeta_t=0207 ± 105
λt=1\lambda_t=1, ζt=0\zeta_t=08 ± 4

The two terms play different roles. Without the conditional-velocity term (second row), training still works but is slower: learning starts about 1.5M environment steps later and is still improving at the end of training, so the final return is 207 instead of 825. Without the singularity cancellation λt=t2\lambda_t=t^2 (third row), training breaks and the return never improves. The training curves are shown in this figure. We will add the new variant to Appendix D in the revised paper.

Q4: Direct comparison of the learned action distribution with the Boltzmann distribution of the critic.

A4: Thank you for this suggestion. We added this test on the multi-goal environment [4], a standard benchmark for multimodal policies that is also used by DACER [5] and MaxEntDP [6]. In this environment, a point mass with two-dimensional actions is rewarded for reaching any of four goals. With two-dimensional actions, both densities in the comparison are tractable: the Boltzmann distribution πQ\pi^Q of a fixed critic is normalized numerically on a fine action grid, and the policy density πθ\pi^\theta is computed exactly by the change-of-variables formula along the sampling ODE. We train SCF with the hyperparameters of our DMC experiments and five seeds; the reward scale and temperature follow the original example. We then compare the learned policy with the Boltzmann distribution of its critic at the three states visualized in [4], where this distribution is unimodal, four-modal, and bimodal.

Unimodal stateFour-modal stateBimodal state
KL(πθ∥πQ\pi^\theta \Vert \pi^Q)0.004 ± 0.0030.015 ± 0.0040.076 ± 0.029

The KL divergence is below 0.1 at all three states, including the multimodal ones. Averaged over the three states, it decreases from 3.2 at initialization to 0.03 at the end of training. The trajectories and the samples of the learned policy at the three states are shown in this figure for one seed. We will add this experiment to the revised paper.

References:

[4] T. Haarnoja, H. Tang, P. Abbeel, and S. Levine. Reinforcement Learning with Deep Energy-Based Policies. ICML, 2017.

[5] Y. Wang, L. Wang, Y. Jiang, W. Zou, T. Liu, X. Song, W. Wang, L. Xiao, J. Wu, J. Duan, and S. E. Li. Diffusion Actor-Critic with Entropy Regulator. NeurIPS, 2024.

[6] X. Dong, J. Cheng, and X. S. Zhang. Maximum Entropy Reinforcement Learning with Diffusion Policy. ICML, 2025.

Reviewer_SuU7OpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Score-Calibrated Flow (SCF), a method for training flow-matching generative policies to sample from unnormalized target densities in online reinforcement learning. By establishing score-calibrated self-consistency conditions between the target score and the flow velocity field, SCF constructs a stop-gradient regression objective on self-generated endpoints, bypassing importance sampling and BPTT. The authors demonstrate strong control performance and computational efficiency on DeepMind Control Suite and HumanoidBench.

Critical strengths
  1. Formulating policy learning as sampling from an unnormalized density provides a principled alternative to standard SAC actor losses, avoiding explicit policy likelihood evaluations and trajectory differentiation while addressing bounded action bounds and numerical singularities.

  2. The paper establishes score-calibrated optimality conditions with uniqueness proofs for both the target density and velocity field, offering a theoretical guarantee that self-generated training recovers the true target distribution rather than reproducing sampler bias.

  3. Using a stop-gradient regression objective on self-generated endpoints bypasses costly importance sampling and BPTT, yielding marked wall-clock training speedups over existing generative baselines.

  4. The empirical evaluation covers synthetic multimodal density estimation and complex robotic control (DeepMind Control Suite and HumanoidBench) against a wide array of state-of-the-art generative policies.

Critical weaknesses
  1. The manuscript should more explicitly delineate SCF’s core technical advance over closely related flow methods like Reverse Flow Matching (RFM) and Flow Sampling, clarifying how its self-consistency formulation uniquely avoids both stochastic diffusion noise and posterior-mean estimation.

  2. Varied baseline hyperparameters (such as best of n candidate action selection and NFE step counts) make it difficult to isolate performance gains resulting strictly from the SCF loss versus evaluation-time sampling choices.

  3. Although SCF reduces per-iteration wall-clock time relative to other generative policies, it remains slower than Gaussian SAC; demonstrating end-to-end wall-clock time to reach specific return thresholds would better validate its practical efficiency.

  4. General unnormalized sampling is evaluated on a single 2D Gaussian mixture, leaving open how effectively the score-calibrated optimality conditions handle higher-dimensional or more complex target density landscapes.

Questions for authors

Please address the weaknesses; more specificallly:

  1. What is the primary technical distinction between SCF and Reverse Flow Matching (RFM)? Beyond eliminating explicit posterior-mean estimation, in which sampling regimes or density geometries is SCF expected to fundamentally outperform RFM?
  2. Could the authors provide a controlled comparison holding evaluation-time candidate action selection best of n and NFE step counts strictly matched across all baselines to isolate gains resulting specifically from the SCF objective? I think this is an important missing result discussion in the paper.
  3. How do the population-level uniqueness guarantees bridge to practical SGD training under finite network capacity and evolving critic landscapes, and what prevents self-generated regression from settling into poor local optima?
5 comments
AuthorsOpenReview ↗

Response to Reviewer SuU7 (1/3)

We sincerely thank the reviewer for the time and effort devoted to reviewing our paper and for the constructive comments. We address the questions below.

Q1: Primary technical distinction from RFM and expected advantages across sampling regimes.

A1: Thank you for this question.

  1. Primary technical distinction. The two methods construct the regression target from different posteriors. RFM [1] uses the posterior of the target: for a noisy sample XtX_t, its regression target is a posterior mean under the Boltzmann density. This posterior is known only up to a normalizing constant, so RFM’s implementation estimates the mean by self-normalized importance sampling over many proposals for every training example, with control variates for variance reduction. SCF uses the posterior of the current model: the endpoint is generated by the flow itself, and the target density enters only through its score at the generated endpoint. Establishing that this replacement still identifies the target is the core theoretical contribution of SCF: we prove that the resulting fixed-point equation holds if and only if the generated density is the target and the velocity field is the ideal CFM velocity of the target (Theorems 3.4–3.5). The proof is highly nontrivial, because the candidate velocity is not assumed to be consistent with the CFM velocity of its own generated density. It combines a near-source limit of the fixed-point equation with analytic continuation to an identity between Fourier transforms, and it rules out spurious self-consistent solutions. Theorem 3.5 further provides a broad family of operators that share this unique fixed point, which opens a large design space for the algorithm, exemplified by our singularity-cancelling coefficients. As a result, each regression target requires only one generated endpoint and one target-score evaluation, without auxiliary importance samples or importance weights.
  2. Regimes where SCF is expected to outperform RFM. The accuracy of importance sampling depends on how well the proposal covers the target posterior, and the number of proposals required grows exponentially with the Kullback–Leibler divergence between the two [2]. This mismatch can become severe in high-dimensional problems when discrepancies accumulate across coordinates, or when sharply concentrated and separated posterior modes are poorly covered by the proposal. In these settings, many proposal samples receive negligible weight, making accurate posterior supervision expensive. RFM’s control variates mitigate estimation variance, while SCF avoids the auxiliary importance-sampling step altogether: each generated endpoint supplies a regression target through one target-score evaluation, without normalized importance weights. We therefore expect SCF’s computational advantage to be largest when posterior–proposal overlap is poor, including high-dimensional, sharply concentrated, or multimodal problems that induce difficult conditional posteriors.

We will add this discussion to the revised paper.

References:

[1] Z. Li, S. Tang, and N. Azizan. Reverse Flow Matching: A Unified Framework for Online Reinforcement Learning with Diffusion and Flow Policies. ICML, 2026.

[2] S. Chatterjee and P. Diaconis. The Sample Size Required in Importance Sampling. The Annals of Applied Probability, 2018.

AuthorsOpenReview ↗

Response to Reviewer SuU7 (2/3)

Q2: Controlled comparison with matched sampling budgets.

A2: Thank you for this suggestion. We ran this controlled comparison on both benchmarks: SCF and all diffusion and flow baselines of the paper are evaluated under the same setting, n=1n=1 (a single action is generated by the policy and executed, without critic-based selection) and 20 NFEs. DIME is trained by backpropagation through the discretized sampling dynamics, and its runs in the paper use 16 NFEs; to align the NFEs, we retrain it with 20. The tables report the final return (average over the last five evaluation checkpoints; mean ± standard deviation over five seeds), with the top three methods on each task in bold.

  1. DMC. SCF attains the highest average return among all methods (815, compared with 740 for the best baseline) and is the only method among the top three on every task.
TaskSCFRFMQFlexQSMQVPODQSMaxEntDPSDACDIME
cartpole-swingup851 ± 7814 ± 18841 ± 17673 ± 43790 ± 34858 ± 15837 ± 9450 ± 29736 ± 32
fish-upright752 ± 48392 ± 32537 ± 75810 ± 38690 ± 25417 ± 113686 ± 81456 ± 20675 ± 90
finger-spin903 ± 76666 ± 56907 ± 90613 ± 49631 ± 28465 ± 303876 ± 161561 ± 49126 ± 280
finger-turn_hard762 ± 85611 ± 77523 ± 38394 ± 98696 ± 111117 ± 21573 ± 189585 ± 99184 ± 138
reacher-hard930 ± 20908 ± 35827 ± 86934 ± 28861 ± 58930 ± 30828 ± 133596 ± 87222 ± 293
cheetah-run685 ± 4614 ± 1483 ± 92692 ± 80452 ± 37624 ± 97643 ± 5918 ± 1658 ± 79
walker-walk956 ± 378 ± 10912 ± 99962 ± 4760 ± 85859 ± 128948 ± 14201 ± 10955 ± 21
walker-run684 ± 632 ± 0613 ± 81694 ± 60291 ± 48695 ± 23524 ± 14747 ± 2545 ± 220
Average815439705721646620740364513
  1. HumanoidBench. SCF achieves the highest return on five of the six tasks and is on par with the best method on h1-pole.
TaskSCFRFMQFlexQSMQVPODQSMaxEntDPSDACDIME
h1-walk869 ± 1029 ± 9818 ± 36785 ± 15137 ± 41777 ± 234 ± 017 ± 270 ± 58
h1-run825 ± 58 ± 0731 ± 44660 ± 6936 ± 24683 ± 1013 ± 05 ± 09 ± 4
h1-sit923 ± 12119 ± 14807 ± 34725 ± 14282 ± 28773 ± 494 ± 19031 ± 3536 ± 295
h1-crawl867 ± 49554 ± 208572 ± 13794 ± 125501 ± 113717 ± 226717 ± 265296 ± 22524 ± 106
h1-pole593 ± 23913 ± 5595 ± 83287 ± 37519 ± 8211 ± 26014 ± 1711 ± 1012 ± 15
h1-reach7533 ± 5743240 ± 5205999 ± 2783094 ± 3372974 ± 2533952 ± 8266668 ± 2834079 ± 2086649 ± 746

These results demonstrate the superior performance of SCF in RL experiments. An interesting finding is that SDAC and RFM make use of best-of-nn action selection during training, so evaluating them with n=1n=1 introduces a mismatch between training and inference, and their performance degrades compared with their results in the paper. In contrast, SCF does not rely on best-of-nn action selection: on DMC, its average return with n=1n=1 is only slightly lower than with n=32n=32 in the paper (815 versus 826). We will add these results to the revised paper.

AuthorsOpenReview ↗

Response to Reviewer SuU7 (3/3)

Q3: Bridging the population-level uniqueness guarantees to practical training, and what prevents poor local optima.

A3: Thank you for this insightful question.

  1. What the uniqueness guarantee provides. The stop-gradient regression of SCF is constructed to enforce the fixed-point equation: the sampled target is an unbiased estimate of the operator output, so the expected discrepancy between target and prediction is exactly the fixed-point residual. Theorems 3.4–3.5 show that this residual vanishes only at the desired velocity field. At the population level, the regression therefore has no spurious stationary point: any other field, for example one that covers only part of the target, has a nonzero residual.

  2. Finite capacity, SGD, and an evolving critic. A guarantee that covers these aspects would require a convergence analysis of non-convex stochastic optimization coupled with temporal-difference learning of the critic. This is a long-standing open problem for deep RL in general, and guarantees in this area are typically established in idealized settings. For example, soft policy iteration is proved to converge in the tabular case [3], and sampling guarantees for diffusion models assume an accurately learned score [4]. Such an analysis is beyond the scope of this paper, and we consider it an important direction for future work.

  3. Empirical support. We complement the theory with experiments that examine each of these aspects in practice.

    (i) Finite capacity and SGD: training approaches the fixed point. For the eight-Gaussian experiment of the paper, we track along training the forward and reverse KL divergences between the generated density ρv\rho^v and the target π\pi, computed from the exact model density, which is tractable in two dimensions, together with the fixed-point residual

    R=12E∥(Iλ,ζv)t(Xt)−vt(Xt)∥2.R=\frac{1}{2}\mathbb{E}\Vert(\mathcal I_{\lambda,\zeta}v)_t(X_t)-v_t(X_t)\Vert^2.

    The table reports the mean over the five seeds along training; the last column is the ratio between the initial and final values.

    Updates010k20k50k100kReduction
    KL(ρv∥π\rho^v \Vert \pi)242.80.150.00990.00673,600×
    KL(π∥ρv\pi \Vert \rho^v)301.00.110.00800.00545,500×
    Fixed-point residual RR1.71.80.0280.00160.00131,300×

    RR decreases by three orders of magnitude, and both KL divergences fall below 0.01. The residual also tracks sampling accuracy: over all snapshots along training, the Spearman rank correlation (a measure of how consistently two quantities rise and fall together) between RR and KL(ρv∥π\rho^v \Vert \pi) is 0.92. The KL divergences are shown in this figure, the residual in this figure, and the generated density along training in this figure.

    (ii) Evolving critic. We use the multi-goal environment [5], a standard benchmark for multimodal policies, in which the critic is learned jointly with the policy. With two-dimensional actions, both densities are tractable: the Boltzmann distribution πQ\pi^Q of the critic is normalized numerically on a fine action grid, and the policy density πθ\pi^\theta is computed exactly by the change-of-variables formula along the sampling ODE. The table compares them at the three states visualized in [5], where the Boltzmann distribution is unimodal, four-modal, and bimodal (mean ± standard deviation over five seeds).

    Unimodal stateFour-modal stateBimodal state
    KL(πθ∥πQ\pi^\theta \Vert \pi^Q)0.004 ± 0.0030.015 ± 0.0040.076 ± 0.029

    Averaged over the three states, the KL divergence decreases from 3.2 at initialization to 0.03 at the end of training. The trajectories and the samples of the learned policy at the three states are shown in this figure for one seed.

    (iii) Large-scale tasks. On 14 continuous-control tasks with five seeds each, SCF ranks among the top-performing methods, including under the matched evaluation in A2.

References:

[3] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. ICML, 2018.

[4] S. Chen, S. Chewi, J. Li, Y. Li, A. Salim, and A. R. Zhang. Sampling is as Easy as Learning the Score: Theory for Diffusion Models with Minimal Data Assumptions. ICLR, 2023.

[5] T. Haarnoja, H. Tang, P. Abbeel, and S. Levine. Reinforcement Learning with Deep Energy-Based Policies. ICML, 2017.

Reviewer_WQ4GOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Score-Calibrated Flow (SCF), a method for training a deterministic flow (an ODE sampler) to sample from an unnormalized density. Training uses samples generated by the current flow plus one evaluation of the target score per sample, so no samples from the target are needed. Under stated regularity assumptions, the theory gives score-based conditions that hold if and only if the flow's output distribution equals the target and its velocity field equals the target's conditional flow matching (CFM) velocity. In practice these conditions become a regression loss with a stop-gradient target whose fixed point is the desired flow. A change of variables with a Jacobian correction handles bounded action spaces.

Critical strengths
  1. Clean theory that leads to a cheap training rule. Theorems 3.2, 3.4 and 3.5 give necessary and sufficient conditions for the flow to sample the target. Theorems 3.4 and 3.5 are the stronger results: they apply to any admissible velocity field that satisfies the growth and moment conditions (23)–(24), not only to fields that are already the CFM field of their own output distribution. The resulting update (the regression target in Eq. 18) is computed per sample and needs neither importance weights nor backpropagation through the sampling trajectory. The details are handled carefully. The coefficient schedules cancel the singularity of the regression target as t → 0 while still satisfying condition (16) of Theorem 3.5 (Appendix B.4), and the score of the tanh-transformed target includes the correct Jacobian term (Proposition 3.6). This combination of a uniqueness result and a cheap update is the main reason for my positive score.
  2. Competitive RL results, with the lowest per-iteration training time of the generative-policy methods in Table 2. SCF is at or near the top on all six HumanoidBench tasks and on most of the eight DMC tasks (Figures 2–3). On HumanoidBench it evaluates with a single sampled action (n = 1, Appendix C.2), so these results do not rely on picking the best of several actions with the critic. On DMC it executes the best of 32 sampled actions, while DQS, QSM and QFlex use one (see Question 2). All methods share one codebase per benchmark and are run with five seeds. Per training iteration, SCF is the fastest generative-policy method on both tasks timed in Table 2, ahead of the next fastest, QSM, by a small margin (0.87 vs 1.08 ms on walker-run and 1.12 vs 1.21 ms on h1-run); SAC is faster still (0.47 and 1.02 ms).
Critical weaknesses

The critic does not match the soft-policy-iteration motivation, and the RL temperature is not reported. Section 2.3 motivates the actor update with soft policy iteration, which requires a soft Q-function. The critic in Section 4, however, is trained with an ordinary Bellman target without an entropy term. Using an ordinary Q-function is a reasonable shortcut; RFM does the same to avoid computing log-densities (https://arxiv.org/html/2601.08136v1). But the method is then no longer soft policy iteration, and I think the paper should say so.

Questions for authors

The critic target in Section 4 has no entropy term. Is that intentional, i.e. does the critic estimate the ordinary rather than the soft Q-function? Please also give the inverse temperature used in each RL experiment, how it was chosen or adapted, and any per-task tuning.

1 comment
AuthorsOpenReview ↗

Response to Reviewer WQ4G

We sincerely thank the reviewer for the time and effort devoted to reviewing our paper and for the constructive comments. We address the questions below.

Q1: Clarification on the entropy term in the critic update.

A1: Thank you for this question. Section 2.3 provides the motivation for the Boltzmann policy. Targeting the Boltzmann distribution of the critic is widely adopted for generative policies (e.g., MaxEntDP, SDAC, DQS, and RFM), and it originates from maximum-entropy RL, notably SAC and its soft policy iteration. All experiments use the critic update in Section 4, that is, the standard Q-function without the entropy term. In practice, we find that dropping this term works well empirically, and we do so for three reasons.

  1. Computational cost. A flow policy has no closed-form density. Evaluating the action log-density requires integrating the divergence of the velocity field along the sampling ODE, which SCF does not need anywhere else. This adds substantial cost to every training step (see the table below).
  2. Numerical difficulty. Even with the additional compute, the log-density is hard to evaluate reliably in general. The divergence integral accumulates discretization error, stochastic trace estimators add variance to the critic target, and accurate likelihood estimation for ODE-based generative models often depends on carefully designed objectives and problem-specific treatment [1].
  3. Established practice. We follow RFM [2], which adopts the standard Q-function for flow policies and reports that a soft-Q variant slightly degrades performance. QSM [3] likewise matches the policy to the Boltzmann distribution of the standard Q-function rather than the soft Q-function.

To quantify the effect for SCF, we compare with a soft-Q variant on h1-run and h1-walk. The soft-Q variant adds the log-density term to the critic target, computed with the exact divergence. The table reports the final return (average over the last five evaluation checkpoints; mean ± standard deviation over five seeds) and the time per training step on h1-run.

Methodh1-runh1-walkTime per training step
SCF (standard Q)825 ± 5868 ± 101.12 ms
SCF (soft Q)771 ± 30782 ± 731.54 ms

The soft-Q variant reaches comparable but slightly lower returns, while each training step takes about 37% longer. The training curves are shown in this figure for h1-run and this figure for h1-walk. We will clarify the critic update and add this comparison in the revised paper.

References:

[1] K. Zheng, C. Lu, J. Chen, and J. Zhu. Improved Techniques for Maximum Likelihood Estimation for Diffusion ODEs. ICML, 2023.

[2] Z. Li, S. Tang, and N. Azizan. Reverse Flow Matching: A Unified Framework for Online Reinforcement Learning with Diffusion and Flow Policies. ICML, 2026.

[3] M. Psenka, A. Escontrela, P. Abbeel, and Y. Ma. Learning a Diffusion Model Policy from Rewards via Q-Score Matching. ICML, 2024.

Q2: Values of τ\tau in the RL experiments.

A2: Thank you for this question. The values of τ\tau used in the RL experiments are given below.

BenchmarkTasksτ\tau
HumanoidBenchh1-walk, h1-run, h1-sit, h1-reach10
HumanoidBenchh1-crawl, h1-pole50
DMCall eight tasks50
  1. HumanoidBench. The inverse temperature τ\tau sets the exploration level of the Boltzmann policy, and we choose it according to the characteristics of each task. Crawl and pole require precise movements in confined configurations and tolerate little action noise, so they use the larger τ\tau, while the other tasks benefit from more exploration. For each task, all baselines that target a Boltzmann distribution (RFM, DQS, MaxEntDP) use the same temperature.
  2. DMC. These experiments are built on the codebase of RFM [2] and keep its settings for the baselines. Following RFM, we set τ=50\tau=50.

We will report these values in the revised paper.

References:

[2] Z. Li, S. Tang, and N. Azizan. Reverse Flow Matching: A Unified Framework for Online Reinforcement Learning with Diffusion and Flow Policies. ICML, 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.