- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Score-Calibrated Flow (SCF) for sampling from unnormalized densities when target scores are available but target samples are not. The method uses self-consistency conditions and stop-gradient regression with self-generated samples. It is also applied to online RL with bounded actions. It is evaluated on an eight-component Gaussian mixture and 14 continuous-control tasks with several sampling and RL baselines. The results show promising performance on the tested tasks.
Critical strengths
The velocity-uniqueness result is a meaningful theoretical contribution. Under stated assumptions, the target flow is the only exact fixed point. The result does not require the candidate velocity field to be the ideal CFM velocity of its generated density. This gives a useful guarantee that exact self-consistency identifies the intended sampler.
Critical weaknesses
1. The experiments do not sufficiently support the practical reliability of the proposed sampling method.
Theorems 3.4–3.5 show that the target flow is the unique exact fixed point, but they do not guarantee that the proposed training procedure can reach this solution or that a small fixed-point residual leads to accurate sampling. The paper only evaluates standalone sampling on one symmetric 2D Gaussian mixture and does not test unequal mixture weights, recovery when the initial model misses some modes, or higher dimensions. In addition, the coefficient ablation in Appendix D changes several settings at the same time, so it is unclear which component improves training stability. These limitations make it difficult to assess whether the theoretical results lead to reliable sampling in practice.
2. There is a mismatch between the maximum-entropy RL formulation and the proposed RL algorithm.
Section 2.3 defines soft policy evaluation with an entropy term, but the critic update in Section 4 does not include this term. The paper does not explain this difference. As a result, even if SCF learns the Boltzmann distribution defined by the critic, this does not guarantee policy improvement under the stated maximum-entropy objective, since the missing entropy term can affect the critic values and action rankings. The RL algorithm as written uses a critic without entropy regularization, so its connection to the maximum-entropy objective is unclear. The paper also does not evaluate the effect of this difference. In addition, the RL inverse-temperature values and how they are selected are not reported, although they directly affect the target policy. These issues weaken the connection between the proposed formulation and the experimental results.
Questions for authors
-
Section 2.3 includes an entropy term in policy evaluation, but the critic update in Section 4 does not. Which update is actually used in the experiments, and why do the two formulations differ?
-
Theorems 3.4–3.5 establish uniqueness of the exact fixed point, but not convergence of the proposed training procedure. Is there theoretical or empirical evidence that the training updates approach this solution? How does the fixed-point residual, rather than the sampled regression loss, relate to sampling accuracy during training?
-
The coefficient ablation in Appendix D changes both the regression coefficients and the time-sampling interval. Can the authors compare the default setting with under the same training settings to isolate the effect of the conditional-velocity correction?
-
The RL experiments report policy returns but do not directly test whether SCF learns the intended Boltzmann distribution. For a fixed critic, can the authors compare the learned action distribution with a numerical reference at representative states?
-
Proposition 3.1 is closely related to the Target Score Identity in Target Score Matching (De Bortoli et al., 2024), which is not cited. Implicit Tilt Matching (Potaptchik et al., 2026) also uses a related fixed-point regression approach. How does SCF differ from these methods, and which parts of its formulation and training procedure are new?
-
The sampling experiment only uses one symmetric 2D Gaussian mixture. How does SCF perform with unequal mixture weights or when some modes are poorly represented at initialization?
-
What values of are used in the RL experiments, and how are they selected?
-
On DeepMind Control, SCF uses best-of-32 action selection, while QSM, QFlex, and DQS use a single action sample. What are SCF's results with n=1?
References:
- De Bortoli et al. (2024). Target Score Matching. arXiv:2402.08667.
- Potaptchik et al. (2026). Tilt Matching for Scalable Sampling and Fine-Tuning. ICML 2026.
8 comments
Response to Reviewer Qcx5 (2/6)
Q2: Evidence that training approaches the fixed point, and how the fixed-point residual relates to sampling accuracy.
A2: Thank you for this question. Theorems 3.4–3.5 guarantee that the desired velocity field is the only solution of the fixed-point equation. The training objective is constructed to enforce this equation. The sampled target is an unbiased estimate of the operator output (Section 3.3), so the expected discrepancy between target and prediction is exactly the fixed-point residual, which vanishes only at the desired velocity field. Self-consistency therefore cannot be met by a spurious solution. We complement this characterization with empirical evidence on how training approaches the fixed point.
For the eight-Gaussian experiment of the paper, we track four quantities along training: the forward and reverse KL divergences between the generated density and the target , computed from the exact model density, which is tractable in two dimensions; the sampled regression loss ; and the fixed-point residual
The table reports the mean over the five seeds along training; the last column is the ratio between the initial and final values.
| Updates | 0 | 10k | 20k | 50k | 100k | Reduction |
|---|---|---|---|---|---|---|
| KL() | 24 | 2.8 | 0.15 | 0.0099 | 0.0067 | 3,600× |
| KL() | 30 | 1.0 | 0.11 | 0.0080 | 0.0054 | 5,500× |
| Fixed-point residual | 1.7 | 1.8 | 0.028 | 0.0016 | 0.0013 | 1,300× |
| Sampled loss | 9.3 | 6.5 | 4.2 | 3.9 | 4.0 | 2.3× |
- Training approaches the fixed point. decreases by three orders of magnitude, and both KL divergences fall below 0.01.
- The residual tracks sampling accuracy, whereas the sampled loss plateaus. Over all snapshots along training, the Spearman rank correlation (a measure of how consistently two quantities rise and fall together) between and KL() is 0.92. The sampled loss instead plateaus near 4 while and the KL keep decreasing, because , where is the conditional variance of the sampled target, which remains positive at the solution, as for the conditional flow matching (CFM) loss.
The KL divergences along training are shown in this figure, the residual and the sampled loss in this figure, and the generated density in this figure. We will add this study to the revised paper.
Response to Reviewer Qcx5 (3/6)
Q3: Isolating the effect of the conditional-velocity term in the coefficient ablation.
A3: Thank you for this suggestion. We ran the requested variant, and , on h1-run with five seeds. It differs from the default only in ; all other training settings, including the uniform sampling of on , are unchanged. The table reports the final return (average over the last five evaluation checkpoints; mean ± standard deviation over five seeds).
| Coefficients | h1-run |
|---|---|
| Default | 825 ± 5 |
| , | 207 ± 105 |
| , | 8 ± 4 |
The two terms play different roles. Without the conditional-velocity term (second row), training still works but is slower: learning starts about 1.5M environment steps later and is still improving at the end of training, so the final return is 207 instead of 825. Without the singularity cancellation (third row), training breaks and the return never improves. The training curves are shown in this figure. We will add the new variant to Appendix D in the revised paper.
Q4: Direct comparison of the learned action distribution with the Boltzmann distribution of the critic.
A4: Thank you for this suggestion. We added this test on the multi-goal environment [4], a standard benchmark for multimodal policies that is also used by DACER [5] and MaxEntDP [6]. In this environment, a point mass with two-dimensional actions is rewarded for reaching any of four goals. With two-dimensional actions, both densities in the comparison are tractable: the Boltzmann distribution of a fixed critic is normalized numerically on a fine action grid, and the policy density is computed exactly by the change-of-variables formula along the sampling ODE. We train SCF with the hyperparameters of our DMC experiments and five seeds; the reward scale and temperature follow the original example. We then compare the learned policy with the Boltzmann distribution of its critic at the three states visualized in [4], where this distribution is unimodal, four-modal, and bimodal.
| Unimodal state | Four-modal state | Bimodal state | |
|---|---|---|---|
| KL() | 0.004 ± 0.003 | 0.015 ± 0.004 | 0.076 ± 0.029 |
The KL divergence is below 0.1 at all three states, including the multimodal ones. Averaged over the three states, it decreases from 3.2 at initialization to 0.03 at the end of training. The trajectories and the samples of the learned policy at the three states are shown in this figure for one seed. We will add this experiment to the revised paper.
References:
[4] T. Haarnoja, H. Tang, P. Abbeel, and S. Levine. Reinforcement Learning with Deep Energy-Based Policies. ICML, 2017.
[5] Y. Wang, L. Wang, Y. Jiang, W. Zou, T. Liu, X. Song, W. Wang, L. Xiao, J. Wu, J. Duan, and S. E. Li. Diffusion Actor-Critic with Entropy Regulator. NeurIPS, 2024.
[6] X. Dong, J. Cheng, and X. S. Zhang. Maximum Entropy Reinforcement Learning with Diffusion Policy. ICML, 2025.
Response to Reviewer Qcx5 (1/6)
We sincerely thank the reviewer for the time and effort devoted to reviewing our paper and for the constructive comments. We address the questions below.
Q1: Clarification on the entropy term in the critic update.
A1: Thank you for this question. Section 2.3 provides the motivation for the Boltzmann policy. Targeting the Boltzmann distribution of the critic is widely adopted for generative policies (e.g., MaxEntDP, SDAC, DQS, and RFM), and it originates from maximum-entropy RL, notably SAC and its soft policy iteration. All experiments use the critic update in Section 4, that is, the standard Q-function without the entropy term. In practice, we find that dropping this term works well empirically, and we do so for three reasons.
To quantify the effect for SCF, we compare with a soft-Q variant on h1-run and h1-walk. The soft-Q variant adds the log-density term to the critic target, computed with the exact divergence. The table reports the final return (average over the last five evaluation checkpoints; mean ± standard deviation over five seeds) and the time per training step on h1-run.
The soft-Q variant reaches comparable but slightly lower returns, while each training step takes about 37% longer. The training curves are shown in this figure for h1-run and this figure for h1-walk. We will clarify the critic update and add this comparison in the revised paper.
References:
[1] K. Zheng, C. Lu, J. Chen, and J. Zhu. Improved Techniques for Maximum Likelihood Estimation for Diffusion ODEs. ICML, 2023.
[2] Z. Li, S. Tang, and N. Azizan. Reverse Flow Matching: A Unified Framework for Online Reinforcement Learning with Diffusion and Flow Policies. ICML, 2026.
[3] M. Psenka, A. Escontrela, P. Abbeel, and Y. Ma. Learning a Diffusion Model Policy from Rewards via Q-Score Matching. ICML, 2024.