- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes ReFormFlow, an on-policy framework motivated by reverse KL minimization toward an exponentially reward-tilted reference distribution.
Building on the Trajectory Balance condition, it proposes an improvement over DiffusionNFT by directly modeling the partition function .
ReFormFlow outperforms DiffusionNFT in reward optimization speed across several reward fine-tuning tasks.
Critical strengths
-
Substantial wall-clock efficiency gains. ReFormFlow reaches DiffusionNFT’s best single-reward scores with 3.2–4.9× fewer GPU-hours (Fig. 7).
-
Robustness to hyperparameters. Ablations in Figs. 5-6 demonstrate robustness to hyperparameter selection.
Critical weaknesses
-
Disagreement between "Section 4.1" and "Fig. 6(b) & Appendix A.3" The connection between TB and reverse KL hold for relies on eq. (26), which relies on the condition that must be a normalized probability density. However, the method replaces with the velocity-regression surrogate (Appendix A.3). The gradient of this surrogate is not the score of a normalized probability density, thus we cannot guarantee that in the second line of eq. (26). Therefore, an in correct can bias the policy gradient (acknowledged in Appendix A.3). The ablation in Fig. 6(b) also shows that different changes results. I am confused about the authors' stance around whether is necessary.
-
Questionable connection between GFlowNet theory and practical implementation. I am skeptical about the practical utility of GFlowNet theory for large-scale diffusion model fine-tuning, given that recent work [1] had shown that Nabla-GFlowNet [2] can be better explained without GFlowNet theory; , a term that enforces trajectory balance, actually harms performance without adding benefit. Instead, the practical benefits come from additional free hyperparameters. In conjunction with (Weakness 1), I am unsure whether the empirical gains are explained by the work's theory.
[1] https://openreview.net/forum?id=jqsxOdmX88
[2] https://openreview.net/forum?id=Aye5wL6TCn
- Fairness of multi-reward results. Optimizing for GenEval/OCR hurts performance on HPSv2.1, Aesthetic Score, ImageReward, and PickScore, whereas optimizing for PickScore improves performance on those rewards. Thus, in Table 2, "Given two fine-tuned models, the model with -0.01 GenEval and +0.16 PickScore outperforms the other on DrawBench metrics" is not a surprising result. In other words, Table 2 results only show that "ReFormFlow achieved lower GenEval but higher PickScore than DiffusionNFT for this particular multi-reward training protocol", not that "ReFormFlow has les reward hacking on multi-reward training".
Questions for authors
Critical for assessment
-
(Weakness 1) Is necessary or not?
-
(Weakness 2) Can you provide convincing evidence that the gains cannot be explained simply by the introduction of additional scaling factors?
-
(Weakness 3) Can you provide convincing evidence that ReFormFlow converges to a point that is Pareto optimal to DiffusionNFT, rather than simply trading off spatial compositional reasoning for generic human preference?
Clarification
- Table 1 and Figure 3 seem to report different values. For instance, in Table 1, DiffusionNFT reaches OCR in 400 steps, but in Figure 3(c), DiffusionNFT reaches OCR ~ 0.90 in 500 steps.
7 comments
Our Response to Reviewer AzpK (follow-up): Framing without TB, relation to [1, 2], and scope of the contribution
We thank the reviewer for the continued engagement, and for confirming that Q4 and W3/Q3 are resolved.
W1 & W2: TB is not needed; is a per-prompt baseline
A1: Thank you for this reformulation, which gives us the opportunity to clarify our stance. Both points are consistent with our formulation. The on-policy identity follows from the reverse-KL gradient and the score identity alone, so TB is not needed to state it. Under the surrogate, tracks the per-prompt soft value , the expected scaled reward minus the surrogate KL; it coincides with only for exact densities at , which is where our terminology came from. We therefore emphasize that ReFormFlow is reverse-KL minimization toward the reward-tilted target (Eq. 6), with TB contributing only the form of the residual, and that under the surrogate acts as a learned per-prompt soft-value baseline.
W2 & Q2: Relation to [1, 2]
A2: The key difference from [1, 2] lies in which log-ratio is regressed against the reward. [1] applies PPO-clip to the ratio of flow-matching losses to the old policy, linear in the advantage. [2] also targets , but reaches it through proximal steps on the log-ratio to the old policy (PEPG, or its regression variant PAR), with the reference entering through a separate velocity-space KL term. ReFormFlow instead regresses the terminal log-ratio to the fixed reference, which defines (Eq. 6), directly against the scaled reward, so that each sample's deviation from the reference is matched to its own reward. This choice has two consequences that the REINFORCE view makes explicit:
- Baseline. Exact REINFORCE uses a zero-mean score, so any baseline only changes the variance. With the surrogate, , and a baseline shifts the expected update by relative to . In [2], the log-ratio to the old policy vanishes at the sampling policy, so the reward mean is the appropriate centering. In ReFormFlow, moves away from zero as the policy departs from the reference (the mean head outputs in Fig. 9 take both signs), so the centering must track it. The relevant point is therefore not that a baseline helps, but which value it must track; Fig. 6(b), where the learned head outperforms the Monte Carlo , is consistent with this. To test this prediction directly, we replaced by fixed centering , i.e., the group reward mean as in [2], with all other settings identical (PickScore, 500 updates, † DrawBench):
| Centering | GenEval | OCR | PickScore | CLIP† | HPS† | Aes.† | ImgRwd† | PickScore† |
|---|---|---|---|---|---|---|---|---|
| Learned head | 0.55 | 0.64 | 23.96 | 0.274 | 0.309 | 6.31 | 1.34 | 24.14 |
| 0.48 | 0.59 | 23.80 | 0.269 | 0.308 | 6.27 | 1.29 | 24.01 |
The learned head is higher on all eight metrics, with a PickScore gap more than ten times the seed-to-seed standard deviation (0.01; response to Reviewer MRky).
- Aggregation. With the weighted contribution at the -th timestep and their mean, Eqs. 8 and 12 satisfy exactly
[2] recommends a single-timestep estimator and reports results comparable to multi-timestep ones. For a ratio to the old policy, each is typically close to zero within an update, and we expect the variance term to be small, consistent with [2]. For a ratio to the fixed reference, the log-ratio accumulates over training, and so can this term; Fig. 5 isolates it with all other components identical and favors the aggregate on all three tasks.
Informativeness: Scope of the contribution
A3: We clarify our contribution as follows. Within the reverse-KL view, it consists of (i) a terminal residual against the fixed reference in which timestep contributions are aggregated before squaring, with the exact relation to per-step regression above (Fig. 5); the same aggregation also improves DMD in a first stage without reward (response to Reviewer MRky, Q5); (ii) the analysis of which baseline removes the surrogate-induced bias when regressing against a fixed reference, with a learned head that tracks it; and (iii) the empirical results, including Table 1, which you accept as showing an advantage over DiffusionNFT. This positioning is consistent with our response to Reviewer ijzB, where the contribution relative to Diffusion-DPO also rests on the identity above.
Our Response to Reviewer AzpK (follow-up 2): ReFormFlow as one design, the clean KL versus noised KLs, and relation to [2]
We thank the reviewer again for the discussion. The ablation announced in our previous comment is now reported there. Here we clarify how the components of ReFormFlow follow from a single objective, why the objective is defined on the clean terminal sample, and how it relates to [2].
W1 & W2: ReFormFlow as one design
A1: ReFormFlow starts from the reward-tilted target (Eq. 6) and aims to match it directly through on-policy reverse-KL minimization. For flow models, this goal raises two obstacles: the terminal log-density ratio is intractable, and the scalar reward must enter the update of each sample. We address the first by evaluating velocity-regression contributions on forward re-noised endpoints and aggregating them across timesteps into , and the second by a squared residual that regresses each sample's against its own scaled reward. The prompt-dependent offset is the remaining component of this residual, learned by an MLP on prompt embeddings; its mean output stays bounded throughout training on all three tasks (Fig. 9, App. B.2). Aggregation, the residual, and the learned offset thus follow from one objective rather than being separate design choices, and the gradient decomposition in our previous comments describes how they interact under the surrogate.
W2: Why a terminal objective: the clean KL versus noised KLs
A2: The common alternative is to match noised marginals, as in DMD. With the generator distribution , the teacher distribution , and the shared kernel (our convention: clean), Bayes' rule gives for each sample
and hence
DMD optimizes the first term at one sampled timestep and omits the second, which vanishes only as and approaches the full clean KL as . ReFormFlow instead targets the clean terminal ratio and combines contributions from noise levels for each sample before the reward enters. Its surrogate also omits a posterior-gap difference (Eq. 31), but this is a difference of two nonnegative gaps that coincide at initialization: the omitted term is zero there and can become nonzero only as the policy departs from the reference, whereas the posterior KL omitted by DMD is generally nonzero whenever the two distributions differ. In this sense, ReFormFlow is a closer approximation of the clean KL. Empirically, applied to DMD without reward, the same aggregation improves all four metrics (response to Reviewer MRky, Q5), the first step of our ongoing joint DMD–RL work (Sec. 6).
W2 & Q2: Relation to [2]
A3: [2] designs policy-gradient objectives on the log-ratio to the old policy and shows that iterating them converges to ; ReFormFlow starts from and constructs a sample-level matching objective toward it. ReFormFlow is designed for CFG-free deployment (Sec. 1, Sec. 4.3), which avoids the additional unconditional evaluation at every sampling step, and all our results are CFG-free. Without CFG, ReFormFlow reaches 0.98 on both GenEval and OCR and a PickScore of 24.20 (Table 1). Our DiffusionNFT baselines follow its official configurations and reproduce its official performance, which already reaches 0.98 on GenEval and OCR (Table 1); the DiffusionNFT results reported in [2] (0.95 and 0.93) differ notably from these and reflect its specific setting (e.g., CFG at scale 4.5 for GenEval and OCR, its App. D). Our gains are thus measured against DiffusionNFT at its official performance in the general setting; the multi-reward comparison likewise uses its official checkpoint, and AWM and TempFlow-GRPO use their official configurations (Fig. 8 and the 500-update table in our response to Reviewer MRky).
We hope that these results and clarifications, together with the controlled ablations and baseline comparisons, address the remaining concerns, and we would be grateful if they could be taken into account in the final assessment.
I appreciate the authors’ detailed response, and also the honest disclosure of caveats throughout the Appendix and rebuttal.
Resolved concerns. Q4 is resolved: the authors clarified that Table 1 and Fig. 3(c) use different DiffusionNFT configurations (OCR-specific vs. general), whereas ReFormFlow uses a single configuration across tasks. On W3/Q3, I accept that the single-reward comparisons in Table 1 show an empirical advantage over DiffusionNFT.
Remaining concerns. I remain unconvinced about the paper’s informativeness and soundness. My major concerns (W1/Q1, W2/Q2) have not been fully resolved.
Most importantly, the method does not need GFlowNet theory; it is more intuitively and accurately framed as learning a per-prompt baseline. In A2 of their response to my review, the authors explicitly state that GFlowNet theory is not required to explain ReFormFlow. logZϕ does not learn a partition function; it learns a per-prompt baseline. When this baseline is perfectly learned, the update is ∇θVarc(Δ^−r). This is simply on-policy REINFORCE on the reward minus a surrogate KL term, with two modifications: (i) a learned per-prompt value function as the baseline, and (ii) the log-ratio replaced by an ELBO-based surrogate [1, 2]. The on-policy gradient identity can be stated without Trajectory Balance. Therefore, if "TB is used only for the on-policy gradient identity," TB is not needed at all.
With the GFlowNet framing unnecessary, the paper’s technical novelty is very limited. I agree with Reviewer ijzB’s Weakness 1, and with TB no longer necessary, the authors’ A1 to Reviewer ijzB is not satisfactory. The only non-trivial design choice then appears to be aggregating timestep contributions before squaring, supported by ReFormFlow outperforming Per-Step Reg. The finding that learning a per-prompt baseline improves performance is not very interesting. Furthermore, as it stands, the writing does not make it clear that its contribution is limited to this scope.
[1] https://openreview.net/forum?id=eoEmoKoQpJ
[2] https://openreview.net/forum?id=TxWV9mKim9