acceptodds
Under review as a conference paper at ICLR 2027

ReFormFlow: Reforming Flow Reinforcement via Directly Matching Reward Distributions

Abstract

Online reinforcement learning for flow matching is challenged by costly terminal-density evaluation. Recent approaches introduce stochastic transitions for likelihood-ratio updates or use local reward-weighted velocity regression. This motivates a terminal-sample objective that retains deterministic generation while specifying the reward-tilted distribution toward which the policy is optimized. We propose ReFormFlow, an on-policy framework motivated by reverse Kullback–Leibler minimization toward an exponentially reward-tilted reference distribution. An auxiliary path-space analysis motivates a velocity-regression surrogate for the terminal log-density ratio. ReFormFlow aggregates self-normalized regression contributions across re-noised timesteps into a single residual against the scaled, group-centered reward, jointly learning the velocity field and a prompt-conditioned log-partition. Unlike per-step regression, reward supervision applies to the combined contribution. Both training rollouts and inference retain deterministic ODE sampling without classifier-free guidance. Experiments cover single- and multi-reward optimization and out-of-domain evaluation. ReFormFlow is up to faster than DiffusionNFT at matched reward levels in the single-reward comparisons, measured in sampling-and-training GPU-hours. On PickScore, it achieves 24.20 in about 1k updates, compared with 23.82 in 2k updates for DiffusionNFT.

Primary area:
generative models
Keywords:
Generative Models, Reinforcement Learning, Flow Matching

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_AzpKOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes ReFormFlow, an on-policy framework motivated by reverse KL minimization toward an exponentially reward-tilted reference distribution.

Building on the Trajectory Balance condition, it proposes an improvement over DiffusionNFT by directly modeling the partition function ZZ.

ReFormFlow outperforms DiffusionNFT in reward optimization speed across several reward fine-tuning tasks.

Critical strengths
  1. Substantial wall-clock efficiency gains. ReFormFlow reaches DiffusionNFT’s best single-reward scores with 3.2–4.9× fewer GPU-hours (Fig. 7).

  2. Robustness to hyperparameters. Ablations in Figs. 5-6 demonstrate robustness to hyperparameter selection.

Critical weaknesses
  1. Disagreement between "Section 4.1" and "Fig. 6(b) & Appendix A.3" The connection between TB and reverse KL hold for Zϕ≠ZZ_\phi \ne Z relies on eq. (26), which relies on the condition that pθp_\theta must be a normalized probability density. However, the method replaces log⁡pθ\log p_\theta with the velocity-regression surrogate (Appendix A.3). The gradient of this surrogate is not the score of a normalized probability density, thus we cannot guarantee that Epθˉ[sθˉ]=0\mathbb E_{p_{\bar \theta}}[s_{\bar \theta}] = 0 in the second line of eq. (26). Therefore, an in correct ZϕZ_\phi can bias the policy gradient (acknowledged in Appendix A.3). The ablation in Fig. 6(b) also shows that different ZϕZ_\phi changes results. I am confused about the authors' stance around whether Zϕ=ZZ_\phi = Z is necessary.

  2. Questionable connection between GFlowNet theory and practical implementation. I am skeptical about the practical utility of GFlowNet theory for large-scale diffusion model fine-tuning, given that recent work [1] had shown that Nabla-GFlowNet [2] can be better explained without GFlowNet theory; Lbackward\mathcal L_\text{backward}, a term that enforces trajectory balance, actually harms performance without adding benefit. Instead, the practical benefits come from additional free hyperparameters. In conjunction with (Weakness 1), I am unsure whether the empirical gains are explained by the work's theory.

[1] https://openreview.net/forum?id=jqsxOdmX88

[2] https://openreview.net/forum?id=Aye5wL6TCn

  1. Fairness of multi-reward results. Optimizing for GenEval/OCR hurts performance on HPSv2.1, Aesthetic Score, ImageReward, and PickScore, whereas optimizing for PickScore improves performance on those rewards. Thus, in Table 2, "Given two fine-tuned models, the model with -0.01 GenEval and +0.16 PickScore outperforms the other on DrawBench metrics" is not a surprising result. In other words, Table 2 results only show that "ReFormFlow achieved lower GenEval but higher PickScore than DiffusionNFT for this particular multi-reward training protocol", not that "ReFormFlow has les reward hacking on multi-reward training".
Questions for authors

Critical for assessment

  1. (Weakness 1) Is Zϕ=ZZ_\phi = Z necessary or not?

  2. (Weakness 2) Can you provide convincing evidence that the gains cannot be explained simply by the introduction of additional scaling factors?

  3. (Weakness 3) Can you provide convincing evidence that ReFormFlow converges to a point that is Pareto optimal to DiffusionNFT, rather than simply trading off spatial compositional reasoning for generic human preference?

Clarification

  1. Table 1 and Figure 3 seem to report different values. For instance, in Table 1, DiffusionNFT reaches OCR in 400 steps, but in Figure 3(c), DiffusionNFT reaches OCR ~ 0.90 in 500 steps.
7 comments

I appreciate the authors’ detailed response, and also the honest disclosure of caveats throughout the Appendix and rebuttal.

Resolved concerns. Q4 is resolved: the authors clarified that Table 1 and Fig. 3(c) use different DiffusionNFT configurations (OCR-specific vs. general), whereas ReFormFlow uses a single configuration across tasks. On W3/Q3, I accept that the single-reward comparisons in Table 1 show an empirical advantage over DiffusionNFT.

Remaining concerns. I remain unconvinced about the paper’s informativeness and soundness. My major concerns (W1/Q1, W2/Q2) have not been fully resolved.

Most importantly, the method does not need GFlowNet theory; it is more intuitively and accurately framed as learning a per-prompt baseline. In A2 of their response to my review, the authors explicitly state that GFlowNet theory is not required to explain ReFormFlow. log⁡Zϕ\log Z_\phi does not learn a partition function; it learns a per-prompt baseline. When this baseline is perfectly learned, the update is ∇θVarc(Δ^−r)\nabla_\theta \mathrm{Var}_c(\hat\Delta - r). This is simply on-policy REINFORCE on the reward minus a surrogate KL term, with two modifications: (i) a learned per-prompt value function as the baseline, and (ii) the log-ratio replaced by an ELBO-based surrogate [1, 2]. The on-policy gradient identity can be stated without Trajectory Balance. Therefore, if "TB is used only for the on-policy gradient identity," TB is not needed at all.

With the GFlowNet framing unnecessary, the paper’s technical novelty is very limited. I agree with Reviewer ijzB’s Weakness 1, and with TB no longer necessary, the authors’ A1 to Reviewer ijzB is not satisfactory. The only non-trivial design choice then appears to be aggregating timestep contributions before squaring, supported by ReFormFlow outperforming Per-Step Reg. The finding that learning a per-prompt baseline improves performance is not very interesting. Furthermore, as it stands, the writing does not make it clear that its contribution is limited to this scope.

[1] https://openreview.net/forum?id=eoEmoKoQpJ

[2] https://openreview.net/forum?id=TxWV9mKim9

AuthorsOpenReview ↗

Our Response to Reviewer AzpK (follow-up): Framing without TB, relation to [1, 2], and scope of the contribution

We thank the reviewer for the continued engagement, and for confirming that Q4 and W3/Q3 are resolved.

W1 & W2: TB is not needed; log⁡Zϕ\log Z_\phi is a per-prompt baseline

A1: Thank you for this reformulation, which gives us the opportunity to clarify our stance. Both points are consistent with our formulation. The on-policy identity follows from the reverse-KL gradient and the score identity alone, so TB is not needed to state it. Under the surrogate, log⁡Zϕ\log Z_\phi tracks the per-prompt soft value ℓ∗(c)=Ec[r−Δ^]\ell^*(c) = \mathbb{E}_c[r - \hat\Delta], the expected scaled reward minus the surrogate KL; it coincides with log⁡Z(c)\log Z(c) only for exact densities at pθ=p∗p_\theta = p^*, which is where our terminology came from. We therefore emphasize that ReFormFlow is reverse-KL minimization toward the reward-tilted target (Eq. 6), with TB contributing only the form of the residual, and that under the surrogate log⁡Zϕ\log Z_\phi acts as a learned per-prompt soft-value baseline.

W2 & Q2: Relation to [1, 2]

A2: The key difference from [1, 2] lies in which log-ratio is regressed against the reward. [1] applies PPO-clip to the ratio of flow-matching losses to the old policy, linear in the advantage. [2] also targets p∗p^*, but reaches it through proximal steps on the log-ratio to the old policy (PEPG, or its regression variant PAR), with the reference entering through a separate velocity-space KL term. ReFormFlow instead regresses the terminal log-ratio to the fixed reference, which defines p∗p^* (Eq. 6), directly against the scaled reward, so that each sample's deviation from the reference is matched to its own reward. This choice has two consequences that the REINFORCE view makes explicit:

  • Baseline. Exact REINFORCE uses a zero-mean score, so any baseline only changes the variance. With the surrogate, Ec[∇θΔ^]≠0\mathbb{E}_c[\nabla_\theta\hat\Delta] \neq 0, and a baseline ℓ\ell shifts the expected update by 2(ℓ−ℓ∗),Ec[∇θΔ^]2(\ell - \ell^*)\\,\mathbb{E}_c[\nabla_\theta\hat\Delta] relative to ℓ=ℓ∗\ell = \ell^*. In [2], the log-ratio to the old policy vanishes at the sampling policy, so the reward mean is the appropriate centering. In ReFormFlow, Ec[Δ^]\mathbb{E}_c[\hat\Delta] moves away from zero as the policy departs from the reference (the mean head outputs in Fig. 9 take both signs), so the centering must track it. The relevant point is therefore not that a baseline helps, but which value it must track; Fig. 6(b), where the learned head outperforms the Monte Carlo log⁡Z\log Z, is consistent with this. To test this prediction directly, we replaced log⁡Zϕ\log Z_\phi by fixed centering ℓ=0\ell = 0, i.e., the group reward mean as in [2], with all other settings identical (PickScore, 500 updates, † DrawBench):
CenteringGenEvalOCRPickScoreCLIP†HPS†Aes.†ImgRwd†PickScore†
Learned head0.550.6423.960.2740.3096.311.3424.14
ℓ=0\ell = 00.480.5923.800.2690.3086.271.2924.01

The learned head is higher on all eight metrics, with a PickScore gap more than ten times the seed-to-seed standard deviation (0.01; response to Reviewer MRky).

  • Aggregation. With dkd_k the weighted contribution at the kk-th timestep and dˉ\bar d their mean, Eqs. 8 and 12 satisfy exactly

LPer-Step=LReFormFlow+E[1K∑k(dk−dˉ)2].\mathcal{L}_{\text{Per-Step}} = \mathcal{L}_{\text{ReFormFlow}} + \mathbb{E}\Bigl[\tfrac{1}{K}\textstyle\sum_{k}(d_k - \bar d)^2\Bigr].

[2] recommends a single-timestep estimator and reports results comparable to multi-timestep ones. For a ratio to the old policy, each dkd_k is typically close to zero within an update, and we expect the variance term to be small, consistent with [2]. For a ratio to the fixed reference, the log-ratio accumulates over training, and so can this term; Fig. 5 isolates it with all other components identical and favors the aggregate on all three tasks.

Informativeness: Scope of the contribution

A3: We clarify our contribution as follows. Within the reverse-KL view, it consists of (i) a terminal residual against the fixed reference in which timestep contributions are aggregated before squaring, with the exact relation to per-step regression above (Fig. 5); the same aggregation also improves DMD in a first stage without reward (response to Reviewer MRky, Q5); (ii) the analysis of which baseline removes the surrogate-induced bias when regressing against a fixed reference, with a learned head that tracks it; and (iii) the empirical results, including Table 1, which you accept as showing an advantage over DiffusionNFT. This positioning is consistent with our response to Reviewer ijzB, where the contribution relative to Diffusion-DPO also rests on the identity above.

AuthorsOpenReview ↗

Our Response to Reviewer AzpK (follow-up 2): ReFormFlow as one design, the clean KL versus noised KLs, and relation to [2]

We thank the reviewer again for the discussion. The ℓ=0\ell = 0 ablation announced in our previous comment is now reported there. Here we clarify how the components of ReFormFlow follow from a single objective, why the objective is defined on the clean terminal sample, and how it relates to [2].

W1 & W2: ReFormFlow as one design

A1: ReFormFlow starts from the reward-tilted target p∗∝pref,eR/γp^* \propto p_{\mathrm{ref}}\\, e^{R/\gamma} (Eq. 6) and aims to match it directly through on-policy reverse-KL minimization. For flow models, this goal raises two obstacles: the terminal log-density ratio log⁡pθ(x1∣c)/pref(x1∣c)\log p_\theta(x_1 \mid c) / p_{\mathrm{ref}}(x_1 \mid c) is intractable, and the scalar reward must enter the update of each sample. We address the first by evaluating velocity-regression contributions on forward re-noised endpoints and aggregating them across timesteps into Δ^\hat\Delta, and the second by a squared residual that regresses each sample's Δ^\hat\Delta against its own scaled reward. The prompt-dependent offset log⁡Zϕ\log Z_\phi is the remaining component of this residual, learned by an MLP on prompt embeddings; its mean output stays bounded throughout training on all three tasks (Fig. 9, App. B.2). Aggregation, the residual, and the learned offset thus follow from one objective rather than being separate design choices, and the gradient decomposition in our previous comments describes how they interact under the surrogate.

W2: Why a terminal objective: the clean KL versus noised KLs

A2: The common alternative is to match noised marginals, as in DMD. With the generator distribution pfakep_{\mathrm{fake}}, the teacher distribution prealp_{\mathrm{real}}, and the shared kernel qt(xt∣x1)=N(tx1,(1−t)2I)q_t(x_t \mid x_1) = \mathcal{N}(t x_1, (1-t)^2 I) (our convention: t=1t = 1 clean), Bayes' rule gives for each sample

log⁡pfake(x1)preal(x1)=log⁡pt,fake(xt)pt,real(xt)+log⁡pfake(x1∣xt)preal(x1∣xt),\log\frac{p_{\mathrm{fake}}(x_1)}{p_{\mathrm{real}}(x_1)} = \log\frac{p_{t,\mathrm{fake}}(x_t)}{p_{t,\mathrm{real}}(x_t)} + \log\frac{p_{\mathrm{fake}}(x_1 \mid x_t)}{p_{\mathrm{real}}(x_1 \mid x_t)},

and hence

DKL(pfake∣preal)=DKL(pt,fake∣pt,real)+Ext∼pt,fake[DKL(pfake(⋅∣xt)∣preal(⋅∣xt))].D_{\mathrm{KL}}(p_{\mathrm{fake}} \\| p_{\mathrm{real}}) = D_{\mathrm{KL}}(p_{t,\mathrm{fake}} \\| p_{t,\mathrm{real}}) + \mathbb{E}_{x_t \sim p_{t,\mathrm{fake}}}\bigl[D_{\mathrm{KL}}(p_{\mathrm{fake}}(\cdot \mid x_t) \\| p_{\mathrm{real}}(\cdot \mid x_t))\bigr].

DMD optimizes the first term at one sampled timestep and omits the second, which vanishes only as t→1t \to 1 and approaches the full clean KL as t→0t \to 0. ReFormFlow instead targets the clean terminal ratio and combines contributions from KK noise levels for each sample before the reward enters. Its surrogate also omits a posterior-gap difference (Eq. 31), but this is a difference of two nonnegative gaps that coincide at initialization: the omitted term is zero there and can become nonzero only as the policy departs from the reference, whereas the posterior KL omitted by DMD is generally nonzero whenever the two distributions differ. In this sense, ReFormFlow is a closer approximation of the clean KL. Empirically, applied to DMD without reward, the same aggregation improves all four metrics (response to Reviewer MRky, Q5), the first step of our ongoing joint DMD–RL work (Sec. 6).

W2 & Q2: Relation to [2]

A3: [2] designs policy-gradient objectives on the log-ratio to the old policy and shows that iterating them converges to p∗p^*; ReFormFlow starts from p∗p^* and constructs a sample-level matching objective toward it. ReFormFlow is designed for CFG-free deployment (Sec. 1, Sec. 4.3), which avoids the additional unconditional evaluation at every sampling step, and all our results are CFG-free. Without CFG, ReFormFlow reaches 0.98 on both GenEval and OCR and a PickScore of 24.20 (Table 1). Our DiffusionNFT baselines follow its official configurations and reproduce its official performance, which already reaches 0.98 on GenEval and OCR (Table 1); the DiffusionNFT results reported in [2] (0.95 and 0.93) differ notably from these and reflect its specific setting (e.g., CFG at scale 4.5 for GenEval and OCR, its App. D). Our gains are thus measured against DiffusionNFT at its official performance in the general setting; the multi-reward comparison likewise uses its official checkpoint, and AWM and TempFlow-GRPO use their official configurations (Fig. 8 and the 500-update table in our response to Reviewer MRky).

We hope that these results and clarifications, together with the controlled ablations and baseline comparisons, address the remaining concerns, and we would be grateful if they could be taken into account in the final assessment.

Reviewer_ijzBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ReFormFlow, an online flow RL algorithm based on matching the policy distribution to the tilted energy-based distribution specified by the reward function. To address the issue that in flow policy training obtaining the terminal probability ratio is prohibitively expensive, the paper introduces a velocity regression surrogate that aggregates over different flow timesteps. Experiments on guided image generation are conducted on both single-reward and multi-reward settings. The proposed ReFormFlow shows superior performance compared to baselines such as DiffusionNFT.

Critical strengths
  1. The proposed algorithm is supported by theoretical derivations and the experimental setup is also discussed in detail.
  2. The paper is clearly written and well-organized.
Critical weaknesses
  1. The methodological novelty of this paper is limited. The main novelty of the derivation and algorithm is in using the weighted velocity regression in Equation 11 to approximate the density ratio, while the other parts of the derivation are already covered in prior works including TB [1] and Diffusion-DPO [2]. However, the derivation for this relationship in Appendix A.3 is not rigorous and is more theory-inspired than formally derived.
  2. Following the previous point, in the "Local Gaussian Transition Calculation" paragraph of Appendix A.3, the derivation from the local Gaussian relationship to Equation 11 is unclear and not rigorous. In addition, the term Gθ−GrefG_\theta-G_{\mathrm{ref}} is omitted without much analysis.
  3. The main results in Tables 1 and 2 only include DiffusionNFT, FlowGRPO and CFG as baselines. More diffusion/flow finetuning baselines should be included in the main results for a complete evaluation.

[1] Whitammer, Esmeralda S., et al. "Trajectory balance: Improved credit assignment in gflownets." Advances in neural information processing systems 35 (2022): 5955-5967. [2] Wallace, Bram, et al. "Diffusion model alignment using direct preference optimization." 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024.

Questions for authors
  1. In Line 428, the paper mentions that the DiffusionNFT results come from the evaluation of the official checkpoints. Is the DiffusionNFT model also trained with the sequential training used for ReFormFlow in Section 5.3?
  2. Minor issues. The citation format should be corrected (should use \citep in many cases).
Reviewer_MRkyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ReFormFlow is an on-policy framework for reward fine-tuning of flow matching models without explicit terminal-density evaluation. It starts from the reward-tilted optimum p∗(x1∣c)∝pref(x1∣c)exp⁡(R/γ)p^*(x_1|c) \propto p_{ref}(x_1|c)\exp(R/\gamma) and uses a trajectory-balance-style squared residual whose expected on-policy semi-gradient matches the reverse-KL gradient when exact terminal densities are available (Eq. 7). In practice, the log-density ratio is replaced by an aggregate of self-normalized velocity-error differences across KK re-noised timesteps, regressed against the scaled, group-centered reward together with a learned log⁡Zϕ(c)\log Z_\phi(c) (Eq. 12). Training and inference use deterministic ODE sampling without CFG. On SD3.5-Medium with LoRA, the paper reports faster reward convergence than DiffusionNFT, up to a 4.9× speedup in sampling-and-training GPU-hours, and favorable multi-reward and DrawBench results.

Critical strengths
  1. A clear terminal-level formulation with the key ablation. Specifying the reward-tilted distribution first is a conceptually appealing alternative to locally defined velocity-mixing updates. Figure 5 directly compares the central design choice—aggregating timestep contributions before squaring the reward residual (Eq. 12) versus forming one residual per timestep (Eq. 8)—with the aggregated version reporting better results on all three tasks.

  2. Consistent empirical behavior and candid reporting of limitations. ReFormFlow reaches matched or higher rewards than DiffusionNFT across the reported single-reward tasks, and its multi-reward checkpoint is competitive on most metrics. The appendix clearly discloses that its per-update cost is 3–11% higher than DiffusionNFT's, distinguishes the two DiffusionNFT OCR configurations, and acknowledges that the density-ratio approximations have no provided error bounds and do not inherit the exact-density reverse-KL identity.

Critical weaknesses
  1. The novelty and efficiency advantage over the closest prior work are not yet established convincingly. The practical method combines the ODE rollout and re-noising structure of DiffusionNFT, DMD-style velocity-error differences and weighting, and a trajectory-balance residual. Its main algorithmic distinction is the aggregate-before-squaring update, whose ablation gain is modest (e.g., PickScore 24.20 vs. 23.98). AWM (Xue et al., 2025a) already treats the terminal sample as a sequence-level action and estimates its likelihood through a Monte Carlo expectation of re-noised flow-matching losses, with group-relative advantages and velocity-space regularization. ReFormFlow instead matches an aggregate surrogate to the reward through a squared residual and learned partition, but this distinction is not isolated in a controlled comparison: Figure 8 provides curves for AWM and TempFlow-GRPO without tabulated endpoints or sufficiently documented matched protocols. The headline OCR speedup is also favorable to ReFormFlow because Figure 7 uses DiffusionNFT's weaker general configuration. Using Appendix B.1's per-update costs and Table 4's 300 versus 400 updates to reach OCR 0.98, the OCR-specific comparison appears to imply only about a 1.25× speedup rather than 4.9×.

  2. The gap between the motivating theory and the implemented objective leaves the central interpretation under-supported. The reverse-KL identity holds only for exact terminal densities. The practical surrogate additionally assumes an unestablished correspondence between the auxiliary SDE and ODE marginals, approximates endpoint-conditioned paths, drops Gθ−GrefG_\theta-G_{ref}, replaces λt\lambda_t by empirical wtw_t, and places a finite-KK estimate inside a squared residual. These discrepancies have no provided error bounds, so the distribution targeted by the practical update remains theoretically uncharacterized. There is no direct validation against exact Jacobian-trace density ratios on a tractable model, and neither KK nor the timestep-sampling protocol is reported. Empirically, ReFormFlow is trained only on SD3.5-Medium at one resolution with LoRA; the number of runs is unreported, there are no uncertainty estimates, and no systematic diversity, human, or reward-overoptimization evaluation is provided. This is important because the GenEval/OCR checkpoints already reduce aesthetic score relative to the CFG-free base model.

Questions for authors
  1. Could you report tabulated AWM and TempFlow-GRPO results under the same rollout, evaluation, and GPU-hour protocol? In particular, what empirical advantage does the squared trajectory-balance residual provide over AWM's terminal-sample, advantage-weighted formulation?

  2. On a small model with tractable Jacobian-trace densities, how well calibrated and correlated is the aggregate surrogate with the true log pθ(x1∣c)/pref(x1∣c)p_\theta(x_1|c)/p_{ref}(x_1|c)? What KK and timestep distribution are used, and how sensitive are results to them?

  3. From the reported per-update costs and Table 4, the matched OCR 0.98 comparison appears to give roughly a 1.25× speedup against DiffusionNFT's OCR-specific configuration, rather than 4.9×. Can you confirm this and report sensitivity to checkpoint spacing and baseline hyperparameters?

  4. Do the conclusions hold on a second backbone such as FLUX? Diversity metrics, human evaluation, or training-time trajectories of held-out metrics would also help assess reward overoptimization.

  5. Given the DMD-inspired velocity-error surrogate, can ReFormFlow be combined with DMD for joint few-step distillation and reward optimization? What changes would be needed to coordinate teacher-distribution matching with the reward residual, timestep aggregation, and on-policy sampling, and is there preliminary evidence that the two objectives are compatible?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Our General Response: Starting Point, Significance, and Additional Results

We thank all reviewers for their constructive feedback and for recognizing the efficiency gains and robustness to hyperparameters (AzpK); the terminal-level formulation, the key aggregation ablation, and the candid reporting of limitations (MRky); and the theoretical derivations and detailed setup (ijzB). Below, we summarize the starting point and significance of ReFormFlow and the additional results.

Starting point

Since rewards evaluate the clean sample, ReFormFlow starts from the reward-tilted target p∗∝pref,eR/γp^* \propto p_{\mathrm{ref}}\\, e^{R/\gamma} (Eq. 6) and matches it directly through on-policy reverse-KL minimization on the clean terminal sample, while retaining deterministic ODE rollouts and CFG-free inference, whereas existing flow RL methods define local updates without specifying the terminal log-ratio each sample should satisfy. This raises two obstacles: the terminal log-ratio is intractable, and the scalar reward must enter each sample's update. ReFormFlow (i) aggregates velocity-regression contributions across re-noised timesteps (KK per sample) into a surrogate Δ^\hat\Delta, (ii) regresses each sample's deviation from the reference against its own scaled reward in a squared residual, and (iii) learns the offset log⁡Zϕ\log Z_\phi, which under the surrogate acts as a per-prompt soft-value baseline. These components follow from one objective rather than separate design choices; TB contributes only the form of the residual (AzpK, ijzB).

Significance

  • Closer to the clean KL (AzpK, ijzB). Matching noised marginals, as in DMD, relies on DKL(pfake∣preal)=DKL(pt,fake∣pt,real)+Ext∼pt,fake[DKL(pfake(⋅∣xt)∣preal(⋅∣xt))]D_{\mathrm{KL}}(p_{\mathrm{fake}} \\| p_{\mathrm{real}}) = D_{\mathrm{KL}}(p_{t,\mathrm{fake}} \\| p_{t,\mathrm{real}}) + \mathbb{E}_{x_t \sim p_{t,\mathrm{fake}}}\bigl[D_{\mathrm{KL}}(p_{\mathrm{fake}}(\cdot \mid x_t) \\| p_{\mathrm{real}}(\cdot \mid x_t))\bigr] and omits the posterior term, generally nonzero when the two distributions differ. ReFormFlow combines these KK noise levels for each sample before the reward enters, and the posterior-gap difference it omits (Eq. 31) is zero at initialization; in this sense, it is a closer approximation of the clean KL.
  • Aggregation acts through an exact variance term (MRky, ijzB, AzpK). With dkd_k the contribution at the kk-th timestep, LPer-Step=LReFormFlow+E[1K∑k(dk−dˉ)2]\mathcal{L}_{\text{Per-Step}} = \mathcal{L}_{\text{ReFormFlow}} + \mathbb{E}[\frac{1}{K}\sum_k(d_k - \bar d)^2]. Per-step regression, corresponding to the Jensen step of Diffusion-DPO, requires each timestep to explain the full reward; for an objective linear in the advantage, as in AWM, aggregation would make no difference. Fig. 5 isolates this term and favors the aggregate on all three tasks.
  • Which baseline must be tracked (AzpK). Under the surrogate, a baseline ℓ\ell shifts the expected update by 2(ℓ−ℓ∗),Ec[∇θΔ^]2(\ell - \ell^*)\\,\mathbb{E}_c[\nabla_\theta\hat\Delta], where ℓ∗=Ec[r−Δ^]\ell^* = \mathbb{E}_c[r - \hat\Delta] moves away from zero during training, unlike log-ratios to the old policy in the works cited by Reviewer AzpK. The learned head tracks ℓ∗\ell^*, and fixed ℓ=0\ell = 0 lowers all eight metrics, with a PickScore gap more than ten times the seed-to-seed standard deviation.

Additional results

All runs share one 8×H200 node; baselines use official configurations.

ExperimentReviewersKey result
500-update comparison with DiffusionNFT, AWM, TempFlow-GRPOMRky, ijzBPickScore task: highest on 7 of 8 metrics; GenEval task: above AWM on 6 with fewer GPU-hours
Three training seeds (PickScore)MRky23.97 ± 0.01, an order of magnitude below method gaps
Held-out metrics during trainingMRkyimprove with the target on PickScore; aesthetic score declines only as GenEval saturates
FLUX.1-Dev, GenEvalMRky, ijzB0.96 after 700 iterations vs. 1000 for DiffusionNFT
Timestep aggregation in DMD (no reward)MRky, ijzB, AzpKall four metrics above DMD, about 1.1× its training time
Fixed centering ℓ=0\ell = 0 vs. learned headAzpKlearned head higher on all 8 metrics

Shared components are identical: wtw_t, group-relative rewards, and the re-noising timesteps follow DiffusionNFT without tuning, and all single-reward runs share Table 3. The multi-reward comparison follows DiffusionNFT's protocol exactly (ijzB, AzpK), and our DiffusionNFT baselines reproduce its official performance (Table 1); for OCR, we report both of its configurations (approximately 1.25×1.25\times against the OCR-specific one; MRky).

Discussion status

Reviewer AzpK confirmed that Q4 is resolved and that Table 1 shows an empirical advantage over DiffusionNFT (W3/Q3); our follow-up comments address the remaining concerns on W1/W2. We remain available to answer any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.