acceptodds
Under review as a conference paper at ICLR 2027

Distribution-based Regularization for reward fine-tuning diffusion and flow models

11 views

Abstract

Reward fine-tuning is an effective way to adapt pretrained diffusion and flow-matching models to downstream objectives, but aggressive optimization often leads to mode collapse and distributional drift. Existing methods attempt to control this drift with reverse-KL regularization, implemented as an L2 penalty on the velocity field; however, this functions by indirectly constraining sampling dynamics rather than directly acting on the generated distribution. To address this, we propose DFR (Distributional Functional Rewards), which augments the task reward with a distribution-dependent, per-sample reward that acts explicitly on the generated distribution. Formally, we generalize the KL-regularized objective from expected rewards, which are linear in the model's distribution, to convex functionals of it. The first variation of this objective defines an effective, distribution-dependent reward that directly penalizes deviation from the pretrained distribution. This formulation yields additive per-sample rewards for the Fréchet distance (FD), maximum mean discrepancy (MMD), and DreamSim diversity, recovering several recently proposed rewards as special cases. Across various text-to-image fine-tuning tasks, DFR matches the peak reward of KL-tuned baselines while increasing diversity by 11-29% at equal or lower drift.

Primary area:
generative models
Keywords:
diffusion, flow matching, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_wHZwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the reward–diversity–distributional-drift trade-off in reward fine-tuning of diffusion and flow models. It proposes Distributional Functional Rewards (DFR), which supplements task rewards and the usual velocity anchor with penalties defined on the generated distribution. Taking the first variation of these functionals produces distribution-dependent per-sample rewards that can be incorporated into policy-gradient updates. The paper derives instances based on MMD, Fréchet distance, and DreamSim diversity, and implements them within Reinforce Adjoint Matching (RAM).

Experiments fine-tune SD3.5-M on GenEval and OCR and evaluate distributional drift and diversity on task prompts and DrawBench. Against a sweep of RAM’s regularization coefficient, DFR generally improves diversity and/or reduces drift at matched task-reward thresholds. The main contribution is a unified interpretation and practical implementation of distributional regularization, particularly MMD in VAE latent space without an additional feature encoder.

Critical strengths
  1. A useful formulation of the properties that reward fine-tuning should preserve. The paper distinguishes regularizing sampling dynamics from controlling properties of the output distribution. The functional-derivative formulation gives an explicit objective for interpreting several distributional rewards. The derivation of the full-covariance FD reward, its connection to leave-one-out rewards, and the shrinkage correction in Appendix C.6 are useful technical details. Together, these make the framework more informative than an unexplained auxiliary diversity bonus.

  2. The comparison with a regularization sweep provides meaningful empirical evidence. Comparing at matched task reward against several values of RAM’s regularization coefficient addresses an important alternative explanation: whether the same improvement could be obtained by simply strengthening the existing anchor. Three training seeds, common evaluation noise seeds, per-prompt distributional metrics, and evaluation outside the training reward’s feature space strengthen this comparison. For example, at GenEval 0.96, DFR-MMD reduces reported FID from 119.87 to 110.75 and increases DreamSim diversity from 0.174 to 0.196 relative to the successful RAM configuration. Obtaining these improvements without an additional feature encoder is practically attractive.

Critical weaknesses
  1. The advance beyond existing distributional rewards is narrower than the broad framework framing suggests, and is not sufficiently isolated experimentally. Distribution-aware policy fine-tuning already includes MMD-based image-set rewards in Miao et al. (2024), KL-regularized MMD optimization through per-sample score-function coefficients in kCGM (Diamant and Trippe, 2026), and combined instance/distribution rewards in LOO-FID (Baran et al., 2026). The DreamSim instance also recovers the diversity signal used by DRIFT (Liu et al., 2026), as the submission acknowledges.

    The remaining contribution is therefore principally the unified functional interpretation, the analytic FD implementation, and the application of latent-space MMD as a pretrained-distribution regularizer within RAM. These are useful increments, but the experiments primarily establish improvement over KL-only RAM. Although DFR-DS represents an existing diversity signal, the paper does not sufficiently isolate the benefit of the remaining methodological choices against closely related distribution-matching implementations under the same reference distribution and training backend.

  2. The correspondence between the population objective and Algorithm 1 requires further justification. Beyond the acknowledged ELBO and lagged-policy approximations, group centering introduces an additional finite-sample effect. The MMD reward contains a repulsive term, given by a sample's mean distance to the other generated samples, and an attractive term, given by its mean distance to independent reference samples. Even with exact on-policy scores and independent reference samples, subtracting the group mean scales the expected repulsive gradient by (G - 2)/G, but the attractive gradient by (G - 1)/G, where G is the group size. Thus, the relative weighting changes and cannot be corrected by a single rescaling of the distributional-reward coefficient. At the reported group size of 24, these factors are 22/24 and 23/24, respectively.

    This appears fixable, but its effect on the intended distribution-matching objective and reported improvements is uncharacterized. There is also an implementation ambiguity for FD: Section 3.2 states that the functionals are conditional on each prompt, whereas Appendix C.6.3 does not clearly specify whether the moment EMA and reference statistics are maintained separately per prompt. Pooling moments across prompts would define a different objective.

Questions for authors
  1. Contribution relative to prior work: With the reference distribution, feature map/kernel, and RAM backend held fixed, what substantive update differs from existing MMD score-function optimization or FD-based distributional rewards? Where the updates are equivalent, please identify the remaining contribution explicitly. A controlled comparison of the analytic full-covariance FD reward with an adapted LOO-FID reward would help assess its practical significance.

  2. Finite-sample estimator: Can you derive the expected distributional update after the group-centering operation in Algorithm 1, accounting for the dependence of other samples’ pairwise rewards on xix_i? Does correcting this effect, or using a baseline independent of the scored sample, materially change the matched-reward results?

  3. Conditional FD implementation: Are generated moment EMAs and reference moments maintained separately for every prompt? Please specify the effective sample counts and how statistics are initialized and updated. If moments are pooled, how should the conditional-objective statement in Section 3.2 be interpreted?

  4. Independent evaluation: Appendix D.2 uses the task’s full test prompt set to select checkpoints, and the regularization configuration depends on reaching thresholds on this set. Do the main comparisons persist when configuration and checkpoint selection use a separate validation set, followed by evaluation on untouched test prompts? Uncertainty estimates for paired differences would also help assess the smaller improvements beyond the current mean/SD tie heuristic.

4 comments
AuthorsOpenReview ↗

Reply to reviewer wHZw (1/2)

W1 / Q1 (advance over prior distributional rewards). We thank the reviewer for their insightful comments and attention to detail. To clarify, our contribution lies in demonstrating that distributional rewards can be effectively integrated into large-scale flow-model fine-tuning and deliver improvements over state-of-the-art baselines, rather than introducing new reward estimators. We build on established estimators for MMD [3, 4], FD [6], and DreamSim [5], which we explicitly credit in Section 5. With the reference distribution, kernel, and backend fixed, our implementations closely follow these estimators; direct comparisons against essentially identical formulations would therefore not isolate our contribution. The key technical distinction arises for FD: LOO-FID's leave-one-out reward matches ours only to first order (Appendix C.7.2). We directly evaluate this distinction in Appendix E.8, finding no consistent advantage for either variant across the tested configurations (one seed per configuration). Our advance is the practical realization and systematic evaluation of these distributional rewards at scale, including the engineering required to make them effective in flow-model fine-tuning and the empirical demonstration of improvements over state-of-the-art baselines.

Further, our question is a different one: in reward fine-tuning of flow models, does a distributional reward toward the pretrained model add value beyond a tuned KL coefficient? Prior work leaves this open (general response, part 3). The controlled comparison for this question is against KL alone, with everything else fixed. We run it against a full sweep of RAM's β\beta, with three seeds, at matched task reward (Tables 1 and 2). Furthermore, we add additional experiments that show DFR's gains are robust to its new hyperparameters (Appendix E.6). On GenEval, DFR's gains over RAM persist across a range of MMD and FD weights around the main settings (Tables 14 and 15). On OCR, a larger DreamSim weight trades drift for diversity (Table 13), and a larger MMD weight requires a smaller β\beta to reach the target (Table 12).

W2 / Q2 (group centering). We reply to W2 in part 1 of the general response, which we repeat here for convenience:

Group centering (Fy27 Q1, wHZw W2). We thank both reviewers for this derivation. Our implementation already applies the correction. It rescales the centered repulsive and attractive terms by G/(G−2) and G/(G−1), which restores the population update. Algorithm 1 as submitted did not show this step. We have revised Algorithm 1 to point to Appendix C.5, which now gives the derivation for all three instances (Eqs. 65–66). All reported DFR-MMD results use the correction, so no numbers change. For DFR-DS, centering only rescales the expected update by (G−2)/G(G-2)/G, which is absorbed into λ\lambda. For DFR-FD, the reward depends nonlinearly on shared moment estimates, so no rescaling by GG corrects it. We accept this finite-sample bias, similar to how LOO-FID does (Appendix C.5). For increased clarity, we have also updated the manuscript to more accurately describe our implementation.

Q3 (FD statistics). We thank the reviewer for their astute comment. Please note, the FD moments are not maintained separately for each prompt. The MMD and DreamSim rewards are computed within each prompt group, whereas the FD moments are pooled across prompts (Section 3.2). The mean term uses the pooled mean. The covariance term uses the within-prompt covariance: features are centered within each prompt group before pooling, which removes the spread between prompts (Appendix E.8). The generated moments are an EMA with κ\kappa=0.95, warm-started from pretrained-model features and updated once per training step (Appendix C.7.3). Its effective window is about 20 steps of 1,152 samples each. The reference statistics are computed once from 25$\times$1152 pretrained-model samples. The FD objective is therefore not conditional on each prompt: it matches the pooled mean and the average within-prompt covariance, as stated in Appendix E.8.

AuthorsOpenReview ↗

Reply to reviewer wHZw (2/2)

Q4 (independent evaluation and uncertainty). We clarify on held-out evaluation in part 2 of the general response, which we repeat here for convenience:

Held-out evaluation. (Fy27 Q2, wHZw Q4). The DrawBench entries in Tables 1 and 2 already follow the protocol the reviewers describe: DrawBench prompts were used for neither training nor checkpoint selection. For the task-prompt entries, we followed prior work built on the Flow-GRPO codebase [7–9, 12] and evaluated on the prompt lists used for checkpoint selection, which therefore served as a validation set. We have added an untouched test set for the task reward prompts (Appendix E.4). It re-evaluates the selected checkpoints, without reselection, on 256 prompts per task that were used for neither training, the reference samples, nor checkpoint selection. It also reports the achieved task reward (Table 11). All three instances of DFR keep their results on OCR, and DFR-MMD keeps its result at GenEval 0.96. The exceptions are DFR-FD and DFR-DS for GenEval 0.96; they reach 0.953 against RAM's 0.958.

Both β\beta and the checkpoints were selected on the original test lists, which thus served as the validation set. Appendix E.4 evaluates the selected configurations and checkpoints on untouched test prompts, without reselection. For uncertainty, Appendix E.3 adds paired differences between each DFR instance and the RAM baseline at the same target, with 95% bootstrap intervals over prompts, on the task, DrawBench and held-out prompts. The intervals are conditional on the three training runs, and across-seed SDs are reported separately. Among drift and diversity comparisons that do not use an instance's own reward encoder, the intervals favor DFR-MMD in 22 of 27, DFR-FD in 25 of 27, and DFR-DS in 11 of 18. Only one favors RAM: DFR-DS's DrawBench FID on OCR. For the held-out task reward, the interval favors RAM over DFR-FD at GenEval 0.96.

Dear Reviewer, thank you again for your time and constructive feedback. We have addressed your concerns by documenting the group-centering correction already used in our implementation, clarifying the pooled FD objective and statistics, adding an analytic-versus-LOO FD comparison, and evaluating on untouched task prompts with paired confidence intervals. We have also clarified our contribution relative to existing distributional rewards. As the discussion deadline approaches, we would appreciate the opportunity to clarify any remaining questions. If our response resolves your concerns, we would also appreciate an updated assessment. Thank you again.

AuthorsOpenReview ↗

Discussion phase ends in 22 hours

We would like to kindly follow up on our rebuttal and additional experiments. We hope our responses have addressed your concerns and would greatly appreciate any further feedback you may have. Please let us know if any questions remain, as we would be happy to provide additional clarification.

Reviewer_Fy27OpenReview ↗
Rating
3(+1)
Confidence
4(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper augments reward fine-tuning with penalties on the generated distribution. It derives per-sample rewards from the functional derivatives of MMD, feature-space Fréchet distance, and DreamSim diversity, and incorporates them into RAM while retaining its velocity anchor. Experiments on SD3.5 Medium, using GenEval and OCR, compare checkpoints at similar task reward against RAM regularization and existing optimizers.

Critical strengths
  1. The population derivatives in Propositions 1–3, including the FD covariance term, appear correct. The resulting rewards can be incorporated without differentiating through the feature encoders or generated samples.

  2. The RAM anchor-coefficient sweep, three training seeds, and comparisons at similar task reward provide useful evidence beyond an unregularized baseline. The paper also distinguishes evaluation features from those used in its regularization rewards.

Critical weaknesses

1.Algorithm 1, line 6 subtracts the group mean of rewards that depend on other generated samples. Excluding a sample from its own neighbor average does not make this baseline independent of that sample.

In the on-policy, exact-score limit:

\qquad B=\mathbb{E}[\rho(X,Y)s_\theta(X)],$$ where $X,X'\sim p_\theta$, $Y\sim p_{\mathrm{ref}}$ are independent and $s_\theta=\nabla_\theta\log p_\theta$. For the distributional reward alone, with exact reference expectations or an independent reference bank, the expected centered score-function update is $$\frac{G-2}{G}A-\frac{G-1}{G}B,$$ whereas the population construction in Eqs. (16) and (19) requires A-B. The update can therefore be nonzero. The closely related kCGM explicitly addresses this dependence in its baseline construction. 2. kCGM already combines KL-regularized MMD with per-sample score-function coefficients [1]; Appendix C recovers DRIFT's diversity reward up to scaling and the stated implementation differences [2]; and the general effective-reward principle precedes this work [3]. These connections make a controlled empirical distinction particularly important. All DFR results use RAM with SD3.5 Medium, without a comparison against the closest existing reward estimators in the same setup. In addition, Appendix D uses the full test prompt set every ten steps to select checkpoints, so the reported test results include model-selection use. Together, these issues limit the evidence establishing the specific contribution. 3. Declaring a win when a difference exceeds the larger of two seed standard deviations is a descriptive rule, not a significance test. Replace “significantly” or provide uncertainty on paired differences. 4. The Flow-GRPO-relative speedups are extrapolated from RAM's previously reported speedup. Label them as extrapolations and state whether reference generation and FD calibration are included in the cost. [1] Diamant and Trippe. *Calibrating Generative Models to Feature Distributions with MMD Finetuning.* 2026. Section 2 and Appendix A.4. https://arxiv.org/abs/2606.19496 [2] Liu et al. *Beyond the Dirac Delta: Mitigating Diversity Collapse in Reinforcement Fine-Tuning for Versatile Image Generation.* 2026. Section 5.3 and Appendix C. https://arxiv.org/abs/2601.12401 [3] Kumar, Wang, Levy, and Mannor. *Policy Gradient for Reinforcement Learning with General Utilities.* 2022, revised 2023. Theorem 1. https://arxiv.org/abs/2210.00991 [4] Ghosh, Hajishirzi, and Schmidt. *GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment.* 2023. Section 5.1. https://arxiv.org/abs/2310.11513 [5] Saharia et al. *Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding.* 2022. Section 3, DrawBench. https://arxiv.org/abs/2205.11487
Questions for authors
  1. Please derive the expected update after Algorithm 1's group centering. Compare the current estimator with a dependence-aware baseline. With RAM, the reward threshold, and the compute budget fixed, does the distribution-preservation improvement remain?

  2. Please provide the training, checkpoint-selection, and final evaluation prompt lists, their unique counts, and their overlaps. Are these expanded or repeated sets? Please evaluate the selected checkpoints on a disjoint final set and report the achieved task reward.

  3. Are the generated and reference moments in Appendix C maintained separately for each prompt?

7 comments
AuthorsOpenReview ↗

Reply to reviewer Fy27 (1/2)

W1 / Q1 (group centering). We thank the reviewer for their insightful comments. We reply to this in the general response (part 1), which we repeat here for convenience:

Our implementation already applies the correction. It re-scales the centered repulsive and attractive terms by G/(G−2)G/(G−2) and G/(G−1)G/(G−1), which restores the population update. Algorithm 1 as submitted did not clearly show this step. We have revised Algorithm 1 to point to Appendix C.5, which now gives the derivation for all three instances (Eqs. 65–66). All reported DFR-MMD results use the correction, so no numbers change. For DFR-DS, centering only re-scales the expected update by (G−2)/G(G-2)/G, which is absorbed into λ\lambda. For DFR-FD, the reward depends nonlinearly on shared moment estimates, so no rescaling by GG corrects it. We accept this finite-sample bias, similar to how LOO-FID does (Appendix C.5). For increased clarity, we have also updated the manuscript to more accurately describe our implementation.

W2 (closest existing estimators). The reviewer correctly points out similarities to prior work; however, in order to be clear, we state that our main contribution is not a new reward estimator, but a controlled and generalizable study and demonstration of how distributional rewards improve flow-model fine-tuning beyond a tuned KL penalty. We build on established estimators for MMD [3, 4], FD [6], and DreamSim [5], which we explicitly credit in Section 5. Our MMD and FD estimators closely follow existing ones, so a direct comparison would be uninformative. A noteworthy distinction arises only for FD: LOO-FID's leave-one-out reward matches ours only to first order (Appendix C.7.2). We directly evaluate this difference in Appendix E.8, finding no consistent advantage for either variant across the tested configurations.

Our experiments isolate the value of distributional regularization beyond KL. Prior work does not establish whether distributional rewards toward the pre-trained model provide benefits beyond tuning the KL coefficient (general response, part 3b). We address this question through controlled comparisons against KL-only regularization, holding all other factors fixed. Specifically, we evaluate DFR against a full sweep of RAM's β\beta, using three seeds and matching task reward (Tables 1 and 2). We further establish robustness to DFR's hyperparameters (Appendix E.6). On GenEval, DFR consistently improves over RAM across a range of MMD and FD weights around our main settings (Tables 14 and 15). On OCR, increasing the DreamSim weight controls the drift–diversity trade-off (Table 13), while increasing the MMD weight shifts the optimal β\beta downward (Table 12). Together, these results establish the added value and robustness of distributional rewards beyond KL tuning.

The checkpoint-selection concern is addressed in general response (part 2), which we repeat below for convenience:

Held-out evaluation. (Fy27 Q2, wHZw Q4). The DrawBench entries in Tables 1 and 2 already follow the protocol the reviewers describe: DrawBench prompts were used for neither training nor checkpoint selection. For the task-prompt entries, we followed prior work built on the Flow-GRPO codebase [7–9, 12] and evaluated on the prompt lists used for checkpoint selection, which therefore served as a validation set. We have added an untouched test set for the task reward prompts (Appendix E.4). It re-evaluates the selected checkpoints, without reselection, on 256 prompts per task that were used for neither training, the reference samples, nor checkpoint selection. It also reports the achieved task reward (Table 11). All three instances of DFR keep their results on OCR, and DFR-MMD keeps its result at GenEval 0.96. The exceptions are DFR-FD and DFR-DS for GenEval 0.96; they reach 0.953 against RAM's 0.958.

W3 (descriptive rule). We distinguish descriptive win/tie outcomes from statistical significance and explicitly clarify this distinction in Section 4.1, as the reviewer requests. To quantify the reliability of our results, Appendix E.3 reports paired differences between each DFR instance and RAM at matched task reward, with 95% bootstrap confidence intervals over prompts across the task, DrawBench, and held-out evaluations. These intervals condition on the three training runs, with across-seed SDs reported separately.

The confidence intervals provide further evidence of DFR's advantages in drift and diversity. Excluding comparisons that use an instance's own reward encoder, the intervals favour DFR-MMD in 22/27 comparisons, DFR-FD in 25/27, and DFR-DS in 11/18. Only one comparison favours RAM: DrawBench FID for DFR-DS on OCR. For held-out task reward, RAM is favoured over DFR-FD at GenEval 0.96 (Appendix E.3). These results reinforce DFR's improvements across independent evaluation metrics while making the scope of the statistical evidence explicit. We hope this adequately addresses the reviewers comments.

AuthorsOpenReview ↗

Reply to reviewer Fy27 (2/2)

W4 (speedups and cost). To provide further clarity surrounding our speed improvements, Table 1's caption now labels the Flow-GRPO speedups as extrapolated from RAM's reported speedup (Appendix E.2). Each speedup is based on training time, defined as the number of steps to the target times the time per step on the same hardware (Appendix E.2). The time per step includes the overhead of computing the distributional rewards (Appendix D.3). The FD reference statistics and calibration procedure are described in Appendix C.7.3. Reference generation and FD calibration are one-time costs shared across all runs of a task, and are not part of this training time. Appendix D.3 now also discloses the one-off cost of generating the FD reference samples, as a relative estimate to total fine-tuning cost.

Q2 (prompt lists). The lists and their counts are below (Section 4.1, Appendices D.5 and E.4).

ListUseEntriesUnique prompts
GenEval traintraining50,00033,199
GenEval testcheckpoint selection2,212553 (each four times)
GenEval eval subsetTable 1256, from GenEval test256
OCR traintraining19,65319,649
OCR testcheckpoint selection1,0181,018
OCR eval subsetTable 2256, from OCR test256
DrawBench subsetTables 1 and 2256200 (DrawBench has only 200 unique prompts)
Held-out task promptsAppendix E.4256 per task256 per task

The DrawBench subset keeps repeated entries. The paired bootstrap in Appendix E.3 resamples by prompt text, so repeats are not treated as independent prompts. The GenEval training and selection lists share no prompts; the OCR lists share one. DrawBench shares no prompts with either task's lists. The held-out prompts are excluded from training, the reference samples and checkpoint selection. The task evaluation subsets lie inside the selection lists; general response (part 2) discusses this and the held-out evaluation.

Q3 (FD moments). The FD moments are not maintained separately for each prompt. The MMD and DreamSim rewards are computed within each prompt group, whereas the FD moments are pooled across prompts (Section 3.2). The mean term uses the pooled mean. The covariance term uses the within-prompt covariance: features are centered within each prompt group before pooling, which removes the spread between prompts (Appendix E.8). The generated moments are an EMA with κ\kappa=0.95, warm-started from pretrained-model features and updated once per training step (Appendix C.7.3). Its effective window is about 20 steps of 1,152 samples each. The reference statistics are computed once from 25$\times$1152 pretrained-model samples.

Thank you for the response and the additional experiments. Some of my earlier concerns (in Questions and Weakness section) have been addressed, and I appreciate the authors' efforts.

However, I still have a concern regarding the methodological contribution, which i feel it not sufficient. Paper build on RVM, and only experiments with distributional functions to preserve the diversity. The theoretical analysis and results are useful, and diversity is an important avenue in post-training recently. This paper/work does not propose anything new fundamental, i.e. I find the overall methodological contribution rather limited.

Therefore, I would like to keep my original rating for now.

Reviewer_4qPZOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper attempts to address the diversity collapse issue seen when reward fine-tuning diffusion and flow models. The authors propose Distributional Functional Rewards (DFR), a general framework based on replacing the standard expected reward Ex∼p[r(x)]\mathbb{E}_{x \sim p}[r(x)] with a convex functional F(p)\mathcal{F}(p) and optimizing it instead during fine-tuning. The authors theoretically show that, under some mild assumptions, this is equivalent to standard reward fine-tuning with a modified effective reward. They derive this reward for several choices of functionals that better encourage diversity, including MMD, FD, and DS. DFR is evaluated on standard text-to-image benchmarks and compared against simply increasing the KL regularization weight β\beta. The results show that DFR improves diversity while maintaining good reward optimization performance.

Critical strengths

The paper is clearly written and easy to follow. The main idea of generalizing the standard expected reward Ex∼p[r(x)]\mathbb{E}_{x \sim p}[r(x)] to arbitrary convex functionals R(p)R(p) is interesting, and the theoretical derivations seem sound. Additionally, the authors show that several prior approaches to maintaining diversity can be viewed as special cases of DFR, which makes the overall framework quite elegant. The experimental results are also encouraging.

Critical weaknesses

The paper's main contribution appears to be the general framework of DFR, rather than achieving substantially better experimental results compared to prior works. However, the idea of generalizing mean rewards to more general functionals was previously introduced in Flow Density Control [1], where the authors explored it in smaller-scale settings. This seems to be the closest prior work, yet it is neither cited nor discussed in the paper. Clarifying the connections and differences between these two approaches would be very helpful.

Moving on to the experimental results, the benefits of DFR compared to standard KL regularization are clear. However, as mentioned previously, the difference between DFR-MMD and DFR-FD (which is very similar to what has been done in prior works) is quite small and hard to notice. Furthermore, including comparisons against additional baselines, such as training-free alternatives [2, 3], would help better assess how training-based reward augmentation methods like DFR compare against simpler alternatives.


[1] De Santi, Riccardo, et al. "Flow density control: Generative optimization beyond entropy-regularized fine-tuning." Advances in neural information processing systems 38 (2026): 11056-11088.

[2] Sorokin, Dmitrii, et al. "Imagerefl: Balancing quality and diversity in human-aligned diffusion models." arXiv preprint arXiv:2505.22569 (2025).

[3] Jena, Rohit, et al. "Elucidating optimal reward-diversity tradeoffs in text-to-image diffusion models." 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). IEEE, 2025.

Questions for authors
  • Could the authors explain why they chose to use only the largest β\beta value for which all RAM seeds pass the reward threshold, rather than sweeping over different values of β\beta?
  • Does modifying the effective reward through DFR have any impact on training stability or the overall training dynamics?
  • Have the authors tested how DFR's performance changes with different choices of group size GG or sample size MM?
5 comments
AuthorsOpenReview ↗

Reply to reviewer 4qPZ

W1 (FDC). We thank the reviewer for bringing FDC to our attention. We agree that FDC is an example of extending mean rewards to more general functions; as a result, we have added a comparison with FDC in Sections 1, 3 and 5, and Appendix A. The reviewer also points out that although FDC demonstrates these distributional rewards, it proves this approach on small-scale settings, while our proposed work demonstrates (1) scalability to large-scale models; and (2) a targeted recipe that mitigates diversity collapse of SOTA diffusion-RL methods.

W2 (DFR-MMD vs. DFR-FD). We thank the reviewer for their insightful observations, and would like to clarify the core differences between DFR-MMD and DFR-FD. Most importantly, DFR-MMD demonstrates that effective distributional rewards do not require external pre-trained encoders. It operates directly on the flow model's VAE latents using only eight reference samples per prompt (even one suffices; see Q3 below), avoiding the pre-trained encoders, pre-computed statistics, and per-encoder calibration required by DFR-FD, whose reward is first-order equivalent to LOO-FID (Appendix C.7.2). Despite this simplification, DFR-MMD matches DFR-FD on every drift metric across all targets and on GenEval diversity. This comparison is discussed in section 4.2 (Table 1 and 2) of the original submission.

W3 (training-free baselines). To further demonstrate the utility of DFR, we have added both methods suggested by the reviewer [13, 14]. We apply them to RAM and to every DFR instance on GenEval, using one training seed (Section 4.2, Appendix E.9). Both methods use the pre-trained model at low-SNR timesteps. This restores diversity and keeps much of the reward gain when the reward depends on generation at high-SNR timesteps, as non-semantic rewards do [9]. For semantic rewards, generation at low-SNR timesteps largely decides the reward [9, 15], so using the pre-trained model there undoes most of the gain. This is the same observation as we noted on GenEval. With their original schedules, both methods restore diversity, but GenEval falls close to the pretrained model's level. Reversed schedules keep more of the reward but recover little diversity. The suggested methods thus move along the reward–drift–diversity frontier, whereas DFR advances it (Figure 2).

Q1 (choice of β\beta). We would like to emphasize that, in fact, we did sweep β\beta for RAM, over {1, …, 4} on GenEval and {2, …, 6} on OCR, with three seeds each (Section 4.1, Tables 1 and 2). DFR is compared against every RAM configuration that reaches the same target, not only the largest β\beta. Among these, the largest β\beta gives RAM's best drift–diversity trade-off, so it is the strongest KL baseline. As the reviewer correctly points out, comparing with a single β\beta would be an unfair comparison, hence our more comprehensive study. Each DFR default uses the β\beta selected for RAM at the highest target, or the next smaller tested value if it does not converge on every seed (Appendices D.2). The additional comparisons show that DFR’s gains persist at neighbouring MMD and FD weights on Geneval, while larger DreamSim weights increase diversity at the cost of greater drift on OCR (Appendix E.6).

Q2 (training dynamics). To more clearly demonstrate the stability of DFR, we have added per-step gradient norms for RAM and the three DFR instances on GenEval (Appendix E.5, Figure 5). After the first 50 steps, all configurations stay in the same range, and the distributional rewards do not increase the gradient norm relative to RAM at the same β\beta. DFR therefore does not compromise stability. Furthermore, we add additional results showing DFR's robustness to new hyperparameters. Varying the DFR weight around the main settings, with two or three seeds each (Appendix E.6, Tables 12–15), every weight below the tested maximum reaches the target on all seeds, in training time comparable to the main setting. Only larger MMD weights, which add further regularization, can prevent reaching the target. Time required to reach a target is comparable to that of raising β\beta (Appendix E.2).

Q3 (group size GG and sample size MM). We use G=24G=24 for every method in the main comparisons, following the standard setting in prior works [7, 8, 12]; To further understand how DFR behaves across group and sample sizes, we have added two ablations, each with one training seed (Appendix E.7) to sweep over GG and MM. For GG, we compare G∈{24,12}G\in \lbrace 24, 12 \rbrace for RAM and every DFR instance on OCR. At GG=12, DFR-FD keeps lower drift and higher diversity than RAM at the same GG, and DFR-DS keeps its diversity gain, while DFR-MMD falls just short of the target (0.95 vs. 0.96). For MM, we compare DFR-MMD over M∈{8,4,1}M\in \lbrace 8, 4, 1 \rbrace at GenEval 0.96. All three reach the target with comparable drift and diversity, so even one reference sample per prompt suffices in this setting.

Dear Reviewer, thank you again for your time and constructive feedback. We have addressed your concerns by clarifying our contribution relative to FDC, adding the suggested training-free baselines, and providing ablations on training stability, group size, and reference sample size. As the discussion deadline approaches, we would appreciate the opportunity to clarify any remaining questions. If our response resolves your concerns, we would also appreciate an updated assessment. Thank you again.

AuthorsOpenReview ↗

Discussion phase ends in 22 hours

We would like to kindly follow up on our rebuttal and additional experiments. We hope our responses have addressed your concerns and would greatly appreciate any further feedback you may have. Please let us know if any questions remain, as we would be happy to provide additional clarification.

Comments on the paper

6 comments
AuthorsOpenReview ↗

General Response (Overview)

Overview

We thank all reviewers for their constructive reviews. We are encouraged that they found the paper clear (4qPZ, wHZw), its derivations sound (4qPZ, Fy27, wHZw), and that our experiments demonstrate clear gains over a tuned KL coefficient (4qPZ, Fy27, wHZw).

First, we address the three common points raised by reviewers as follows.

  1. Group centering. Reviewers Fy27 (W1/Q1), wHZw (W2/Q2) expressed concerns about missing scaling factors in the MMD reward. We have added the MMD scaling correction in Algorithm 1, where it was previously omitted. Our implementation already included this correction, so all reported results are unchanged.

  2. Held-out evaluation. Reviewers Fy27 (Q2), wHZw (Q4) expressed concerns about overlap of prompts used for checkpoint selection and evaluation. We clarify that our DrawBench evaluation already used prompts excluded from checkpoint selection. We performed additional evaluations on held-out task prompts, with confidence intervals, confirming our original findings on DFR’s drift and diversity benefits over RAM, at every target.

  3. Positioning. Reviewers 4qPZ (W1), Fy27 (W2/Q1), wHZw (W1/Q1) expressed concerns about our specific contribution relative to prior works. Building on prior general-utility RL formulations, we have clarified our contributions as:

    • (a) applying general-utility RL to large scale flow-model fine-tuning with black-box rewards and approximate likelihoods, and
    • (b) demonstrating that regularizing the generated distribution directly enables ELBO-based methods to advance the speed-drift-diversity frontier beyond what path-KL regularization and Flow-GRPO achieve.

    Please refer to the Positioning Table in Part 3a of the General Response for comparisons against prior work.

We detail each point below, and address remaining concerns in the individual responses. Revised text in the PDF is shown in blue.

AuthorsOpenReview ↗

General Response (Parts 1-2)

General response (Part 1): group centering

Group centering (Fy27 W1/Q1, wHZw W2/Q2). We thank both reviewers for this derivation. Our implementation already applies the correction. It rescales the centered repulsive and attractive terms by G/(G−2) and G/(G−1), which restores the population update. Algorithm 1 as submitted did not show this step. We have revised Algorithm 1 to point to Appendix C.5, which now gives the derivation for all three instances . All reported DFR-MMD results use the correction, so no numbers change. For DFR-DS, centering only rescales the expected update by (G−2)/G(G-2)/G, which is absorbed into λ\lambda. For DFR-FD, the reward depends nonlinearly on shared moment estimates, so no rescaling by GG corrects it. We accept this finite-sample bias, similar to how LOO-FID does (Appendix C.5). For increased clarity, we have also updated the manuscript to more accurately describe our implementation.

General response (Part 2): held-out evaluation

Held-out evaluation (Fy27 Q2, wHZw Q4). The DrawBench entries in Tables 1 and 2 already follow the protocol the reviewers describe: DrawBench prompts were used for neither training nor checkpoint selection. For the task-prompt entries, we followed prior work built on the Flow-GRPO codebase [7–9, 12] and evaluated on the prompt lists used for checkpoint selection, which therefore served as a validation set. We have added an untouched test set for the task reward prompts (Appendix E.4). It re-evaluates the selected checkpoints, without reselection, on 256 prompts per task that were used for neither training, the reference samples, nor checkpoint selection. It also reports the achieved task reward (Table 11). All three instances of DFR keep their results on OCR, and DFR-MMD keeps its result at GenEval 0.96. The exceptions are DFR-FD and DFR-DS for GenEval 0.96; they reach 0.953 against RAM's 0.958.

AuthorsOpenReview ↗

General Response (Part 3a)

General response (Part 3a): Application and relationship to prior work on general-utility RL.

We thank reviewers 4qPZ and Fy27 for pointing us to [1, 2]. The revised manuscript credits these works for their formulation and clarifies our application: large-scale flow model fine-tuning with black box rewards and approximate likelihoods.

The table below shows that none of the compared works studies this setting. Positioning

FormulationEmpirical demonstration
General-utility RL formulationPretrained model as target*Approximate likelihoodBlack-box rewardLarge scaleKL sweep
FDC [1]✓✓
kCGM [3]✓✓
Miao et al. [4]
DRIFT [5]✓✓
LOO-FID [6]✓✓
DFR✓✓✓✓✓✓

* A distributional term, beyond KL, regularizes toward the pre-trained model's distribution.

  1. Connecting distributional rewards to the general-utility formulation. kCGM (Δ\Delta) derives per-sample coefficients from the score-function gradient of squared MMD, equivalent to its first variation, without explicitly invoking the general-utility formulation. LOO-FID and Miao et al. motivate their rewards through each sample's leave-one-out contribution to a set-level metric, while DRIFT introduces a diversity bonus. Our work explicitly connects distributional rewards to the general-utility framework.

  2. Regularizing toward the pre-trained model's distribution. We use distributional rewards to preserve the pre-trained model's distribution beyond KL regularization. Among prior methods, only FDC adopts this reference distribution. The others target fixed distributions (kCGM: a feature distribution; Miao et al.: curated images; LOO-FID: real-image statistics) or introduce no reference distribution (DRIFT).

  3. Approximate likelihoods. None of the other works compared here studies distributional rewards optimization using approximate likelihoods. kCGM explicitly excludes flow-matching models without tractable sampling-trajectory probabilities [3, Sec.4]. We use a surrogate building on top of an ELBO-based method [8]. We clarify the resulting policy-gradient approximation and the conditions for baseline subtraction (Section 3.2 and Appendix B).

  4. Scaling to black-box rewards. We demonstrate large-scale flow-model fine-tuning with non-differentiable rewards. While FDC's framework can accommodate a policy-gradient solver, its experiments rely on Adjoint Matching, which requires reward gradients, and do not evaluate black-box rewards. kCGM uses black-box features for distribution matching but does not incorporate an additional task reward [3]. Our approach establishes the practical applicability of distributional regularization in this setting.

  5. Demonstrating large-scale text-to-image fine-tuning. We evaluate distributional rewards beyond single-prompt settings, extending the experimental scope of FDC and Miao et al., whose text-to-image experiments each use a single prompt. FDC uses SD 1.4 with KL regularization in its text-to-image experiment, while its beyond-KL divergence regularization is evaluated in 2D examples. FDC also identifies assessing the concrete benefits of utility and divergence choices in specific applications as future work [1, Sec. 8].

  6. Establishing improvements beyond a fully tuned KL baseline. We systematically sweep the KL coefficient to isolate the benefits of distributional regularization. DRIFT (Δ\Delta) evaluates only three coefficients spaced by factors of 10 (0.1, 0.01, 0.001). LOO-FID does not report a KL-coefficient sweep [6]. Our experiments show that this coarse grid misses stronger KL baselines, highlighting the importance of careful tuning when assessing gains beyond KL.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.