- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the reward–diversity–distributional-drift trade-off in reward fine-tuning of diffusion and flow models. It proposes Distributional Functional Rewards (DFR), which supplements task rewards and the usual velocity anchor with penalties defined on the generated distribution. Taking the first variation of these functionals produces distribution-dependent per-sample rewards that can be incorporated into policy-gradient updates. The paper derives instances based on MMD, Fréchet distance, and DreamSim diversity, and implements them within Reinforce Adjoint Matching (RAM).
Experiments fine-tune SD3.5-M on GenEval and OCR and evaluate distributional drift and diversity on task prompts and DrawBench. Against a sweep of RAM’s regularization coefficient, DFR generally improves diversity and/or reduces drift at matched task-reward thresholds. The main contribution is a unified interpretation and practical implementation of distributional regularization, particularly MMD in VAE latent space without an additional feature encoder.
Critical strengths
-
A useful formulation of the properties that reward fine-tuning should preserve. The paper distinguishes regularizing sampling dynamics from controlling properties of the output distribution. The functional-derivative formulation gives an explicit objective for interpreting several distributional rewards. The derivation of the full-covariance FD reward, its connection to leave-one-out rewards, and the shrinkage correction in Appendix C.6 are useful technical details. Together, these make the framework more informative than an unexplained auxiliary diversity bonus.
-
The comparison with a regularization sweep provides meaningful empirical evidence. Comparing at matched task reward against several values of RAM’s regularization coefficient addresses an important alternative explanation: whether the same improvement could be obtained by simply strengthening the existing anchor. Three training seeds, common evaluation noise seeds, per-prompt distributional metrics, and evaluation outside the training reward’s feature space strengthen this comparison. For example, at GenEval 0.96, DFR-MMD reduces reported FID from 119.87 to 110.75 and increases DreamSim diversity from 0.174 to 0.196 relative to the successful RAM configuration. Obtaining these improvements without an additional feature encoder is practically attractive.
Critical weaknesses
-
The advance beyond existing distributional rewards is narrower than the broad framework framing suggests, and is not sufficiently isolated experimentally. Distribution-aware policy fine-tuning already includes MMD-based image-set rewards in Miao et al. (2024), KL-regularized MMD optimization through per-sample score-function coefficients in kCGM (Diamant and Trippe, 2026), and combined instance/distribution rewards in LOO-FID (Baran et al., 2026). The DreamSim instance also recovers the diversity signal used by DRIFT (Liu et al., 2026), as the submission acknowledges.
The remaining contribution is therefore principally the unified functional interpretation, the analytic FD implementation, and the application of latent-space MMD as a pretrained-distribution regularizer within RAM. These are useful increments, but the experiments primarily establish improvement over KL-only RAM. Although DFR-DS represents an existing diversity signal, the paper does not sufficiently isolate the benefit of the remaining methodological choices against closely related distribution-matching implementations under the same reference distribution and training backend.
-
The correspondence between the population objective and Algorithm 1 requires further justification. Beyond the acknowledged ELBO and lagged-policy approximations, group centering introduces an additional finite-sample effect. The MMD reward contains a repulsive term, given by a sample's mean distance to the other generated samples, and an attractive term, given by its mean distance to independent reference samples. Even with exact on-policy scores and independent reference samples, subtracting the group mean scales the expected repulsive gradient by (G - 2)/G, but the attractive gradient by (G - 1)/G, where G is the group size. Thus, the relative weighting changes and cannot be corrected by a single rescaling of the distributional-reward coefficient. At the reported group size of 24, these factors are 22/24 and 23/24, respectively.
This appears fixable, but its effect on the intended distribution-matching objective and reported improvements is uncharacterized. There is also an implementation ambiguity for FD: Section 3.2 states that the functionals are conditional on each prompt, whereas Appendix C.6.3 does not clearly specify whether the moment EMA and reference statistics are maintained separately per prompt. Pooling moments across prompts would define a different objective.
Questions for authors
-
Contribution relative to prior work: With the reference distribution, feature map/kernel, and RAM backend held fixed, what substantive update differs from existing MMD score-function optimization or FD-based distributional rewards? Where the updates are equivalent, please identify the remaining contribution explicitly. A controlled comparison of the analytic full-covariance FD reward with an adapted LOO-FID reward would help assess its practical significance.
-
Finite-sample estimator: Can you derive the expected distributional update after the group-centering operation in Algorithm 1, accounting for the dependence of other samples’ pairwise rewards on ? Does correcting this effect, or using a baseline independent of the scored sample, materially change the matched-reward results?
-
Conditional FD implementation: Are generated moment EMAs and reference moments maintained separately for every prompt? Please specify the effective sample counts and how statistics are initialized and updated. If moments are pooled, how should the conditional-objective statement in Section 3.2 be interpreted?
-
Independent evaluation: Appendix D.2 uses the task’s full test prompt set to select checkpoints, and the regularization configuration depends on reaching thresholds on this set. Do the main comparisons persist when configuration and checkpoint selection use a separate validation set, followed by evaluation on untouched test prompts? Uncertainty estimates for paired differences would also help assess the smaller improvements beyond the current mean/SD tie heuristic.
4 comments
Reply to reviewer wHZw (2/2)
Q4 (independent evaluation and uncertainty). We clarify on held-out evaluation in part 2 of the general response, which we repeat here for convenience:
Held-out evaluation. (Fy27 Q2, wHZw Q4). The DrawBench entries in Tables 1 and 2 already follow the protocol the reviewers describe: DrawBench prompts were used for neither training nor checkpoint selection. For the task-prompt entries, we followed prior work built on the Flow-GRPO codebase [7–9, 12] and evaluated on the prompt lists used for checkpoint selection, which therefore served as a validation set. We have added an untouched test set for the task reward prompts (Appendix E.4). It re-evaluates the selected checkpoints, without reselection, on 256 prompts per task that were used for neither training, the reference samples, nor checkpoint selection. It also reports the achieved task reward (Table 11). All three instances of DFR keep their results on OCR, and DFR-MMD keeps its result at GenEval 0.96. The exceptions are DFR-FD and DFR-DS for GenEval 0.96; they reach 0.953 against RAM's 0.958.
Both and the checkpoints were selected on the original test lists, which thus served as the validation set. Appendix E.4 evaluates the selected configurations and checkpoints on untouched test prompts, without reselection. For uncertainty, Appendix E.3 adds paired differences between each DFR instance and the RAM baseline at the same target, with 95% bootstrap intervals over prompts, on the task, DrawBench and held-out prompts. The intervals are conditional on the three training runs, and across-seed SDs are reported separately. Among drift and diversity comparisons that do not use an instance's own reward encoder, the intervals favor DFR-MMD in 22 of 27, DFR-FD in 25 of 27, and DFR-DS in 11 of 18. Only one favors RAM: DFR-DS's DrawBench FID on OCR. For the held-out task reward, the interval favors RAM over DFR-FD at GenEval 0.96.
Dear Reviewer, thank you again for your time and constructive feedback. We have addressed your concerns by documenting the group-centering correction already used in our implementation, clarifying the pooled FD objective and statistics, adding an analytic-versus-LOO FD comparison, and evaluating on untouched task prompts with paired confidence intervals. We have also clarified our contribution relative to existing distributional rewards. As the discussion deadline approaches, we would appreciate the opportunity to clarify any remaining questions. If our response resolves your concerns, we would also appreciate an updated assessment. Thank you again.
Discussion phase ends in 22 hours
We would like to kindly follow up on our rebuttal and additional experiments. We hope our responses have addressed your concerns and would greatly appreciate any further feedback you may have. Please let us know if any questions remain, as we would be happy to provide additional clarification.
Reply to reviewer wHZw (1/2)
W1 / Q1 (advance over prior distributional rewards). We thank the reviewer for their insightful comments and attention to detail. To clarify, our contribution lies in demonstrating that distributional rewards can be effectively integrated into large-scale flow-model fine-tuning and deliver improvements over state-of-the-art baselines, rather than introducing new reward estimators. We build on established estimators for MMD [3, 4], FD [6], and DreamSim [5], which we explicitly credit in Section 5. With the reference distribution, kernel, and backend fixed, our implementations closely follow these estimators; direct comparisons against essentially identical formulations would therefore not isolate our contribution. The key technical distinction arises for FD: LOO-FID's leave-one-out reward matches ours only to first order (Appendix C.7.2). We directly evaluate this distinction in Appendix E.8, finding no consistent advantage for either variant across the tested configurations (one seed per configuration). Our advance is the practical realization and systematic evaluation of these distributional rewards at scale, including the engineering required to make them effective in flow-model fine-tuning and the empirical demonstration of improvements over state-of-the-art baselines.
Further, our question is a different one: in reward fine-tuning of flow models, does a distributional reward toward the pretrained model add value beyond a tuned KL coefficient? Prior work leaves this open (general response, part 3). The controlled comparison for this question is against KL alone, with everything else fixed. We run it against a full sweep of RAM's β, with three seeds, at matched task reward (Tables 1 and 2). Furthermore, we add additional experiments that show DFR's gains are robust to its new hyperparameters (Appendix E.6). On GenEval, DFR's gains over RAM persist across a range of MMD and FD weights around the main settings (Tables 14 and 15). On OCR, a larger DreamSim weight trades drift for diversity (Table 13), and a larger MMD weight requires a smaller β to reach the target (Table 12).
W2 / Q2 (group centering). We reply to W2 in part 1 of the general response, which we repeat here for convenience:
Q3 (FD statistics). We thank the reviewer for their astute comment. Please note, the FD moments are not maintained separately for each prompt. The MMD and DreamSim rewards are computed within each prompt group, whereas the FD moments are pooled across prompts (Section 3.2). The mean term uses the pooled mean. The covariance term uses the within-prompt covariance: features are centered within each prompt group before pooling, which removes the spread between prompts (Appendix E.8). The generated moments are an EMA with κ=0.95, warm-started from pretrained-model features and updated once per training step (Appendix C.7.3). Its effective window is about 20 steps of 1,152 samples each. The reference statistics are computed once from 25$\times$1152 pretrained-model samples. The FD objective is therefore not conditional on each prompt: it matches the pooled mean and the average within-prompt covariance, as stated in Appendix E.8.