- Rating
- 3(+2)
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes SERF, a first-order method for reward fine-tuning flow and diffusion models with differentiable rewards. The reward gradient is computed once at each sample, the sample is independently re-noised with the standard Gaussian kernel, and the model velocity at the noisy state is regressed toward the reference velocity shifted by half the endpoint reward gradient. The method is motivated from the memoryless SOC formulation of Adjoint Matching by replacing the path Jacobian with the identity; the paper argues this surrogate is positively aligned with the true value gradient and that independent re-noising has lower gradient variance than trajectory states . On SD3.5-Medium with PickScore and HPSv2.1, SERF matches or beats DRaFT-LV, VGG-Flow, Adjoint Matching, and RSM at matched reward on FID and CLIPScore, trains roughly 10× faster than Adjoint Matching, and remains effective with 5-step rollouts.
Critical strengths
-
The method is simple to implement, requires no trajectory storage or backpropagation through sampling, is agnostic to the endpoint sampler, and decouples the number of training states from the number of sampling steps. These are practically attractive properties.
-
The exposition of the derivation chain (SOC objective → reward/regularization decomposition → identity-Jacobian surrogate → re-noising) is clear and easy to follow, and the conclusion is candid that the identity-Jacobian approximation is the source of the method's simplicity.
-
Prop. 2 gives a concrete formal reason to prefer independent re-noising over trajectory states for this family of targets, and the paper correctly points out that ODE trajectories produce a degenerate (deterministic) coupling .
-
The empirical comparison covers the relevant first-order baselines on a modern flow-matching model, reports trade-off curves rather than single points, includes qualitative evidence of reward-hacking artifacts in baselines (human-like faces in VGG-Flow/RSM), and includes ablations on training CFG, rollout length, timestep sampling, and target weighting.
-
The 5-step rollout result and the training-time curves (Fig. 4) are useful practical findings.
Critical weaknesses
W1. Limited novelty relative to RSM and prior work. The regression target "reference velocity plus a scaled clean-sample reward gradient" is already the first-order redesign proposed in RSM [1], which replaces local Tweedie guidance with the reward gradient at the clean sample, describes it as an SDS-style reward score distillation, and explicitly notes that the exact quantity would require the full Jacobian chain. The identity-Jacobian approximation is borrowed from the identity pullback of ReFL [3] and from [4, 5]; endpoint re-noising is used in DRaFT-LV [6], DiffusionNFT [7], RAM [8], and EAM [9]; Theorems 1–2 are standard results already present in Adjoint Matching [2]. What remains specific to SERF is (i) training on independently re-noised states rather than trajectory states, (ii) a constant time weight, and (iii) the factor one-half. Items (ii)–(iii) are coefficient-level choices within the design space of [1].
W2. The one distinguishing design choice is not ablated. There is no same-codebase experiment that switches only "trajectory states vs. independently re-noised states", and no empirical gradient-variance measurement. Consequently the gap between SERF and RSM in Tables 1–2 cannot be attributed to re-noising rather than to the time weighting, the one-half factor, the timestep distribution, or tuning. The claimed advantage of decoupling targets from rollout steps (four targets per endpoint) is never varied either.
W3. The theoretical justification does not match the algorithm actually run, and the abstract overclaims.
- The endpoint-tilted optimum, Eq. (4), and the claim that the Gaussian re-noising kernel is exact at the optimum all rely on the KL-consistent time weighting (inverse squared noise coefficient). The algorithm uses a constant weight, under which the optimum is no longer the reward-tilted data distribution and Eq. (4) no longer holds.
- The re-noising kernel and the drift-to-velocity conversion are justified as exact "at initialization and at the SOC optimum", but SERF does not converge to the SOC optimum because of the identity-Jacobian approximation. The fixed point of the SERF objective is never characterized, so it is unclear which distribution the method actually targets.
- Prop. 1 requires the posterior of the clean sample given the noisy state to be an isotropic Gaussian location family, which does not hold for real data. The "in particular" clause derives positive-definiteness of the posterior-mean Jacobian from Gaussianity of the forward kernel, but this does not establish the location-family assumption the first part needs; as written it reads as if Gaussian noising alone suffices. Eq. (30) also drops a gradient symbol.
- Prop. 3 assumes a uniformly bounded drift Jacobian over the whole time interval, but the reference drift contains a term proportional to the state divided by time, and the noise coefficient diverges at the noise end; the assumption fails there, and the resulting bound is vacuous for realistic network Lipschitz constants except very close to the data end.
- Prop. 2 compares two estimators of the same SERF target under the assumption that the re-noising kernel is the true conditional law; it says nothing about the estimators actually used by RSM or AM and ignores bias. The introduction's claim of "provably lower gradient variance" than other methods and the abstract's "consistent estimate of the true reward gradient direction" are not supported.
W4. Narrow experimental scope and evaluation issues.
- A single base model (SD3.5-M), 512×512, 300 updates. The title and abstract claim "diffusion and flow models" but no non-flow-matching diffusion model is tested.
- Both training rewards are CLIP-based preference models and all evaluation metrics are learned scorers of the same family; there is no human evaluation, no compositional benchmark (e.g., GenEval/T2I-CompBench), and no non-preference differentiable reward, so reward hacking cannot be ruled out.
- SERF is not the most diverse method in either table (RSM or VGG-Flow is higher), which is at odds with "maintaining diverse outputs".
W5. Asymmetric baseline treatment and limited reproducibility.
- AM does not converge in 300 updates; Table 2 reports its highest-reward checkpoint for quality while Fig. 4 uses its slow convergence to argue efficiency. Only 10 steps of the memoryless SDE are used, whose noise coefficient diverges at the noise end, and no sensitivity to this choice is reported.
- §4.2 states that VGG-Flow and AM need 5–15× more training time "as detailed in §4.3", but Fig. 4 contains only AM and RSM; VGG-Flow timing is never shown and DRaFT-LV cost is never reported.
- The main text says the hyperparameter search spaces are in the appendix; the appendix contains none, limiting the reproducibility stated by the author.
- No per-iteration breakdown (NFEs, VJPs, memory, wall-clock per update), so it is impossible to tell whether SERF's speed comes from cheaper steps or more effective steps—particularly relevant against RSM, whose per-step cost should be nearly identical.
W6. Lots of inconsistencies and errors.
- Appendix C.2 states that t=1 is pure noise and t=0 is clean data, the opposite of the main-text convention; the meaning of "low-noise-biased sampling" and of the (1−t) weighting is therefore ambiguous.
- Eq. (5) has a factor one-half; Algorithm 1 does not. The reward coefficient appears in no loss but takes values of order ten thousand to a hundred thousand in experiments.
- Eq. (5) states uniform timestep sampling, but the default is low-noise-biased sampling whose distribution is never specified.
- Speed-up is reported as 10× (abstract), 5–15× (§4.2), and 13× (Fig. 4).
- Fig. 2 caption says five prompts; only four are shown.
- On page 4 the roles of [4] and [5] are swapped.
- Duplicate bibliography entries: Ho 2020a/b, Liu 2025a/b, Zheng 2026a/b.
- How the reference velocity is defined under CFG 4.5 and to which branches LoRA is applied is not stated.
- [11], which gives bias–variance decompositions for Adjoint Matching, is directly relevant to the variance claims but not discussed.
- No limitations section.
References
[1] Lee, Chang, Kim, Ye. Reward Score Matching: Unifying Reward-based Fine-tuning for Flow and Diffusion Models. arXiv:2604.17415, 2026.
[2] Domingo-Enrich, Drozdzal, Karrer, Chen. Adjoint Matching. ICLR 2025.
[3] Xu et al. ImageReward (ReFL). NeurIPS 2023. [4] McAllister et al. Finite Difference Flow Optimization (FDFO). arXiv:2603.12893, 2026.
[5] Zhou et al. Test-time Gradient Guidance of Flow Policies in RL. 2026.
[6] Clark, Vicol, Swersky, Fleet. Directly Fine-tuning Diffusion Models on Differentiable Rewards (DRaFT/DRaFT-LV). ICLR 2024.
[7] Zheng et al. DiffusionNFT. ICLR 2026.
[8] Bergmeister et al. Reinforce Adjoint Matching (RAM). arXiv:2605.10759, 2026.
[9] Shin et al. Efficient Adjoint Matching (EAM). arXiv:2605.11480, 2026.
[10] Liu et al. Value Gradient Guidance for Flow Matching Alignment (VGG-Flow). NeurIPS 2025.
[11] Domingo-Enrich, Du, Albergo. A Unified Perspective on Fine-tuning and Sampling with Diffusion and Flow Models. arXiv:2605.00229, 2026.
Questions for authors
Questions
-
How does the SERF target compare with the true value gradient as a function of time on SD3.5-M? How well does the identity-Jacobian approximation hold away from the data end?
-
With a constant time weight and the identity-Jacobian approximation, what is the fixed point of the SERF objective? Does it correspond to any tilted distribution?
-
What is the exact low-noise-biased timestep distribution used by default, and why does Eq. (5) state uniform sampling?
-
For AM, were more SDE steps or longer training tried? Given that the memoryless noise coefficient diverges at the noise end, are 10 steps sufficient?
-
Where is the VGG-Flow timing that supports the 5–15× claim, and what is DRaFT-LV's training cost? Are per-update NFEs, VJPs, and peak memory available for all methods?
-
How much wall-clock time does the 5-step rollout variant save relative to 10 steps?
9 comments
Reply to reviewer jdA1 (2/4)
The abstract now describes an approximation to the reward-gradient direction. SOC motivates the construction, while the new diagnostics below directly examine the practical approximation.
The fixed point of the SERF objective is never characterized, so it is unclear which distribution the method actually targets
We thank the reviewer for this observation, and we answer below, in the Questions section.
W4 (experimental scope)
...there is no human evaluation, no compositional benchmark (e.g., GenEval/T2I-CompBench), and no non-preference differentiable reward, so reward hacking cannot be ruled out.
We appreciate the suggestions for broadening the evaluation. We added an unoptimized compositional benchmark and a non-preference training objective to test whether SERF’s benefits extend beyond the preference metrics used for training.
GenEval. Selected checkpoints achieve 67.37% overall accuracy after HPSv2.1 training and 69.00% after PickScore training, versus 63.34% for the base, without further checkpoint selection. They rank first in all six categories after HPSv2.1 training and first or second after PickScore training (Appendix C.2, Table 5). These gains on an unoptimized compositional benchmark provide further evidence of reduced reward hacking.
JPEG compressibility. At approximately matched rewards, SERF better preserves subject structure and detail under JPEG-compressibility training, extending the reward–quality advantage beyond preference objectives (Appendix C.3, Figures 5 and 11).
W4 (Model scope)
The title and abstract claim "diffusion and flow models" but no non-flow-matching diffusion model is tested.
We appreciate the request to distinguish the formulation’s scope from its empirical coverage. Gaussian diffusion and flow matching are connected through score–velocity and ODE–SDE conversions (Section 2), so we do not treat them as disjoint model families. The empirical evaluation uses a single backbone, SD3.5-M, explicitly identified in Section 4.1.
W4 (Diversity)
SERF is not the most diverse method in either table (RSM or VGG-Flow is higher), which is at odds with "maintaining diverse outputs".
Thank you for highlighting this wording. Section 4.2 and the conclusion describe comparable diversity to RSM and VGG-Flow. Tables 1–2 show the lowest FID and highest mean CLIPScore among compared fine-tuned methods across all four settings. The claim is stronger reward–quality trade-offs with competitive diversity.
W5 (AM discretization, training speed and reproducibility)
We appreciate the reviewer observations on the Adjoint Matching discretization used and the imprecise timing statements. We address both topics below, in the Questions section, particularly Q4 and Q5. We want to thank the reviewer for these observations, as carefully addressing them allowed us to improve the quality and the precision of our manuscript.
W5 (reproducibility)
The main text says the hyperparameter search spaces are in the appendix; the appendix contains none, limiting the reproducibility stated by the author.
We appreciate the reviewer’s emphasis on reproducibility. Appendix C.1 and Table 4 now specify search spaces, selected configurations, and checkpoint selection. Tables 1–2 report means sample standard deviations across three independent training seeds; timing-table deviations describe variation across measured updates.
W6 (notation and implementation conventions)
Eq. (5) states uniform timestep sampling, but the default is low-noise-biased sampling whose distribution is never specified.
Thank you for identifying these ambiguities. We reconciled the objective in Section 3, Algorithm 1, and the experimental description in Section 4.1. Default SERF uses on , with denoting noise and denoting data. Reward strength enters through , with constant target factors absorbed into .
We clarify the complementary ablation structure in the General Response (2) and reproduce the relevant passage here for convenience:
Sampling and weighting (Appendix D.4).
- Figure 21: At fixed re-noising and reference weighting, we vary target weighting under , and sampling under t-weighted targets. Default SERF gives better overall trade-offs; SERF* uses uniform sampling and t-weighted targets.
- Figure 22: With uniform sampling and the target fixed, KL-derived reference weighting generally worsens FID and diversity.
- Figure 23: With uniform sampling and constant reference weighting, scalar transport corrections offer no consistent improvement over identity. Default SERF is also shown and achieves better overall trade-offs with . “Identity, no KL” provides the uniform-sampling, constant-target configuration for PickScore. These results favor simpler coefficients [3].
Reply to reviewer jdA1 (3/4)
We also corrected Figure 2’s caption, inconsistent speed statements, the missing gradient in Appendix A.3’s alignment proof, swapped reference attributions, and duplicate bibliography entries. Appendix B now discusses Domingo-Enrich, Du, and Albergo’s analysis and distinguishes single-state target variance from Proposition 2’s multi-state correlation.
Limitations
No limitations section.
Thank you for pointing out this omission. The Conclusion now includes a dedicated limitations paragraph on omitted anisotropic transport, between-endpoint variability, and future evaluation beyond text-to-image generation (Section 6).
Questions
Q1 (comparison with the value gradient)
How does the SERF target compare with the true value gradient as a function of time on SD3.5-M? How well does the identity-Jacobian approximation hold away from the data end?
We appreciate this request for a direct diagnostic, which motivated the new study in Appendix E.3. We evaluate frozen SD3.5-M with HPSv2.1 at initialization, where the control cost vanishes, and the diagnostic isolates reward guidance. At eight denoising times, we sample 32 independent SDE continuations from each fixed starting state and compute endpoint gradients and full pathwise pullbacks through the continuation and final projection. Averaging the pullbacks provides a Monte Carlo value-gradient reference, not exact ground truth.
Here, “SERF-SDE” denotes the SDE-based endpoint-gradient surrogate before the independent re-noising approximation. Its multi-continuation average is a diagnostic estimator, rather than an additional training requirement.
The shared results and interpretation are given in General Response (2), “Gradient-direction diagnostics”:
SERF's single-endpoint curve measures a sampled target, while its loss implicitly targets the conditional mean. Appendix A.3 shows that the two losses have the same expected parameter gradient under a fixed sampling law. SERF cannot directly sample endpoints conditional on a fixed re-noised state; forward SDE continuations, although in practice samples from a different conditional law, give a proxy.
Figure 30a shows that averaging endpoint gradients over SDE continuations from a fixed state produces substantially better-aligned guidance than using a single endpoint. At , alignment reaches approximately 0.3 with disjoint 16+16 continuations, versus below 0.1 for single-endpoint SERF and RSM (shared 32-path estimate: 0.7). Thus, the benefit of conditional averaging persists without shared continuation noise, even in this high-dimensional setting.
Figure 30b checks gradient transport and reference reliability separately. Endpoint gradients increasingly align with their own full pullbacks later in denoising (green), supporting the directional identity approximation there. Meanwhile, independent 16-path value-gradient estimates agree poorly at high noise (grey): the reference direction itself is difficult to estimate. Consequently, weak early alignment against that Monte Carlo reference cannot alone establish failure of the approximation.
Together, the regression equivalence explains the implicit averaging mechanism, the SDE diagnostics support its directional benefit, and the controlled comparison establishes implemented SERF’s practical advantage over RSM (Appendix D.1).
Q2 (fixed points)
With a constant time weight and the identity-Jacobian approximation, what is the fixed point of the SERF objective? Does it correspond to any tilted distribution?
Section 3 clarifies that Eq. (4) concerns the original KL-weighted SOC problem, whose optimum motivates SERF. For linear rewards, SERF's population regression target is the reference velocity plus the constant reward gradient.
The fixed-point conclusions are given in General Response (1), “Fixed points”:
Appendix A.5 characterizes SERF's population fixed points, showing a reward tilt for linear rewards under isotropic Gaussian reference data. We also derive a time-dependent target correction recovering the exact tilt for general rewards in the same setting.
A complete theoretical and causal characterization of these approximated neural-network training dynamics remains very challenging and outside this paper’s scope. We therefore complement the conditional analysis with direct diagnostics and controlled experiments.
Q3 (notation explanation)
What is the exact low-noise-biased timestep distribution used by default, and why does Eq. (5) state uniform sampling?
We thank the reviewer for the question and we answer above, in the Weaknesses section, answer W6.
Reply to reviewer jdA1 (1/4)
Reply to reviewer jdA1
We thank the reviewer for their detailed feedback and for recognizing SERF as “simple to implement” and its exposition as “clear and easy to follow.”
Your comments helped us sharpen the distinction between shared ingredients, SERF’s contribution, and the evidence supporting its approximations. Sections 1 and 5 and Appendix B now clarify the positioning, while Appendices D.1 and D.4 make the complementary controlled comparisons explicit. We also strengthen the theoretical and empirical justification through population fixed-point analysis (Appendix A.5) and direct gradient diagnostics (Appendix E.3). To address the experimental concerns, we broaden the evaluation through GenEval and JPEG experiments (Appendices C.2–C.3), examine AM discretization and measured training costs (Appendix E.1), and separately assess update effectiveness (Appendix E.2). Appendix C.1 and Table 4 complete the training details.
Weaknesses
W1 (contribution relative to prior work)
We appreciate the reviewer’s emphasis on distinguishing shared ingredients from SERF’s contribution. We explicitly credit prior endpoint-gradient and re-noising constructions in Sections 1 and 5, and Appendix B. SERF’s contribution is the SOC-motivated training construction and controlled demonstration of its practical benefits. Independent re-noising changes the supervision coupling and decouples training targets from the generation rollout. Table 3 identifies the shared components and differences; we absorb the factor one-half into the reward coefficient.
W2 (isolating independent re-noising)
We agree that the close relationship between the methods makes this controlled attribution particularly important. We address the comparison in the General Response (3) and reproduce the relevant passage here for convenience:
Regarding endpoint reuse, the reviewer additionally asks:
Thank you for suggesting this additional test. The matched-compute study compares 88×1, 80×2, 64×4, and 48×8 endpoint/re-noising allocations over 300 updates (Appendix D.3, Table 7, Figure 20). The 64×4 default generally improves held-out preference scores at high optimized rewards over allocations with fewer re-noisings and performs comparably to 48×8. This demonstrates flexibility to allocate computation according to the relative costs of endpoint sampling, reward scoring, and local training.
W3 (theoretical scope)
The shared theoretical clarifications are given in General Response (1), “Theoretical motivation, variance comparison, and fixed points”:
In particular, Proposition 2 does not establish a general ordering against implemented AM or a bound on Gaussian-proxy bias.