acceptodds
Under review as a conference paper at ICLR 2027

Simple Endpoint Reward Finetuning

11 views

Abstract

Reward fine-tuning adapts pretrained diffusion and flow models to desired objectives by optimizing a specified reward. For differentiable rewards, transporting reward gradients backwards along trajectories enables gradient computation at noisy states. However, implementing this in practice is prohibitively expensive; existing first order methods imitating this must make aggressive approximations, often estimating the trajectory through few-step samples and evaluating reward gradients off-policy. This motivates the development of a method using only reward gradients at clean samples, while still approximating the true gradient direction. To this end, we introduce SIMPLE ENDPOINT REWARD FINETUNING (SERF), a remarkably simple fine-tuning objective that avoids gradient transport by directly using the reward gradient at the endpoint. Specifically, at renoised states the trained velocity is regressed against the reference velocity shifted by the clean image's reward gradient. Although easy to state and implement, we show that SERF has a rigorous derivation through stochastic optimal control, and that our objective provides a consistent estimate of the true reward gradient direction. When compared with existing methods, SERF achieves the most favorable trade-offs when used to post-train Stable Diffusion 3.5 on PickScore and HPSv2.1, demonstrating its ability to learn rewards while maintaining diverse outputs. SERF is shown to be highly efficient, up to 10 times faster than Adjoint Matching, while remaining flexible enough to allow endpoint sampling in as few as 5 steps.

Primary area:
generative models
Keywords:
diffusion, flow matching, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_jdA1OpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SERF, a first-order method for reward fine-tuning flow and diffusion models with differentiable rewards. The reward gradient is computed once at each sample, the sample is independently re-noised with the standard Gaussian kernel, and the model velocity at the noisy state is regressed toward the reference velocity shifted by half the endpoint reward gradient. The method is motivated from the memoryless SOC formulation of Adjoint Matching by replacing the path Jacobian with the identity; the paper argues this surrogate is positively aligned with the true value gradient and that independent re-noising has lower gradient variance than trajectory states . On SD3.5-Medium with PickScore and HPSv2.1, SERF matches or beats DRaFT-LV, VGG-Flow, Adjoint Matching, and RSM at matched reward on FID and CLIPScore, trains roughly 10× faster than Adjoint Matching, and remains effective with 5-step rollouts.

Critical strengths
  • The method is simple to implement, requires no trajectory storage or backpropagation through sampling, is agnostic to the endpoint sampler, and decouples the number of training states from the number of sampling steps. These are practically attractive properties.

  • The exposition of the derivation chain (SOC objective → reward/regularization decomposition → identity-Jacobian surrogate → re-noising) is clear and easy to follow, and the conclusion is candid that the identity-Jacobian approximation is the source of the method's simplicity.

  • Prop. 2 gives a concrete formal reason to prefer independent re-noising over trajectory states for this family of targets, and the paper correctly points out that ODE trajectories produce a degenerate (deterministic) coupling (Xt,X1)(X_t, X_1).

  • The empirical comparison covers the relevant first-order baselines on a modern flow-matching model, reports trade-off curves rather than single points, includes qualitative evidence of reward-hacking artifacts in baselines (human-like faces in VGG-Flow/RSM), and includes ablations on training CFG, rollout length, timestep sampling, and target weighting.

  • The 5-step rollout result and the training-time curves (Fig. 4) are useful practical findings.

Critical weaknesses

W1. Limited novelty relative to RSM and prior work. The regression target "reference velocity plus a scaled clean-sample reward gradient" is already the first-order redesign proposed in RSM [1], which replaces local Tweedie guidance with the reward gradient at the clean sample, describes it as an SDS-style reward score distillation, and explicitly notes that the exact quantity would require the full Jacobian chain. The identity-Jacobian approximation is borrowed from the identity pullback of ReFL [3] and from [4, 5]; endpoint re-noising is used in DRaFT-LV [6], DiffusionNFT [7], RAM [8], and EAM [9]; Theorems 1–2 are standard results already present in Adjoint Matching [2]. What remains specific to SERF is (i) training on independently re-noised states rather than trajectory states, (ii) a constant time weight, and (iii) the factor one-half. Items (ii)–(iii) are coefficient-level choices within the design space of [1].

W2. The one distinguishing design choice is not ablated. There is no same-codebase experiment that switches only "trajectory states vs. independently re-noised states", and no empirical gradient-variance measurement. Consequently the gap between SERF and RSM in Tables 1–2 cannot be attributed to re-noising rather than to the time weighting, the one-half factor, the timestep distribution, or tuning. The claimed advantage of decoupling targets from rollout steps (four targets per endpoint) is never varied either.

W3. The theoretical justification does not match the algorithm actually run, and the abstract overclaims.

  • The endpoint-tilted optimum, Eq. (4), and the claim that the Gaussian re-noising kernel is exact at the optimum all rely on the KL-consistent time weighting (inverse squared noise coefficient). The algorithm uses a constant weight, under which the optimum is no longer the reward-tilted data distribution and Eq. (4) no longer holds.
  • The re-noising kernel and the drift-to-velocity conversion are justified as exact "at initialization and at the SOC optimum", but SERF does not converge to the SOC optimum because of the identity-Jacobian approximation. The fixed point of the SERF objective is never characterized, so it is unclear which distribution the method actually targets.
  • Prop. 1 requires the posterior of the clean sample given the noisy state to be an isotropic Gaussian location family, which does not hold for real data. The "in particular" clause derives positive-definiteness of the posterior-mean Jacobian from Gaussianity of the forward kernel, but this does not establish the location-family assumption the first part needs; as written it reads as if Gaussian noising alone suffices. Eq. (30) also drops a gradient symbol.
  • Prop. 3 assumes a uniformly bounded drift Jacobian over the whole time interval, but the reference drift contains a term proportional to the state divided by time, and the noise coefficient diverges at the noise end; the assumption fails there, and the resulting bound is vacuous for realistic network Lipschitz constants except very close to the data end.
  • Prop. 2 compares two estimators of the same SERF target under the assumption that the re-noising kernel is the true conditional law; it says nothing about the estimators actually used by RSM or AM and ignores bias. The introduction's claim of "provably lower gradient variance" than other methods and the abstract's "consistent estimate of the true reward gradient direction" are not supported.

W4. Narrow experimental scope and evaluation issues.

  • A single base model (SD3.5-M), 512×512, 300 updates. The title and abstract claim "diffusion and flow models" but no non-flow-matching diffusion model is tested.
  • Both training rewards are CLIP-based preference models and all evaluation metrics are learned scorers of the same family; there is no human evaluation, no compositional benchmark (e.g., GenEval/T2I-CompBench), and no non-preference differentiable reward, so reward hacking cannot be ruled out.
  • SERF is not the most diverse method in either table (RSM or VGG-Flow is higher), which is at odds with "maintaining diverse outputs".

W5. Asymmetric baseline treatment and limited reproducibility.

  • AM does not converge in 300 updates; Table 2 reports its highest-reward checkpoint for quality while Fig. 4 uses its slow convergence to argue efficiency. Only 10 steps of the memoryless SDE are used, whose noise coefficient diverges at the noise end, and no sensitivity to this choice is reported.
  • §4.2 states that VGG-Flow and AM need 5–15× more training time "as detailed in §4.3", but Fig. 4 contains only AM and RSM; VGG-Flow timing is never shown and DRaFT-LV cost is never reported.
  • The main text says the hyperparameter search spaces are in the appendix; the appendix contains none, limiting the reproducibility stated by the author.
  • No per-iteration breakdown (NFEs, VJPs, memory, wall-clock per update), so it is impossible to tell whether SERF's speed comes from cheaper steps or more effective steps—particularly relevant against RSM, whose per-step cost should be nearly identical.

W6. Lots of inconsistencies and errors.

  • Appendix C.2 states that t=1 is pure noise and t=0 is clean data, the opposite of the main-text convention; the meaning of "low-noise-biased sampling" and of the (1−t) weighting is therefore ambiguous.
  • Eq. (5) has a factor one-half; Algorithm 1 does not. The reward coefficient appears in no loss but takes values of order ten thousand to a hundred thousand in experiments.
  • Eq. (5) states uniform timestep sampling, but the default is low-noise-biased sampling whose distribution is never specified.
  • Speed-up is reported as 10× (abstract), 5–15× (§4.2), and 13× (Fig. 4).
  • Fig. 2 caption says five prompts; only four are shown.
  • On page 4 the roles of [4] and [5] are swapped.
  • Duplicate bibliography entries: Ho 2020a/b, Liu 2025a/b, Zheng 2026a/b.
  • How the reference velocity is defined under CFG 4.5 and to which branches LoRA is applied is not stated.
  • [11], which gives bias–variance decompositions for Adjoint Matching, is directly relevant to the variance claims but not discussed.
  • No limitations section.

References

[1] Lee, Chang, Kim, Ye. Reward Score Matching: Unifying Reward-based Fine-tuning for Flow and Diffusion Models. arXiv:2604.17415, 2026.

[2] Domingo-Enrich, Drozdzal, Karrer, Chen. Adjoint Matching. ICLR 2025.

[3] Xu et al. ImageReward (ReFL). NeurIPS 2023. [4] McAllister et al. Finite Difference Flow Optimization (FDFO). arXiv:2603.12893, 2026.

[5] Zhou et al. Test-time Gradient Guidance of Flow Policies in RL. 2026.

[6] Clark, Vicol, Swersky, Fleet. Directly Fine-tuning Diffusion Models on Differentiable Rewards (DRaFT/DRaFT-LV). ICLR 2024.

[7] Zheng et al. DiffusionNFT. ICLR 2026.

[8] Bergmeister et al. Reinforce Adjoint Matching (RAM). arXiv:2605.10759, 2026.

[9] Shin et al. Efficient Adjoint Matching (EAM). arXiv:2605.11480, 2026.

[10] Liu et al. Value Gradient Guidance for Flow Matching Alignment (VGG-Flow). NeurIPS 2025.

[11] Domingo-Enrich, Du, Albergo. A Unified Perspective on Fine-tuning and Sampling with Diffusion and Flow Models. arXiv:2605.00229, 2026.

Questions for authors

Questions

  1. How does the SERF target compare with the true value gradient as a function of time on SD3.5-M? How well does the identity-Jacobian approximation hold away from the data end?

  2. With a constant time weight and the identity-Jacobian approximation, what is the fixed point of the SERF objective? Does it correspond to any tilted distribution?

  3. What is the exact low-noise-biased timestep distribution used by default, and why does Eq. (5) state uniform sampling?

  4. For AM, were more SDE steps or longer training tried? Given that the memoryless noise coefficient diverges at the noise end, are 10 steps sufficient?

  5. Where is the VGG-Flow timing that supports the 5–15× claim, and what is DRaFT-LV's training cost? Are per-update NFEs, VJPs, and peak memory available for all methods?

  6. How much wall-clock time does the 5-step rollout variant save relative to 10 steps?

9 comments
AuthorsOpenReview ↗

Reply to reviewer jdA1 (1/4)

Reply to reviewer jdA1

We thank the reviewer for their detailed feedback and for recognizing SERF as “simple to implement” and its exposition as “clear and easy to follow.”

Your comments helped us sharpen the distinction between shared ingredients, SERF’s contribution, and the evidence supporting its approximations. Sections 1 and 5 and Appendix B now clarify the positioning, while Appendices D.1 and D.4 make the complementary controlled comparisons explicit. We also strengthen the theoretical and empirical justification through population fixed-point analysis (Appendix A.5) and direct gradient diagnostics (Appendix E.3). To address the experimental concerns, we broaden the evaluation through GenEval and JPEG experiments (Appendices C.2–C.3), examine AM discretization and measured training costs (Appendix E.1), and separately assess update effectiveness (Appendix E.2). Appendix C.1 and Table 4 complete the training details.

Weaknesses

W1 (contribution relative to prior work)

The regression target "reference velocity plus a scaled clean-sample reward gradient" is already the first-order redesign proposed in RSM [1]...

We appreciate the reviewer’s emphasis on distinguishing shared ingredients from SERF’s contribution. We explicitly credit prior endpoint-gradient and re-noising constructions in Sections 1 and 5, and Appendix B. SERF’s contribution is the SOC-motivated training construction and controlled demonstration of its practical benefits. Independent re-noising changes the supervision coupling and decouples training targets from the generation rollout. Table 3 identifies the shared components and differences; we absorb the factor one-half into the reward coefficient.

W2 (isolating independent re-noising)

Consequently the gap between SERF and RSM in Tables 1–2 cannot be attributed to re-noising rather than to the time weighting, the one-half factor, the timestep distribution, or tuning.

We agree that the close relationship between the methods makes this controlled attribution particularly important. We address the comparison in the General Response (3) and reproduce the relevant passage here for convenience:

Appendix D.1 (Figures 13–14) now presents the existing timestep experiment separately. RSM and SERF* match uniform timestep sampling, t-weighted targets, coefficient sweeps, and optimization; only training-state construction changes.

Re-noising improves held-out rewards, prompt alignment, and FID under both training rewards. Matching pretraining corruption may provide better states; Appendices A.3 and E.3 supply complementary conditional-averaging and gradient diagnostics.

Regarding endpoint reuse, the reviewer additionally asks:

The claimed advantage of decoupling targets from rollout steps (four targets per endpoint) is never varied either.

Thank you for suggesting this additional test. The matched-compute study compares 88×188 \times 1, 80×280 \times 2, 64×464 \times 4, and 48×848 \times 8 endpoint/re-noising allocations over 300 updates (Appendix D.3, Table 7, Figure 20). The 64×464 \times 4 default generally improves held-out preference scores at high optimized rewards over allocations with fewer re-noisings and performs comparably to 48×848 \times 8. This demonstrates flexibility to allocate computation according to the relative costs of endpoint sampling, reward scoring, and local training.

W3 (theoretical scope)

The theoretical justification does not match the algorithm actually run, and the abstract overclaims.

The shared theoretical clarifications are given in General Response (1), “Theoretical motivation, variance comparison, and fixed points”:

Alignment. Both the posterior Gaussian location-family and Gaussian-corruption assumptions are retained in Proposition 1. We clarified this in Section 3 and Appendix A.3 and corrected the abstract’s consistency wording. The proposition motivates our loss under idealized assumptions; the new conditional-averaging diagnostics provide empirical support for directional alignment on SD3.5-M (Appendix E.3, Figure 30a).

Gradient transport. Appendix A.2 works on [δ,1][\delta,1], with δ>0\delta>0, and motivates Jt,1≈IJ_{t,1}\approx I near the data endpoint. The direct endpoint–pullback comparison shows strong directional agreement on SD3.5-M (Appendix E.3, Figure 30b). We also derive and test scalar alternatives based on the known linear drift and Gaussian-reference Jacobian (Figure 23).

Variance. Proposition 2 proves a conditional covariance comparison under a common SDE joint law. It motivates independent sampling but does not compare SERF with RSM’s deterministic ODE coupling. The controlled RSM comparison and stochastic-trajectory ablation separately assess the practical sampling choices (Figures 13–16).

In particular, Proposition 2 does not establish a general ordering against implemented AM or a bound on Gaussian-proxy bias.

AuthorsOpenReview ↗

Reply to reviewer jdA1 (2/4)

The abstract now describes an approximation to the reward-gradient direction. SOC motivates the construction, while the new diagnostics below directly examine the practical approximation.

The fixed point of the SERF objective is never characterized, so it is unclear which distribution the method actually targets

We thank the reviewer for this observation, and we answer below, in the Questions section.

W4 (experimental scope)

...there is no human evaluation, no compositional benchmark (e.g., GenEval/T2I-CompBench), and no non-preference differentiable reward, so reward hacking cannot be ruled out.

We appreciate the suggestions for broadening the evaluation. We added an unoptimized compositional benchmark and a non-preference training objective to test whether SERF’s benefits extend beyond the preference metrics used for training.

GenEval. Selected checkpoints achieve 67.37% overall accuracy after HPSv2.1 training and 69.00% after PickScore training, versus 63.34% for the base, without further checkpoint selection. They rank first in all six categories after HPSv2.1 training and first or second after PickScore training (Appendix C.2, Table 5). These gains on an unoptimized compositional benchmark provide further evidence of reduced reward hacking.

JPEG compressibility. At approximately matched rewards, SERF better preserves subject structure and detail under JPEG-compressibility training, extending the reward–quality advantage beyond preference objectives (Appendix C.3, Figures 5 and 11).

W4 (Model scope)

The title and abstract claim "diffusion and flow models" but no non-flow-matching diffusion model is tested.

We appreciate the request to distinguish the formulation’s scope from its empirical coverage. Gaussian diffusion and flow matching are connected through score–velocity and ODE–SDE conversions (Section 2), so we do not treat them as disjoint model families. The empirical evaluation uses a single backbone, SD3.5-M, explicitly identified in Section 4.1.

W4 (Diversity)

SERF is not the most diverse method in either table (RSM or VGG-Flow is higher), which is at odds with "maintaining diverse outputs".

Thank you for highlighting this wording. Section 4.2 and the conclusion describe comparable diversity to RSM and VGG-Flow. Tables 1–2 show the lowest FID and highest mean CLIPScore among compared fine-tuned methods across all four settings. The claim is stronger reward–quality trade-offs with competitive diversity.

W5 (AM discretization, training speed and reproducibility)

We appreciate the reviewer observations on the Adjoint Matching discretization used and the imprecise timing statements. We address both topics below, in the Questions section, particularly Q4 and Q5. We want to thank the reviewer for these observations, as carefully addressing them allowed us to improve the quality and the precision of our manuscript.

W5 (reproducibility)

The main text says the hyperparameter search spaces are in the appendix; the appendix contains none, limiting the reproducibility stated by the author.

We appreciate the reviewer’s emphasis on reproducibility. Appendix C.1 and Table 4 now specify search spaces, selected configurations, and checkpoint selection. Tables 1–2 report means ±\pm sample standard deviations across three independent training seeds; timing-table deviations describe variation across measured updates.

W6 (notation and implementation conventions)

Eq. (5) states uniform timestep sampling, but the default is low-noise-biased sampling whose distribution is never specified.

Thank you for identifying these ambiguities. We reconciled the objective in Section 3, Algorithm 1, and the experimental description in Section 4.1. Default SERF uses p(t)=2tp(t) = 2t on (0,1)(0,1), with t=0t=0 denoting noise and t=1t=1 denoting data. Reward strength enters through R=βR0R = \beta R_0 , with constant target factors absorbed into β\beta.

We clarify the complementary ablation structure in the General Response (2) and reproduce the relevant passage here for convenience:

Sampling and weighting (Appendix D.4).

  • Figure 21: At fixed re-noising and reference weighting, we vary target weighting under p(t)=2tp(t)=2t, and sampling under t-weighted targets. Default SERF gives better overall trade-offs; SERF* uses uniform sampling and t-weighted targets.
  • Figure 22: With uniform sampling and the target fixed, KL-derived reference weighting generally worsens FID and diversity.
  • Figure 23: With uniform sampling and constant reference weighting, scalar transport corrections offer no consistent improvement over identity. Default SERF is also shown and achieves better overall trade-offs with p(t)=2tp(t)=2t. “Identity, no KL” provides the uniform-sampling, constant-target configuration for PickScore. These results favor simpler coefficients [3].
AuthorsOpenReview ↗

Reply to reviewer jdA1 (3/4)

We also corrected Figure 2’s caption, inconsistent speed statements, the missing gradient in Appendix A.3’s alignment proof, swapped reference attributions, and duplicate bibliography entries. Appendix B now discusses Domingo-Enrich, Du, and Albergo’s analysis and distinguishes single-state target variance from Proposition 2’s multi-state correlation.

Limitations

No limitations section.

Thank you for pointing out this omission. The Conclusion now includes a dedicated limitations paragraph on omitted anisotropic transport, between-endpoint variability, and future evaluation beyond text-to-image generation (Section 6).

Questions

Q1 (comparison with the value gradient)

How does the SERF target compare with the true value gradient as a function of time on SD3.5-M? How well does the identity-Jacobian approximation hold away from the data end?

We appreciate this request for a direct diagnostic, which motivated the new study in Appendix E.3. We evaluate frozen SD3.5-M with HPSv2.1 at initialization, where the control cost vanishes, and the diagnostic isolates reward guidance. At eight denoising times, we sample 32 independent SDE continuations from each fixed starting state and compute endpoint gradients and full pathwise pullbacks through the continuation and final projection. Averaging the pullbacks provides a Monte Carlo value-gradient reference, not exact ground truth.

Here, “SERF-SDE” denotes the SDE-based endpoint-gradient surrogate before the independent re-noising approximation. Its multi-continuation average is a diagnostic estimator, rather than an additional training requirement.

The shared results and interpretation are given in General Response (2), “Gradient-direction diagnostics”:

SERF's single-endpoint curve measures a sampled target, while its loss implicitly targets the conditional mean. Appendix A.3 shows that the two losses have the same expected parameter gradient under a fixed sampling law. SERF cannot directly sample endpoints conditional on a fixed re-noised state; forward SDE continuations, although in practice samples from a different conditional law, give a proxy.

Figure 30a shows that averaging endpoint gradients over SDE continuations from a fixed state produces substantially better-aligned guidance than using a single endpoint. At t≈0.85t\approx 0.85, alignment reaches approximately 0.3 with disjoint 16+16 continuations, versus below 0.1 for single-endpoint SERF and RSM (shared 32-path estimate: 0.7). Thus, the benefit of conditional averaging persists without shared continuation noise, even in this high-dimensional setting.

Figure 30b checks gradient transport and reference reliability separately. Endpoint gradients increasingly align with their own full pullbacks later in denoising (green), supporting the directional identity approximation there. Meanwhile, independent 16-path value-gradient estimates agree poorly at high noise (grey): the reference direction itself is difficult to estimate. Consequently, weak early alignment against that Monte Carlo reference cannot alone establish failure of the approximation.

Together, the regression equivalence explains the implicit averaging mechanism, the SDE diagnostics support its directional benefit, and the controlled comparison establishes implemented SERF’s practical advantage over RSM (Appendix D.1).

Q2 (fixed points)

With a constant time weight and the identity-Jacobian approximation, what is the fixed point of the SERF objective? Does it correspond to any tilted distribution?

Section 3 clarifies that Eq. (4) concerns the original KL-weighted SOC problem, whose optimum motivates SERF. For linear rewards, SERF's population regression target is the reference velocity plus the constant reward gradient.

The fixed-point conclusions are given in General Response (1), “Fixed points”:

Appendix A.5 characterizes SERF's population fixed points, showing a reward tilt for linear rewards under isotropic Gaussian reference data. We also derive a time-dependent target correction recovering the exact tilt for general rewards in the same setting.

A complete theoretical and causal characterization of these approximated neural-network training dynamics remains very challenging and outside this paper’s scope. We therefore complement the conditional analysis with direct diagnostics and controlled experiments.

Q3 (notation explanation)

What is the exact low-noise-biased timestep distribution used by default, and why does Eq. (5) state uniform sampling?

We thank the reviewer for the question and we answer above, in the Weaknesses section, answer W6.

Reviewer_cgQvOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Reward fine-tuning adapts diffusion and flow models to specific objectives, but backpropagating reward gradients along trajectories is computationally expensive and forces existing methods into aggressive approximations. To solve this, the authors propose SIMPLE ENDPOINT REWARD FINETUNING (SERF), which uses only endpoint reward gradients without gradient transport. By regressing the trained velocity against the reference velocity shifted by the clean image's reward gradient, SERF provides a mathematically sound, consistent estimate of the true gradient direction via stochastic optimal control. Post-training Stable Diffusion 3.5 on PickScore and HPSv2.1 demonstrates that SERF effectively balances reward learning with output diversity. Ultimately, SERF operates up to ten times faster than Adjoint Matching and enables endpoint sampling in as few as five steps.

Critical strengths
  • SERF provides a simple and computationally efficient alternative to full-trajectory reward backpropagation, avoiding trajectory storage and expensive adjoint computation.
  • The method achieves a strong empirical trade-off among reward improvement, image quality, diversity, and training efficiency.
Critical weaknesses
  1. The related work discussion in the introduction regarding "Several methods seek to propagate reward information from clean samples to noisy states without the full cost of BTT" is incomplete, lacking references to DRTune (arXiv:2405.00760), Direct-Align/SRPO (arXiv:2509.06942), EasyTune (arXiv:2602.07967), LeapAlign (arXiv:2604.15311), and Reward Sharpness-Aware Fine-Tuning (arXiv:2603.21175).
  2. The assertion in the introduction that "cheaper alternatives use noisy reward gradients at predicted rather than fully denoised samples that can lead to reward hacking" lacks direct literature or experimental backing.
  3. SERF applies the same endpoint gradient and identical weights to all XtX_t across different noise variances, yet the paper fails to adequately justify why this time-step-agnostic design is reasonable.
  4. Although the authors claim that "SERF enjoys many advantages," features such as being on-policy, sampler-agnostic, and reusing endpoint gradients are primarily implementation and efficiency perks; the authors fail to explain what practical benefits these yield or why they improve reward, FID, and diversity.
  5. It remains unclear why SERF outperforms existing methods in performance. The authors should analyze its differences, approximations, and expected advantages relative to differentiable reward methods and policy-gradient methods under a unified theoretical framework.

[minor] Inconsistencies exist across Eq (12), Algorithm 1, and the experiments regarding the 1/21/2 coefficient, the reward coefficient β\beta, and the time-step sampling strategy, making the actual optimization objective difficult to confirm.

Questions for authors

see weaknesses. If my concerns are adequately addressed, I would consider increasing my score; otherwise, I may consider lowering it.

11 comments
AuthorsOpenReview ↗

Reply to reviewer cgQv (1/3)

Reply to reviewer cgQv

We thank the reviewer for recognizing SERF as a “simple and computationally efficient alternative” and highlighting its “strong empirical trade-off among reward improvement, image quality, diversity, and training efficiency.”

We appreciate your request for a clearer account of why the simple construction works and how it relates to existing methods. We expanded the positioning in Sections 1 and 5 and added a common comparison framework in Appendix B and Table 3. To strengthen the design justification, we clarify the ablations in Appendices D.1 and D.4 and add gradient-direction, reward-per-update, and policy-deviation diagnostics in Appendices E.2–E.3. These connect the approximations to their practical consequences. We also removed the unsupported introductory claim and reconciled the notation across Section 3, Algorithm 1, and the experimental description in Section 4.1.

Weaknesses

W1 (related work)

The related work discussion in the introduction regarding "Several methods seek to propagate reward information from clean samples to noisy states without the full cost of BTT" is incomplete...

We appreciate these suggestions, which helped us better position SERF within the literature. We added DRTune, Direct-Align/SRPO, EasyTune, LeapAlign, and Reward Sharpness-Aware Fine-Tuning to the introduction (Section 1) and related work (Section 5). Appendix B and Table 3 clarify their connections and differences in reward evaluation, gradient construction, and training states.

W2 (predicted-image gradients)

The assertion in the introduction that "cheaper alternatives use noisy reward gradients at predicted rather than fully denoised samples that can lead to reward hacking" lacks direct literature or experimental backing.

We thank the reviewer for identifying this unsupported attribution. We removed the claim from Section 1 and revised the introduction to describe the computational trade-offs of existing methods.

AuthorsOpenReview ↗

Reply to reviewer cgQv (2/3)

W3 (timestep choices and weighting)

SERF applies the same endpoint gradient and identical weights ... across different noise variances, yet the paper fails to adequately justify why this time-step-agnostic design is reasonable.

We appreciate the request to justify these choices separately. Appendix A.2 motivates the identity approximation near the data endpoint. Default sampling p(t)=2tp(t) = 2t emphasizes these low-noise states and perceptual refinement, consistent with RSM’s timestep-dependent guidance analysis [1].

We clarify the complementary ablation structure in the General Response (2) and reproduce the relevant passage here for convenience:

Sampling and weighting (Appendix D.4).

  • Figure 21: At fixed re-noising and reference weighting, we vary target weighting under p(t)=2tp(t)=2t, and sampling under t-weighted targets. Default SERF gives better overall trade-offs; SERF* uses uniform sampling and t-weighted targets.
  • Figure 22: With uniform sampling and the target fixed, KL-derived reference weighting generally worsens FID and diversity.
  • Figure 23: With uniform sampling and constant reference weighting, scalar transport corrections offer no consistent improvement over identity. Default SERF is also shown and achieves better overall trade-offs with p(t)=2tp(t)=2t. “Identity, no KL” provides the uniform-sampling, constant-target configuration for PickScore. These results favor simpler coefficients [3].

These results support the simpler coefficients, consistent with earlier observations on simplified weighting [3].

Direct gradient diagnostics. Appendix E.3 empirically assesses the endpoint-gradient approximation at initialization, using the reference policy so that the control cost vanishes and the experiment isolates reward guidance. The setup, results, and interpretation are given in General Response (2), “Gradient-direction diagnostics”:

Appendix E.3 tests whether endpoint reward gradients align with the gradient of expected terminal reward on frozen SD3.5-M. We estimate the latter by backpropagating the average reward over independent SDE continuations from each fixed noisy state.

SERF's single-endpoint curve measures a sampled target, while its loss implicitly targets the conditional mean. Appendix A.3 shows that the two losses have the same expected parameter gradient under a fixed sampling law. SERF cannot directly sample endpoints conditional on a fixed re-noised state; forward SDE continuations, although in practice samples from a different conditional law, give a proxy.

Figure 30a shows that averaging endpoint gradients over SDE continuations from a fixed state produces substantially better-aligned guidance than using a single endpoint. At t≈0.85t\approx 0.85, alignment reaches approximately 0.3 with disjoint 16+16 continuations, versus below 0.1 for single-endpoint SERF and RSM (shared 32-path estimate: 0.7). Thus, the benefit of conditional averaging persists without shared continuation noise, even in this high-dimensional setting.

Figure 30b checks gradient transport and reference reliability separately. Endpoint gradients increasingly align with their own full pullbacks later in denoising (green), supporting the directional identity approximation there. Meanwhile, independent 16-path value-gradient estimates agree poorly at high noise (grey): the reference direction itself is difficult to estimate. Consequently, weak early alignment against that Monte Carlo reference cannot alone establish failure of the approximation.

Together, the regression equivalence explains the implicit averaging mechanism, the SDE diagnostics support its directional benefit, and the controlled comparison establishes implemented SERF’s practical advantage over RSM (Appendix D.1).

W4 (practical benefits)

...the authors fail to explain what practical benefits these yield or why they improve reward, FID, and diversity.

We appreciate the request to connect the implementation choices to benefits beyond computational convenience. The revised experiments examine the training-state construction, update effectiveness, and preservation of pretrained dynamics.

Independent re-noising. We address the controlled comparison in the General Response (3) and reproduce the relevant passage here for convenience:

Appendix D.1 (Figures 13–14) now presents the existing timestep experiment separately. RSM and SERF* match uniform timestep sampling, t-weighted targets, coefficient sweeps, and optimization; only training-state construction changes.

Re-noising improves held-out rewards, prompt alignment, and FID under both training rewards. Matching pretraining corruption may provide better states; Appendices A.3 and E.3 supply complementary conditional-averaging and gradient diagnostics.

AuthorsOpenReview ↗

Reply to reviewer cgQv (3/3)

More effective updates. Reward-versus-update curves demonstrate faster early improvement under both objectives (Appendix E.2, Figure 28). Against RSM, whose update cost is nearly identical, this directly explains the training-time advantage.

Preservation of pretrained dynamics. SERF’s velocity deviation stabilizes early and is substantially smaller than RSM, VGG-Flow, and DRaFT-LV at comparable rewards across all three diagnostic samplers (Appendix E.2, Figures 27 and 29). These results help explain its stronger performance through better preservation of pretrained dynamics [2].

We additionally hypothesize that matching the pretraining Gaussian corruption supplies better training states. The gradient diagnostics above provide complementary support for the directional approximation and conditional averaging as motivation for the design (Appendix E.3).

Endpoint reuse. General Response (2) additionally examines the practical benefit of reusing endpoint gradients:

At approximately matched compute, 64x464x4 generally improves held-out preference scores at high optimized rewards over allocations with fewer re-noisings and performs comparably to 48x848x8, supporting flexible allocation of the pipeline costs (Appendix D.3, Table 7, Figure 20).

W5 (unified comparison)

The authors should analyze its differences, approximations, and expected advantages relative to differentiable reward methods and policy-gradient methods under a unified theoretical framework.

We thank the reviewer for suggesting a common formulation to make these relationships clearer. Appendix B now introduces a reward-maximization perspective covering likelihood-based policy gradients and differentiable gradient-transport methods. Table 3 identifies differences in reward evaluation, gradient construction, and training states; the controlled ablations assess their practical consequences.

This provides a common basis for comparing the methods and their approximations. A complete theoretical characterization of the resulting neural-network training dynamics remains challenging and outside this paper’s scope.

Notation

Inconsistencies exist across Eq (12), Algorithm 1, and the experiments ... making the actual optimization objective difficult to confirm.

Thank you for pointing out these inconsistencies. We have reconciled the notation across the objective in Section 3, Algorithm 1, and the experimental description in Section 4.1.

Questions

Questions are addressed in the previous section (Weaknesses)

Concluding remarks

We thank the reviewer again for helping us strengthen the paper. The controlled experiments establish the benefit of independent re-noising, while the gradient diagnostics and regression equivalence strengthen the justification for endpoint guidance. The disjoint-sample checks support the later-denoising alignment advantage and clarify the uncertainty affecting high-noise measurements. More effective early updates and better preservation of pretrained dynamics further explain SERF’s practical advantages. Together with the weighting ablations and expanded comparison framework, these results substantiate the design beyond computational simplicity. We would be grateful for your reassessment in light of these clarifications and added evidence, and would be happy to discuss any remaining concerns.

References

[1] Jeongjae Lee, Jinho Chang, Jeongsol Kim, and Jong Chul Ye. Reward Score Matching: Unifying Reward-based Fine-tuning for Flow and Diffusion Models. arXiv:2604.17415, 2026.

[2] Rohit Jena, Ali Taghibakhshi, Sahil Jain, Gerald Shen, Nima Tajbakhsh, and Arash Vahdat. Elucidating Optimal Reward-Diversity Tradeoffs in Text-to-Image Diffusion Models. WACV, 2025.

[3] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising Diffusion Probabilistic Models. NeurIPS, 2020.

Reviewer_EQSEOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Simple Endpoint Reward Finetuning (SERF), a method for fine-tuning diffusion and flow-matching models with differentiable rewards. SERF independently re-noises samples generated by the current model and regresses the velocity at the re-noised states toward the reference velocity plus a scaled endpoint reward gradient, avoiding backpropagation through sampling trajectories and backward adjoint integration. Motivated by stochastic optimal control, the paper establishes, under the stated assumptions, positive alignment between the conditionally averaged endpoint-gradient surrogate and the exact reward-gradient target, and a covariance comparison showing that independent re-noising yields gradient estimates with no larger covariance than estimates constructed from a single SDE trajectory. Experiments fine-tuning SD3.5-M on PickScore and HPSv2.1 report favorable trade-offs among reward, distribution shift, and diversity against four baselines, comparing checkpoints at approximately matched optimized rewards where attainable. The paper also reports substantial training-time savings over Adjoint Matching.

Critical strengths

1. Simple and computationally attractive training design. SERF combines endpoint reward gradients, reference-velocity anchoring, and independent re-noising into a straightforward regression objective (Eq. 5; Algorithm 1). It avoids backpropagation through sampling trajectories, backward adjoint integration, and an additional learned guidance network. Each endpoint and its reward gradient can supervise multiple intermediate states, decoupling the number of training targets from the rollout length. In the tested setting, Figure 4 shows that SERF needs substantially less training time than Adjoint Matching to reach a comparable HPSv2.1 improvement, while the rollout-length ablations show that five-step training rollouts still yield substantial reward gains. These results make the design practically appealing for reducing the computational burden of reward fine-tuning.

2. Favorable trade-offs among reward, CLIPScore, and distribution shift. The evaluation covers two training rewards, held-out prompts and DrawBench, auxiliary reward scores, and within-prompt diversity. Across all four settings in Tables 1–2, which use approximately reward-matched checkpoints where attainable, SERF attains the lowest FID relative to base-model samples and the highest mean CLIPScore among the compared fine-tuning methods. For example, under PickScore training on Pick-a-Pic prompts, SERF has an FID of 144.9 versus RSM’s 172.6, and a CLIPScore of 0.2969 versus RSM’s 0.2850 and Adjoint Matching’s 0.2929. Although SERF does not lead on every auxiliary reward or diversity metric, these results support a favorable trade-off among optimized reward, CLIPScore, and distribution shift as measured by FID. This provides evidence of benefits beyond improving the optimized reward alone.

Critical weaknesses

1. The scope of the theoretical guarantees needs clearer qualification. The first part of Proposition 1 establishes alignment under a Gaussian location-family assumption for X1∣Xt=xX_1 \mid X_t=x, with covariance independent of xx, and a positive-definiteness condition on the symmetric part of the conditional-mean Jacobian. However, Gaussian corruption in the opposite direction, Xt∣X1X_t \mid X_1, does not by itself imply this posterior assumption. The covariance identity in Appendix A.3 establishes positive semidefiniteness of the conditional-mean Jacobian but does not remove the conditional Gaussian assumption used earlier in the proof. The “in particular” clause therefore requires clarification. Likewise, Proposition 2 shows that, under an exact conditional-law assumption, the re-noised estimator has no larger covariance than an estimator of the same surrogate using states from a single SDE trajectory. The broader variance claim in the introduction should be qualified accordingly. Although Appendix B explicitly invokes Proposition 2’s conditions, their applicability to the implemented RSM comparison is not established. Moreover, the paper justifies the re-noising kernel and the drift-to-velocity conversion by their exactness at the KL-regularized optimum b⋆b^\star (p. 5). However, after the identity-Jacobian approximation and the changes to regularization and conditional sampling, the paper does not establish that the final SERF update preserves that optimum. These limitations do not invalidate the empirical results, but they warrant clarifying the abstract’s consistency claim and distinguishing the conditional theoretical results from guarantees for the implemented algorithm.

2. The contribution of independent re-noising is not fully isolated, and some implementation details require clarification. Appendix B acknowledges that RSM also constructs local regression targets from endpoint reward gradients and identifies the construction of training states as a key difference. Although the paper includes useful ablations of rollout length, timestep sampling, and target weighting, it does not report a controlled comparison that changes only trajectory-state sampling versus independent re-noising while matching the target, timestep distribution, reward scaling, and optimization settings. Such a comparison would strengthen the attribution of the empirical gains to re-noising. The actual configuration is also unclear: Eq. (5) and Algorithm 1 specify uniform timestep sampling, whereas Appendix C.2 describes low-noise-biased sampling as the default without specifying its precise distribution, and reverses the noise/data endpoint convention used in the main text. The role of the tuned reward coefficient in the objective, the definition of the reported “±” values, and the number of independent repetitions should also be specified. The complete hyperparameter search spaces, which the paper states are given in the appendix, are not listed in the appendix of the submitted PDF. These clarifications would improve reproducibility and confidence in the interpretation of the comparisons.

Questions for authors
  1. Alignment assumptions and consistency. Does the “in particular” clause of Proposition 1 retain the earlier assumption that X1∣Xt=xX_1 \mid X_t=x is a Gaussian location family with covariance independent of xx? Please clarify how Gaussian corruption Xt∣X1X_t \mid X_1 relates to this posterior assumption, and what precise notion of consistency is intended by the abstract’s “consistent estimate” claim.

  2. Relationship to the SOC optimum. After the identity-Jacobian approximation and the changes to regularization and conditional sampling, is the original KL-regularized optimum b⋆b^\star still a fixed point of the final SERF update? Please clarify whether exactness of the re-noising kernel and drift-to-velocity conversion at b⋆b^\star is used as motivation or is intended to establish preservation of that optimum.

  3. Contribution of independent re-noising. How do Proposition 2’s assumptions apply to the implemented RSM comparison? Can you provide a controlled comparison that changes only trajectory-state sampling versus independent re-noising while matching the regression target, timestep distribution, reward scaling, and optimization settings? This would help isolate the contribution of re-noising to the reported gains.

  4. Training configuration and reproducibility. Please reconcile the uniform timestep sampling in Eq. (5)/Algorithm 1 with the low-noise-biased default and reversed noise/data endpoint convention in Appendix C.2. What precise sampling distribution, target weighting, and reward-gradient scaling are used in the main experiments? Please also provide the referenced hyperparameter search spaces and selected configurations, and define the reported “±” values and the number of independent repetitions.

6 comments
AuthorsOpenReview ↗

Reply to reviewer EQSE (1/3)

Reply to reviewer EQSE

We thank the reviewer for their thoughtful review and recognition of SERF’s “computationally attractive training design” and “evidence of benefits beyond improving the optimized reward alone.”

Your comments helped us clarify two central points: what the theoretical results establish for SERF, and whether re-noising itself accounts for the empirical gains. We revised the abstract and Section 3 to distinguish conditional guarantees from design motivation, added population fixed-point analysis (Appendix A.5), and directly examined the gradient approximation (Appendix E.3). We also present the existing controlled RSM comparison separately in Appendix D.1 and clarify how the complementary ablations examine the remaining design choices (Appendix D.4). Finally, Appendix C.1 and Table 4 supply the missing training details, while Section 4.2 and the table captions specify uncertainty reporting.

Weaknesses

W1 (Alignment assumptions and consistency)

The covariance identity in Appendix A.3 establishes positive semidefiniteness of the conditional-mean Jacobian but does not remove the conditional Gaussian assumption used earlier in the proof. The “in particular” clause therefore requires clarification.

We thank the reviewer for identifying this ambiguity. Both assumptions are retained in Proposition 1. We clarified this in Section 3 and Appendix A.3 and corrected the abstract’s consistency wording. The proposition motivates our loss under idealized assumptions.

To complement this theoretical motivation, we empirically assess the endpoint-gradient approximation on frozen SD3.5-M (Appendix E.3). At initialization, the model coincides with the reference policy, so the control cost vanishes and the experiment isolates reward guidance. The setup, results, and interpretation are given in General Response (2), “Gradient-direction diagnostics”:

Appendix E.3 tests whether endpoint reward gradients align with the gradient of expected terminal reward on frozen SD3.5-M. We estimate the latter by backpropagating the average reward over independent SDE continuations from each fixed noisy state.

SERF's single-endpoint curve measures a sampled target, while its loss implicitly targets the conditional mean. Appendix A.3 shows that the two losses have the same expected parameter gradient under a fixed sampling law. SERF cannot directly sample endpoints conditional on a fixed re-noised state; forward SDE continuations, although in practice smples from a different conditional law, give a proxy.

Figure 30a shows that averaging endpoint gradients over SDE continuations from a fixed state produces substantially better-aligned guidance than using a single endpoint. At t≈0.85t\approx 0.85, alignment reaches approximately 0.3 with disjoint 16+16 continuations, versus below 0.1 for single-endpoint SERF and RSM (shared 32-path estimate: 0.7). Thus, the benefit of conditional averaging persists without shared continuation noise, even in this high-dimensional setting.

Figure 30b checks gradient transport and reference reliability separately. Endpoint gradients increasingly align with their own full pullbacks later in denoising (green), supporting the directional identity approximation there. Meanwhile, independent 16-path value-gradient estimates agree poorly at high noise (grey): the reference direction itself is difficult to estimate. Consequently, weak early alignment against that Monte Carlo reference cannot alone establish failure of the approximation.

Together, the regression equivalence explains the implicit averaging mechanism, the SDE diagnostics support its directional benefit, and the controlled comparison establishes implemented SERF’s practical advantage over RSM (Appendix D.1).

W1 (Variance comparison and RSM)

Although Appendix B explicitly invokes Proposition 2’s conditions, their applicability to the implemented RSM comparison is not established.

We appreciate the request to distinguish the theoretical comparison from the implemented baseline. Proposition 2 provides a conditional covariance guarantee under a common SDE joint law; it does not directly compare with RSM’s deterministic ODE coupling. Section 3 and Appendix B now make this distinction explicit.

The stochastic-trajectory ablation provides complementary empirical support (Figures 15–16). Because that variant also changes the sampler and endpoint projection, the controlled RSM comparison below provides the direct test of changing the training-state construction. The conditional guarantee and empirical comparison thus address distinct parts of the justification.

AuthorsOpenReview ↗

Reply to reviewer EQSE (2/3)

W1 (Relationship to the SOC optimum)

However, after the identity-Jacobian approximation and the changes to regularization and conditional sampling, the paper does not establish that the final SERF update preserves that optimum.

We appreciate this distinction between motivating a construction and establishing its fixed point. Section 3 presents SOC as motivation for SERF’s design, rather than an exact derivation of the final algorithm or a guarantee that it solves the original SOC objective.

Appendix A.5 separately analyzes SERF’s population fixed points and shows a reward tilt for linear rewards under isotropic Gaussian reference data. This directly characterizes the implemented surrogate in a tractable setting. Understanding general neural-network training dynamics remains challenging and outside this paper’s scope.

W2 Contribution of independent re-noising

The contribution of independent re-noising is addressed below, see Q3 in the Questions section.

In Q4 we address timestep sampling and reward scaling inconsistencies. We want to thank the reviewer for having spotted such inconsistencies.

W2 (Hyperparameters and uncertainty reporting)

Please also provide the referenced hyperparameter search spaces and selected configurations, and define the reported ±\pm values and the number of independent repetitions.

We appreciate the reviewer’s emphasis on making the comparisons reproducible. Appendix C.1 and Table 4 now specify the search spaces, selected configurations, and checkpoint-selection procedure for each evaluation set. Section 4.2 and the table captions define the results as means ±\pm sample standard deviations across three independent training seeds, making the source of the reported uncertainty explicit.

Questions

Q1

Alignment assumptions and consistency. Does the “in particular” clause of Proposition 1 retain the earlier assumption that is a Gaussian location family with covariance independent of ? Please clarify how Gaussian corruption relates to this posterior assumption, and what precise notion of consistency is intended by the abstract’s “consistent estimate” claim.

We thank the reviewer for this question, and we answer above, in W1 (Alignment assumptions and consistency).

Q2

Relationship to the SOC optimum. After the identity-Jacobian approximation and the changes to regularization and conditional sampling, is the original KL-regularized optimum still a fixed point of the final SERF update? Please clarify whether exactness of the re-noising kernel and drift-to-velocity conversion at is used as motivation or is intended to establish preservation of that optimum

We appreciate this reviewer’s observation and we report the answer above, in W1 (Relationship to the SOC optimum). We now also cover the topic in Appendix A.5.

Q3

Contribution of independent re-noising. How do Proposition 2’s assumptions apply to the implemented RSM comparison? Can you provide a controlled comparison that changes only trajectory-state sampling versus independent re-noising while matching the regression target, timestep distribution, reward scaling, and optimization settings? This would help isolate the contribution of re-noising to the reported gains.

We agree that the close relationship between SERF and RSM makes this controlled attribution particularly important. We address the comparison in the General Response (3) and reproduce the relevant passage here for convenience:

Appendix D.1 (Figures 13–14) now presents the existing timestep experiment separately. RSM and SERF* match uniform timestep sampling, t-weighted targets, coefficient sweeps, and optimization; only training-state construction changes.

Re-noising improves held-out rewards, prompt alignment, and FID under both training rewards. Matching pretraining corruption may provide better states; Appendices A.3 and E.3 supply complementary conditional-averaging and gradient diagnostics.

We hypothesize that matching the pretraining corruption supplies better training states. Complementary diagnostics show faster early reward improvement under both objectives (Figure 28), alongside early stabilization of velocity deviation (Figure 27), connecting stronger performance to more effective updates and preservation of pretrained dynamics.

AuthorsOpenReview ↗

Reply to reviewer EQSE (3/3)

Q4

Training configuration and reproducibility. Please reconcile the uniform timestep sampling in Eq. (5)/Algorithm 1 with the low-noise-biased default and reversed noise/data endpoint convention in Appendix C.2. What precise sampling distribution, target weighting, and reward-gradient scaling are used in the main experiments? Please also provide the referenced hyperparameter search spaces and selected configurations, and define the reported ±\pm values and the number of independent repetitions

Thank you for pointing out these inconsistencies. We reconciled the objective in Section 3, Algorithm 1, and the experimental description in Section 4.1. Default SERF uses p(t)=2tp(t) = 2t on (0,1)(0,1), with t=0t=0 denoting noise and t=1t=1 denoting data, and constant reference and reward-gradient weights. Reward strength enters through R=βR0R = \beta R_0, with constant target factors absorbed into β\beta.

The 300-update ablation supports this configuration: default SERF achieves better overall reward–quality trade-offs than both target-weighted alternatives (Appendix D.4, Figure 21). The General Response (2) explains how Figures 21–23 separately examine sampling, target scaling, and reference weighting:

  • Figure 21: At fixed re-noising and reference weighting, we vary target weighting under p(t)=2tp(t)=2t, and sampling under t-weighted targets. Default SERF gives better overall trade-offs; SERF* uses uniform sampling and t-weighted targets.
  • Figure 22: With uniform sampling and the target fixed, KL-derived reference weighting generally worsens FID and diversity.
  • Figure 23: With uniform sampling and constant reference weighting, scalar transport corrections offer no consistent improvement over identity. Default SERF is also shown and achieves better overall trade-offs with p(t)=2tp(t)=2t. “Identity, no KL” provides the uniform-sampling, constant-target configuration for PickScore. These results favor simpler coefficients [3].

We also appreciate the reviewer’s emphasis on reproducibility, and we address this W2 (Hyperparameters and uncertainty reporting), see above.

Concluding remarks

We thank the reviewer again for helping us strengthen the paper. The controlled comparison isolates a substantial benefit from re-noising, while the gradient diagnostics and regression equivalence strengthen the justification for endpoint guidance. The disjoint-sample checks support the later-denoising alignment advantage and clarify the uncertainty affecting high-noise measurements. The new training curves additionally demonstrate more effective early updates. Together with the clarified theoretical scope and completed training details, these results substantiate SERF’s contribution. We would be grateful for your reassessment in light of these clarifications and added evidence, and would be happy to discuss any remaining concerns.

Comments on the paper

4 comments
AuthorsOpenReview ↗

General Response (Overview)

Overview

We thank all reviewers for their constructive feedback. We appreciate their recognition of SERF’s “computationally attractive training design” (EQSE), “strong empirical trade-off” (cgQv), and “clear and easy to follow” derivation (jdA1).

Several concerns overlap. We summarize the corresponding revisions here and address reviewer-specific questions individually.

  1. Theoretical scope and gradient approximation. We clarify which results provide conditional guarantees and which motivate the implemented design, and analyze fixed points in a simplified setting. SD3.5-M experiments directly assess the endpoint-gradient approximation and connect its empirical behavior to our motivational theoretical results.

  2. Practical benefits and design choices. We compare the approximations with principled alternatives and add policy-deviation metrics that help explain SERF’s stronger performance through better preservation of pretrained dynamics. Reward-per-update results additionally show more effective early updates. GenEval, JPEG, AM discretization, and measured-cost comparisons broaden the empirical validation.

  3. Contribution relative to RSM and prior work. We credit the shared ingredients and present the existing controlled RSM comparison separately. Changing only the training-state construction substantially improves reward–quality trade-offs, isolating the practical contribution of independent re-noising.

We detail these shared points below. Revised manuscript text is shown in blue.

AuthorsOpenReview ↗

General Response (1/2)

1. Theoretical motivation, variance comparison, and fixed points

(EQSE W1/Q1/Q2; cgQv W3; jdA1 W3/Q1/Q2)

We revised the abstract and Section 3 to distinguish conditional guarantees from results motivating the implemented approximations, and corrected notation inconsistencies.

Alignment. Both the posterior Gaussian location-family and Gaussian-corruption assumptions are retained in Proposition 1. We clarified this in Section 3 and Appendix A.3 and corrected the abstract’s consistency wording. The proposition motivates our loss under idealized assumptions; the new conditional-averaging diagnostics provide empirical support for directional alignment on SD3.5-M (Appendix E.3, Figure 30a).

Gradient transport. Appendix A.2 works on [δ,1][\delta,1], with δ>0\delta >0, and motivates Jt,1≈IJ_{t,1} \approx I near the data endpoint. The direct endpoint–pullback comparison shows strong directional agreement on SD3.5-M (Appendix E.3, Figure 30b). We also derive and test scalar alternatives based on the known linear drift and Gaussian-reference Jacobian (Figure 23).

Variance. Proposition 2 proves a conditional covariance comparison under a common SDE joint law. It motivates independent sampling but does not compare SERF with RSM’s deterministic ODE coupling. The controlled RSM comparison and stochastic-trajectory ablation separately assess the practical sampling choices (Figures 13–16).

Fixed points. Appendix A.5 characterizes SERF's population fixed points, showing a reward tilt for linear rewards under isotropic Gaussian reference data. We also derive a time-dependent target correction recovering the exact tilt for general rewards in the same setting.

A complete theoretical and causal characterization of these approximated neural-network training dynamics remains very challenging and outside this paper’s scope. We therefore complement the conditional analysis with direct diagnostics and controlled experiments.

2. Practical benefits, design choices, and empirical validation

(EQSE W2/Q3/Q4; cgQv W3/W4/W5; jdA1 W2/W3/W4/W5/Q1)

The controlled re-noising comparison (Appendix D.1), reward-per-update and policy-deviation diagnostics (Appendix E.2), and gradient-alignment study (Appendix E.3) connect SERF’s design to its stronger reward–quality trade-offs.

Gradient-direction diagnostics Appendix E.3 tests whether endpoint reward gradients align with the gradient of expected terminal reward on frozen SD3.5-M. We estimate the latter by backpropagating the average reward over independent SDE continuations from each fixed noisy state.

SERF's single-endpoint curve measures a sampled target, while its loss implicitly targets the conditional mean. Appendix A.3 shows that the two losses have the same expected parameter gradient under a fixed sampling law. SERF cannot directly sample endpoints conditional on a fixed re-noised state; forward SDE continuations, although in practice samples from a different conditional law, give a proxy.

Figure 30a shows that averaging endpoint gradients over SDE continuations from a fixed state produces substantially better-aligned guidance than using a single endpoint. At t≈0.85t \approx 0.85, alignment reaches approximately 0.3 with disjoint 16+16 continuations, versus below 0.1 for single-endpoint SERF and RSM (shared 32-path estimate: 0.7). Thus, the benefit of conditional averaging persists without shared continuation noise, even in this high-dimensional setting.

Figure 30b checks gradient transport and reference reliability separately. Endpoint gradients increasingly align with their own full pullbacks later in denoising (green), supporting the directional identity approximation there. Meanwhile, independent 16-path value-gradient estimates agree poorly at high noise (grey): the reference direction itself is difficult to estimate. Consequently, weak early alignment against that Monte Carlo reference cannot alone establish failure of the approximation.

Together, the regression equivalence explains the implicit averaging mechanism, the SDE diagnostics support its directional benefit, and the controlled comparison establishes implemented SERF’s practical advantage over RSM (Appendix D.1).

More effective updates. Reward-versus-update curves demonstrate faster early improvement under both training objectives (Appendix E.2, Figure 28). Against RSM, whose update cost is identical, this isolates more effective updates as one of the sources of training-time advantage.

Policy deviation. Training-evolution and matched-reward diagnostics show early stabilization and substantially smaller velocity deviation across all three diagnostic samplers (Appendix E.2, Figures 27 and 29), supporting preservation of pretrained dynamics as an explanation for SERF’s stronger performance [2].

AuthorsOpenReview ↗

General Response (2/2)

Endpoint reuse. At approximately matched compute, 64×464 \times 4 generally improves held-out preference scores at high optimized rewards over allocations with fewer re-noisings and performs comparably to 48×848 \times 8, supporting flexible allocation of the pipeline costs (Appendix D.3, Table 7, Figure 20).

Short rollouts and hyperparameter robustness. Ablations show stable training across the tested CFG scales and substantial reward gains with five-step rollouts at 26% lower update cost. Re-noising decouples training-target counts from rollout length (Appendix D.2, Figures 6 and 17–19, Table 8).

Broader validation. GenEval gains provide further evidence of reduced reward hacking, while matched-reward JPEG comparisons show better preservation of subject structure and detail beyond preference objectives (Appendices C.2–C.3, Table 5, Figures 5 and 11). Appendix C.3 additionally tracks a fixed-seed JPEG-training progression, showing how backgrounds and fine detail simplify as reconstruction error decreases (Figure 12).

AM discretization and measured costs. Forty-step integration does not consistently improve AM’s trade-offs despite approximately 2.5×2.5\times higher update cost (Figure 26). Appendix E.1 separately reports update costs (Table 8) and compute at matched rewards (Figure 24). The full 300-update curves show reward improvement largely plateauing after approximately 200 updates (Figure 25), supporting the budget used in these comparisons.

Reproducibility. Appendix C.1 and Table 4 specify search spaces, selected configurations, and checkpoint selection. Main-table results are means ±\pm sample standard deviations across three independent training seeds.

3. Contribution relative to RSM and independent re-noising

(EQSE W2/Q3; cgQv W1/W5; jdA1 W1/W2)

SERF shares RSM's endpoint reward gradients [1] and credits prior re-noising constructions (Sections 1 and 5, Appendix B). The factor one-half is absorbed into the reward coefficient.

Appendix D.1 (Figures 13–14) now presents the existing timestep experiment separately. RSM and SERF* match uniform timestep sampling, t-weighted targets, coefficient sweeps, and optimization; only training-state construction changes.

Re-noising improves held-out rewards, prompt alignment, and FID under both training rewards. Matching pretraining corruption may provide better states; Appendices A.3 and E.3 supply complementary conditional-averaging and gradient diagnostics.

Common comparison framework Appendix B now introduces a common reward-maximization formulation covering likelihood-based policy gradients and differentiable gradient-transport methods. Table 3 identifies differences in reward evaluation, gradient construction, and training states; the controlled ablations above assess their practical consequences. Appendix B also distinguishes prior analyses of conditional target variance at a single state from the correlations across multiple training states addressed by Proposition 2.

Concluding remarks

We respectfully ask reviewers to reassess SERF in light of the controlled re-noising comparison, more effective early updates, expanded validation, additional value gradient analysis, and clarified theoretical scope. We welcome further discussion.

References

[1] Jeongjae Lee, Jinho Chang, Jeongsol Kim, and Jong Chul Ye. Reward Score Matching: Unifying Reward-based Fine-tuning for Flow and Diffusion Models. arXiv:2604.17415, 2026.

[2] Rohit Jena, Ali Taghibakhshi, Sahil Jain, Gerald Shen, Nima Tajbakhsh, and Arash Vahdat. Elucidating Optimal Reward-Diversity Tradeoffs in Text-to-Image Diffusion Models. WACV, 2025.

[3] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising Diffusion Probabilistic Models. NeurIPS, 2020.

Summary of discussion

We sincerely thank all three reviewers for reading our manuscript and engaging with us throughout the discussion. Their feedback helped us sharpen the paper’s claims and strengthen the experimental evaluation.

Initial strengths. All three reviewers highlighted SERF’s simplicity and efficiency. EQSE and cgQv noted that it improves reward while preserving image quality, and jdA1 praised the clear derivation and highlighted the comparisons with relevant baselines across reward levels.

Commonly raised concerns and how we addressed them.

(i) Theoretical scope and approximation (EQSE, cgQv, jdA1). We clarified the assumptions behind SERF and added theoretical and empirical analyses of its approximation (Appendices A.5 and E.3).

(ii) Contribution relative to RSM (EQSE, jdA1). Our controlled comparison isolates the benefits of independent re-noising (Appendix D.1), and observes less velocity drift (Appendix E.2).

(iii) Empirical scope (jdA1) and reproducibility (EQSE, cgQv, jdA1). We broadened the evaluation, added ablations and timing comparisons, and documented the training and evaluation setup in more detail (Appendices C–E).

Follow-up discussion. cgQv asked about SERF’s practical advantages over ReFL and EasyTune. We clarified the differences and added supporting experiments (Figure 3 and Appendix E.4). jdA1 asked whether SERF’s gains extend to another backbone. Our comparison with RSM shows that the findings extend to Cosmos 3 (Appendix F).

Outcome. All three reviewers now rate the paper 3 (weak acceptance). EQSE maintained their original rating. cgQv raised their score after the revisions and follow-up comparisons. jdA1 raised their score after the initial revisions and again after our follow-up response.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.