acceptodds
Under review as a conference paper at ICLR 2027

Learning Where and What to Optimize: From Diffusion Distillation to Reinforcement Learning

Abstract

Teacher-guided distillation provides dense supervision and fast optimization, whereas reward-driven reinforcement learning (RL) directly optimizes scalar rewards at a substantially higher sampling cost. We seek to unify these complementary strengths within a single diffusion post-training framework. A natural alternative is sequential training: distill first, then switch to reward optimization. However, such stage-wise training can suffer from post-switch degradation, late-stage collapse, and sensitivity to the handoff schedule, particularly under limited prompt diversity. We propose a Unified Reward-adaptive Framework (URF) that jointly determines where to optimize and what velocity target to follow for each generated sample. Both decisions are derived from a single reward-adaptive objective and controlled by the same sample-wise reward-preference weight p ∈ [0, 1]. For the velocity target, motivated by the coarse negative-sample update directions, URF uses negative reward feedback to constrain teacher-guided correction rather than directly applying the repulsive NFT update. As the reward increases, p smoothly transitions the update from DiffusionOPD to DiffusionNFT, eliminating the need for an explicit stage-wise switch. To our knowledge, URF is the first diffusion post-training framework to unify teacher-guided distillation and reward-driven RL at both the query and velocity-target levels within a single reward-adaptive objective. Extensive experiments show that URF combines substantially faster convergence than teacher-guided distillation with continued reward-driven improvement, achieving 0.963 on both GenEval and OCR while converging 1.7–7.9× faster than DiffusionOPD in multi-reward training.

Primary area:
learning theory
Keywords:
Diffusion Models, Diffusion Distillation, Reinforcement Learning, Reward Optimization, diffusion NFT

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_qz9YOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies joint distillation and reward-based fine-tuning for image diffusion models. The authors observe an instability when switching between distillation and RL fine-tuning when done sequentially. They then define a unified objective and optimization algorithm by blending both the target and the query from on-policy distillation and diffusion NFT. The authors show that by controlling the parameters of their loss they can recover a certain version of NFT on one end and OPD on the other. They conduct experiments by fine-tuning an image diffusion model using a teacher model and a suite of tasks with reward functions, showing slightly better performance and significantly faster convergence compared to OPD or NFT alone.

Critical strengths

The method is grounded in literature and the authors have a good understanding of the OPD and NFT methods. The URF algorithm design presents a reasonable combination of the two.

Critical weaknesses
  1. The gains reported in Table 1 are slim, but the experiment, according to the appendix, is not averaged over multiple seeds. For score differences of this magnitude, a variance estimation over several training runs is necessary to establish significance, especially with the discrepancies between the starting scores of the methods (see below) and baseline scores fluctuations compared to the original papers.
  2. An unexplained discrepancy between the different methods’ starting performance exists on training curves, particularly those displayed in Figure 6 (a, b, c) and Figure 7 (a). In figures 6a, 6b, 6c and 7a, the score of NFT at the beginning of training is significantly lower than that of the other methods. Additionally, in figure 6c the starting quality of URF and OPD differ by an amount comparable to the final quality difference. These discrepancies put the validity of the experimental setup into question and need to be resolved or explained in the paper.
  3. The authors motivate the need for this method by observing an instability in training when switching between OPD and NFT. The study of switching between OPD and NFT, however, is limited to a few experiments with only one-time-switch setups. The main comparisons in this paper are with OPD-only and NFT-only methods. Given the motivation of the method through the issue of switching between OPD and NFT, I consider a larger study of different switching strategies an essential baseline.
Questions for authors
  1. Are there better ways to combine sequential (or mixed) OPD+NFT updates? What would be the outcome?
  2. What explains the different starting points for the different methods in Figures 6 and 7? What impact do these differences have on the final comparisons?
  3. How are the NFT baselines set up? The appendix contains detailed explanations for OPD and URF, but not NFT. Notably, the scores reported by the original NFT paper are substantially different in several of the reward functions, in some cases closely approaching URF.
  4. What are the CFG settings used for NFT, OPD and URF? How was this choice motivated? The appendix reports various CFG for URF on different datasets, the NFT CFG setting is not clear.
  5. How is the method to compute the specific speedup numbers motivated? Computing the time to reach a baselines peak could produce a misleading number, since URF reaches a higher final reward. This metric is very sensitive to the length of OPD training, with an extreme point on the plateau-ing curve at the end of training inflating the score.
  6. Figure 3 seems to suggest that setting URF’s p to 1 recovers the NFT method, however according to the derivation in Appendix A, it recovers only the positive NFT vector. I think this is an important distinction and should be displayed in Figure 3.
  7. Is the caption to figure 6 accurate? NFT never reaches the OPD peak.

I believe that the method is reasonable and grounded in the literature, however the methodology of the experimentation has severe issues that prevent the motivation and evaluation of URF from being effective. I am open to revising my score if the authors make significant improvements to the experimental suite, and explain and resolve the inconsistencies between the reported scores.

5 comments
AuthorsOpenReview ↗

Author Response (1/2)

We sincerely appreciate the reviewer's careful evaluation and constructive feedback.

In response, we conducted additional experiments on multi-seed robustness [W1] and alternative switching strategies [W3,Q1], clarified the baseline configurations and evaluation protocols [W2, Q2–Q4], and provided further technical clarifications [Q5–Q7]. We address each concern below and would be happy to clarify any remaining questions and make every effort to conduct additional experiments if needed.

[W1] Statistical Reliability and Multi-Seed Evaluation

We acknowledge that the results in Table 1 require further multi-seed statistical validation. To assess robustness, we conducted additional experiments using two independent training seeds (123 and 456), allocating 50 GPU-hours to each method.

SeedGenEval URFGenEval OPDGenEval NFTOCR URFOCR OPDOCR NFT
1230.9590.8430.6160.9520.9170.923
4560.9520.8280.5240.9480.8780.901
Mean0.9560.8360.5700.9500.8980.912

URF consistently outperforms both baselines across the additional seeds, improving over OPD by 0.120 on GenEval and 0.052 on OCR on average under matched compute budgets.

We address the initial-score discrepancies and baseline-score fluctuations in [Q2–Q4] below and Appendix B.3.


[W2, Q2] Clarification of Initial Performance Differences

We thank the reviewer for pointing out the differences in initial performance.

For the DiffusionNFT comparisons in Figures 6(a–c) and 7(a), we use CFG 1.0 following the original DiffusionNFT configuration, whereas URF and OPD are evaluated with CFG 4.5. Since DiffusionNFT is designed for CFG-free optimization and achieved higher reward scores under CFG 1.0 than under stronger guidance in our experiments, we adopted this setting. We have clarified the evaluation CFG settings in the figures and Appendix B.3.

Regarding the remaining initial-score discrepancies between URF and OPD in figure 6(c), we reconfirmed that both methods share the same pretrained initialization and evaluation CFG. Nevertheless, differences may arise from implementation-specific sampling, initial noise difference and numerical procedures.

These initial-score discrepancies do not necessarily indicate an experimental error, as Figures 6(a) and 6(d) also show that URF can start below the baselines but ultimately achieve higher rewards. Together with its consistent convergence advantages across the evaluated settings, these results support URF's effectiveness beyond differences in initial evaluation scores.


[W3, Q1] Comparison with Alternative Switching Strategies

We thank the reviewer for suggesting a broader evaluation of switching strategies.

We previously evaluated the following switching and interpolation strategies:

  • Training-stage switching (Fig. 2(a)): Switching from OPD to NFT at steps 30, 60, or 90 produces schedule-dependent degradation or instability.
  • Direct loss interpolation (Fig. 7(c)): Directly interpolating OPD and NFT losses leads to training collapse.

We additionally evaluate nonlinear preference calibration (see Appendix C, Fig. 9 in the revised paper for the full training curves):

pγ(s)=sγsγ+(1−s)γp_{\gamma}(s)=\frac{s^{\gamma}}{s^{\gamma}+(1-s)^{\gamma}}

where γ→∞\gamma\to\infty corresponds to sample-wise hard switching.

CalibrationOCR
γ=0.5\gamma=0.50.8981
γ=1\gamma=1 (URF)0.9608
γ=2\gamma=20.8812
γ→∞\gamma\to\infty (Hard Switching)0.8641

These experiments consistently favor the continuous linear calibration used by URF over the tested switching and interpolation alternatives.

From a theoretical perspective, linear preference calibration also provides stability-motivated properties. For any Lipschitz function f:[0,1]→[0,1]f:[0,1]\to[0,1] satisfying f(0)=0f(0)=0 and f(1)=1f(1)=1, its Lipschitz constant satisfies Lf≥1L_f\geq1, with equality attained by f(s)=sf(s)=s. Thus, linear calibration minimizes the worst-case sensitivity of the preference weight to reward-normalization perturbations.

Moreover, Appendix A.1 establishes the Wasserstein contraction of URF's adaptive query:

W2(qKp,qfwd)≤(1−p)W2(q,qfwd)W_2(qK_p,q_{\mathrm{fwd}})\leq(1-p)W_2(q,q_{\mathrm{fwd}})

Together, these properties motivate URF's continuous linear sample-wise adaptation without requiring a manually selected switching step. We would welcome suggestions for additional switching baselines and will make every effort to complete the requested experiments within the rebuttal period.

AuthorsOpenReview ↗

Author Response (2/2)

[Q3] DiffusionNFT Reproducibility and Baseline Configuration

We thank the reviewer for raising concerns about DiffusionNFT reproducibility.

To improve reproducibility, we have added configuration details in Tables 4 and 5 of Appendix B.3, specifying the baseline initialization and task-specific training, teacher, and evaluation CFG settings. We follow the official DiffusionNFT implementation's native CFG-free setting (CFG 1.0) for the standalone baseline. All methods are evaluated using the corresponding benchmark prompts, with training costs reported in GPU-hours.

Differences from previously reported baseline scores may arise from inference randomness, initial noise sampling, and implementation-specific numerical procedures. We have clarified the evaluation configurations to make these comparisons more transparent.

Importantly, Table 1 shows that URF outperforms Multi-Task NFT across all three optimized rewards: GenEval (0.963 vs. 0.946), OCR (0.963 vs. 0.951), and PickScore (24.003 vs. 22.490).

Furthermore, the additional matched-compute, multi-seed experiments in [W1] consistently favor URF over NFT, providing complementary evidence of its effectiveness across independent training runs.


[Q4] Clarification of Training and Evaluation CFG

We have added a configuration table in Appendix B.3 that separately reports training CFG, teacher CFG, evaluation CFG, and initialization for the full-step single-reward experiments.


[Q5] Clarification of Convergence Speedup

We compute speedup based on the training time required to reach each baseline's peak reward, measured within 200 GPU-hours for single-reward and 300 GPU-hours for multi-reward training. We agree that the speedup criterion depends on training duration and reward fluctuations. Even beyond this speedup metric, URF's superior training efficiency is evident from the following results:

  • Multi-reward training (Fig. 1): Faster convergence across all three optimized rewards.
  • Single-reward training (Fig. 6): Faster reward improvement and higher final performance.
  • Fixed-budget comparison ([W1]): Under 50 GPU-hours, URF outperforms OPD by 0.120 on GenEval and 0.052 on OCR across two additional seeds, including reward-related computational costs.

[Q6] Clarification of the DiffusionNFT Limit at p = 1

We thank the reviewer for highlighting this distinction. At pi=1p_i=1, URF recovers the NFT-positive target, as DiffusionNFT itself follows only the positive-reward branch when pi=1p_i=1. We have revised Figure 3 to highlight the NFT-positive velocity target and show that URF follows this target at pi=1p_i=1.


[Q7] Clarification of Figure 6 Caption

We thank the reviewer for pointing this out. The caption's description of the experimental comparisons is accurate. DiffusionNFT trained without prior distillation converges substantially more slowly and does not reach OPD's peak reward within the 200-GPU-hour budget.

Additionally, we have corrected the reward ordering in the caption.

Dear Reviewer qz9Y,

Thank you again for your time and constructive comments. As the discussion period is approaching its close, we wanted to follow up and check whether our responses have addressed your concerns. If any concerns remain, we would be glad to address them within the remaining time. Thank you very much for your effort in reviewing our work.

Reviewer_nJUeOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes URF, a framework that interpolates between DiffusionOPD and DiffusionNFT through a unified reward-adaptive objective. Multi-reward experiments on SD3.5-M demonstrate improvements over both baselines.

Critical strengths
  • The paper addresses a practically relevant problem: combining dense teacher supervision with reward-driven optimization to improve the efficiency and performance of diffusion post-training.

  • The experiments cover multi-reward training, single-reward optimization, four-step distillation, and component ablations, providing a reasonably comprehensive evaluation within the DiffusionOPD/DiffusionNFT setting.

Critical weaknesses
  • My main concern is that the paper's claims exceed its empirical scope: it remains unclear whether the key observations generalize beyond the specific DiffusionOPD and DiffusionNFT pair. The core motivational analyses focus heavily on DiffusionNFT's negative-sample dynamics and the OPD-to-NFT transition, which may not represent broader diffusion RL or distillation-to-RL pipelines. While additional baselines are tested, they do not sufficiently substantiate the framework's broader applicability. To convincingly demonstrate generality, experiments combining teacher-guided distillation with alternative diffusion RL objectives (e.g., Reinforce Adjoint Matching / RAM) would be essential.

  • The performance gains do not convincingly justify the additional reliance on online reward evaluation. A key advantage of DiffusionOPD is that it provides dense supervision through the teacher’s velocity field without requiring online reward evaluation during distillation. URF reintroduces this requirement while retaining teacher supervision, yet Table 1 shows only marginal improvements over DiffusionOPD on most metrics, with OCR being the main exception. The practical benefit of this additional dependency therefore remains insufficiently established.

Questions for authors
  1. To what extent do the negative-sample analysis and transition instability observed for DiffusionNFT generalize to other diffusion RL methods? Can you provide evidence from integrating teacher-guided distillation with another RL method, such as Reinforce Adjoint Matching (RAM), to substantiate the claimed generality beyond the OPD/NFT pair?

  2. Given that DiffusionOPD provides dense teacher supervision without online reward evaluation during distillation, what practical gains justify reintroducing this requirement in URF? Beyond the OCR improvement in Table 1, can you demonstrate consistent benefits over DiffusionOPD under matched total compute budgets, including reward evaluation and reward-anchor construction?

  3. The discussion of related work could be strengthened, particularly the analysis of how the proposed method relates to and differs from existing approaches.

5 comments
AuthorsOpenReview ↗

Author Response (1/2)

We sincerely appreciate the reviewer's thoughtful evaluation and constructive feedback.

In response, we provide additional experimental and theoretical analyses on the limitations of integrating alternative RL methods [Q1] and the necessity of online reward evaluation [Q2], and expand the discussion of related work [Q3]. We address each concern below and would be happy to clarify any remaining questions and make every effort to conduct additional experiments if needed.

[Q1] Generality Beyond DiffusionNFT: Integration with RAM

We thank the reviewer for suggesting an evaluation with an alternative RL objective. We first investigate RAM integration experimentally [Q1-1], analyze the structural challenges of direct integration [Q1-2], and explain why OPD/NFT is particularly suitable for URF [Q1-3].

[Q1-1] Experimental Comparison with Alternative RL Integration

We investigate the integration of Reinforce Adjoint Matching (RAM) with teacher-guided distillation by training the following variants on OCR for 50 GPU-hours under a matched evaluation protocol.

Let LT\mathcal{L}_T and LR\mathcal{L}_R denote the OPD and RAM losses evaluated at their respective native queries, and let pip_i denote URF's sample-wise reward preference. For Target Replacement, we retain URF's adaptive query and TG-negative correction, replacing only the NFT-positive target:

vhyb=(1−pi)vTG−+pivRAMv_{\mathrm{hyb}}=(1-p_i)v_{\mathrm{TG-}}+p_i v_{\mathrm{RAM}}

MethodObjective / TargetOCR (50 GPU-hours)
DiffusionOPDLT\mathcal{L}_T0.917
RAMLR\mathcal{L}_R0.942
OPD + RAM (Adaptive)(1−pi)LT+piLR(1-p_i)\mathcal{L}_T+p_i\mathcal{L}_R0.874
URF + RAM (Target Replacement)MSE to vhybv_{\mathrm{hyb}}0.671
URF (Ours)LURF\mathcal{L}_{\mathrm{URF}}0.952

URF achieves the best performance (0.952), while standalone RAM remains competitive (0.942). However, combining RAM with OPD reduces performance to 0.874, and direct target replacement further decreases it to 0.671. We discuss potential structural explanations below.

[Q1-2] Challenges in Combining RAM with Teacher-Guided Distillation

First, loss of second-order reward-attraction control. Appendix A.3 establishes that URF's TG-negative correction suppresses first-order reward attraction. For pi<0.5p_i<0.5, the normalized update component along dt=(ut−vold)/βd_t=(u_t-v_{\mathrm{old}})/\beta satisfies

sURF≤(1−pi)2aT+3pi2.s_{\mathrm{URF}}\leq(1-p_i)^2a_T+3p_i^2.

Unlike NFT, RAM uses a frozen reference and signed group-relative advantage:

vRAM=vref+λAi(ut−vold).v_{\mathrm{RAM}}=v_{\mathrm{ref}}+\lambda A_i(u_t-v_{\mathrm{old}}).

Replacing the NFT-positive target with RAM gives the exact relation

\quad \Delta_i= \frac{\langle v_{\mathrm{ref}}-v_{\mathrm{old}},d_t\rangle}{\|d_t\|^2} +\lambda\beta A_i-1.$$ Unlike NFT, RAM introduces an additional $\Delta_i$ arising from its frozen-reference offset and signed advantage scaling, so its reward updates are not necessarily aligned with URF's old-policy-centered correction. Consequently, ***the first-order cancellation is no longer guaranteed***, allowing weak reward evidence to cause excessive attraction or conflicting updates. ***Second, conflicting reference velocities.*** OPD/NFT shares the current old-policy reference, whereas ***RAM introduces an additional frozen reference.*** Consequently, direct integration involves three potentially different velocities, $v_T$, $v_{\mathrm{old}}$, and $v_{\mathrm{ref}}$. In particular, the component of $v_{\mathrm{ref}}-v_{\mathrm{old}}$ orthogonal to $d_t$ cannot be canceled by URF's TG-negative correction, which acts only along $d_t$. This mismatch may produce competing supervision directions as the policy evolves. Together, these properties offer plausible structural explanations for the observed degradation, without implying that RAM itself is unstable.
AuthorsOpenReview ↗

Author Response (2/2)

[Q1-3] Why Is the OPD/NFT Combination Particularly Suitable?

Among various distillation–RL combinations, OPD and NFT offer both recently demonstrated efficiency and two key structural advantages for URF, as detailed below.

First, compatible velocity geometry. NFT constructs symmetric positive and negative targets around the same old policy:

vNFT±=vold±dt.v_{\mathrm{NFT}\pm}=v_{\mathrm{old}}\pm d_t.

This symmetry provides a common reward-supported direction for URF's TG-negative correction, enabling the first-order attraction suppression established in Appendix A.3 without introducing an additional reference velocity.

Second, compatible query–target construction. OPD supervises student-visited states, whereas NFT constructs reward supervision from forward-noised generated endpoints. These naturally define the two query and velocity endpoints of URF. Moreover, Appendix A.4 shows that changing the reward-derived velocity target also changes the finite-step optimal query through a disagreement-dependent term. Therefore, replacing only the RL target does not generally preserve the original coupled query–target construction.

Thus, NFT is particularly suitable for the current URF formulation because its old-policy-centered velocity geometry and endpoint-derived queries are jointly compatible with URF's correction and interpolation mechanisms.


[Q2] Necessity and Benefits of Online Reward Evaluation

We thank the reviewer for raising this important point. Our goal is not merely teacher-guided distillation, but to unify distillation and reward-driven RL through a single reward-adaptive objective. Since teacher-guided distillation alone is generally limited by teacher performance (Table 1), switching to RL is a natural approach, but explicit stage-wise switching can cause instability and collapse (Fig. 2(a)).

By introducing online reward feedback at an additional computational cost, URF enables an automatic transition from distillation to RL, achieving faster convergence than DiffusionOPD and surpassing teacher performance without the collapse observed in stage-wise switching.

[Q2-1] On Marginal Performance Gains

We acknowledge that several improvements in Table 1 are small. Nevertheless, URF ranks first or second across all eight evaluation metrics, achieving the highest normalized average score of 0.996, compared with 0.984 for DiffusionOPD. Such gains are particularly meaningful in the high-performance regime.

More importantly, Figure 6(a–c) demonstrates substantially faster convergence and higher final rewards across all three single-reward tasks: GenEval, OCR, and PickScore.

[Q2-2] Additional Multi-Seed Experiments Under Matched Compute

To further assess training efficiency and robustness, we conducted additional experiments with two independent seeds (123 and 456), allocating 50 GPU-hours to each method. The comparisons use matched evaluation settings and account for online reward evaluation and reward-anchor construction.

SeedGenEval URFGenEval OPDGenEval NFTOCR URFOCR OPDOCR NFT
1230.9590.8430.6160.9520.9170.923
4560.9520.8280.5240.9480.8780.901
Mean0.9560.8360.5700.9500.8980.912

URF consistently outperforms OPD across both additional seeds, with average improvements of 0.120 on GenEval and 0.052 on OCR.

[Q3] Relation to Existing Diffusion Post-Training Methods

We thank the reviewer for this suggestion. We have expanded the Introduction to discuss recent related methods, including AWM, RAM, RMMD, and DreOPD. While these approaches explore reward-based regression, distribution matching, or teacher-guided extrapolation, URF differs by jointly determining where to optimize and what velocity target to follow through a shared sample-wise reward-preference weight.

We have clarified this distinction and refined our novelty claim accordingly in the revised paper.

Dear Reviewer nJUe,

Thank you again for your time and constructive comments. As the discussion period is approaching its close, we wanted to follow up and check whether our responses have addressed your concerns. If any concerns remain, we would be glad to address them within the remaining time. Thank you very much for your effort in reviewing our work.

Reviewer_7QirOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a Unified Reward-adaptive Framework (URF) for post-training text-to-image flow-matching/diffusion models, aiming to combine on-policy teacher-guided distillation (DiffusionOPD) with forward-process reinforcement learning (DiffusionNFT). Rather than switching sequentially from distillation to RL, URF introduces a sample-wise weight pi∈[0,1]p_i \in [0, 1] based on the generated sample's reward relative to initial-student and teacher anchors. Using a linear convex combination of teacher and reward objectives (Eq. 7), the authors derive closed-form interpolations for both the query state ztqz_t^q and the target velocity v∗v^*, which are then optimized via standard Flow Matching loss (Eq. 19). Experiments on SD3.5-M across GenEval, OCR, and preference rewards demonstrate faster convergence and strong multi-reward performance compared to standalone OPD and NFT baselines.

Critical strengths
  • Well-Motivated Problem & Practical Direction: Bridging dense teacher supervision (distillation) and scalar reward optimization (RL) without brittle stage-wise switching schedules addresses a clear practical bottleneck in diffusion post-training.
  • Self-Contained Presentation: The main text clearly lays out how the query location ("where to optimize") and velocity target ("what to optimize") are instantiated within the standard flow-matching regression loss (Eq. 19).
  • Strong Empirical Speedups: Across single-reward, multi-reward, and 4-step distillation settings on SD3.5-M, the method demonstrates consistent wall-clock efficiency gains (1.7×1.7\times–10.4×10.4\times) and avoids the early collapse seen in naive sequential switching or low-prompt NFT baselines.
Critical weaknesses
  • Simplistic Objective Formulation & Ad-Hoc Weighting (pip_i): The core "unified objective" in is simply a linear convex combination of the two prior frameworks. Particularly for the velocity target, defining the evidence scores as negative squared L2 distances trivially yields a linear interpolation of the two target velocities. There is limited theoretical justification for why a simple linear combination is the optimal mechanism to unify distillation and RL, or why the specific clipped linear form of pip_i is principled from an optimization perspective.

  • Unclear Convergence Dynamics & Conflict Between Objectives: Because URF simultaneously pulls the student toward a frozen teacher vTv_T and an evolving reward-driven target v∗NFTv*_{NFT}, it is unclear whether the two objectives will keep being different (fighting each other) during training or if the procedure is guaranteed to converge to a well-defined fixed point (vTv_T and v∗NFTv*_{NFT} are eventually the same).

  • Evaluation Scope & Lack of Distributional / Diversity Metrics: While the paper reports scalar reward metrics (GenEval, OCR, PickScore, etc.), RL post-training is well-known to suffer from reward over-optimization and diversity collapse. The evaluation lacks standard distributional image quality or diversity metrics (such as FID on standard reference sets or inter-sample diversity scores) to verify that high reward scores do not come at the expense of overall generative diversity. Additionally, the main results in Table 1 are reported for a single seed without standard deviations.

Questions for authors
  • Justification for the Form of pip_i and Linear Interpolation: Beyond simplicity, is there a theoretical reason why the linear combination and the piecewise linear clipping of pip_i between RLR_L and RHR_H should be preferred over non-linear schedules or uncertainty-based weighting? What happens when the student surpasses the teacher (R0,i>RHR_{0,i} > R_H) early in training? Does hard-clipping pi=1p_i = 1 re-introduce the instability of pure DiffusionNFT on subsequent updates?

  • Objective Conflict & Fixed-Point Convergence: Since voldv^{\text{old}} is continuously updated via EMA while vTv_T remains frozen, do the teacher and reward velocity targets conflict when pi∈(0,1)p_i \in (0, 1), and can the authors characterize what fixed point Algorithm 1 converges to?

  • Distributional Quality (FID) & Multi-Seed Robustness: Can the authors provide FID (or diversity metrics) alongside the reward scores in Table 1 to confirm that URF does not suffer from reward hacking or mode collapse, as well as variance across multiple random seeds?

5 comments
AuthorsOpenReview ↗

Author Response (1/3)

We sincerely appreciate the reviewer's careful evaluation and constructive feedback.

In response, we provide additional theoretical analyses and experiments on reward calibration and clipping stability [Q1], objective compatibility and fixed-point convergence [Q2], and diversity and multi-seed robustness [Q3]. We address each concern below and would be happy to clarify any remaining questions and make every effort to conduct additional experiments if needed.

AuthorsOpenReview ↗

Author Response (2/3)

[Q1] Theoretical Justification of Linear Interpolation and Reward Calibration

We thank the reviewer for these valuable questions. We address the theoretical justification of unified objective [Q1-1], linear calibration [Q1-2], teacher surpassing and calibration ablations [Q1-3] and hard-clipping stability [Q1-4].

[Q1-1] Justification of the Unified Objective

Our unified objective can be motivated as the Lagrangian relaxation of a constrained teacher-guided optimization problem, rather than an arbitrary interpolation of velocity targets.

Let PTP_T and PRP_R denote the teacher- and reward-derived joint query–transition distributions. We seek to minimize deviation from teacher supervision while constraining disagreement with reward supervision:

\quad \mathrm{s.t.}\quad \mathrm{KL}(Q\parallel P_R)\leq\epsilon.$$ Introducing a Lagrange multiplier $\lambda\geq0$ and defining $p=\lambda/(1+\lambda)$ yields, up to a positive scaling and an additive constant, $$J_p(Q)=(1-p)\mathrm{KL}(Q\parallel P_T) +p\mathrm{KL}(Q\parallel P_R).$$ This objective defines a single joint target distribution, $Q^\star\propto P_T^{1-p}P_R^p$. As derived in Appendix A.4, the KL chain rule yields coupled query-distribution and velocity-target optimization, with the exact conditional velocity optimum $$𝐯^{\star}=(1-p)𝐯_{\mathrm{TG-}}+p𝐯_{\mathrm{NFT+}}$$ ***Thus, linear velocity interpolation follows from a joint probabilistic objective rather than being independently imposed.*** In URF, we adopt linear reward calibration for $p$, as justified in [Q1-2]. ### **[Q1-2] Justification of Linear Reward Calibration** ### ***Linear calibration provides reward-proportional query contraction and minimizes worst-case sensitivity to normalized reward perturbations.*** Let $s$ denote normalized reward progress, with $p=s$. Appendix A.1 establishes $$W_2(qK_s,q_{\mathrm{fwd}})\leq(1-s)W_2(q,q_{\mathrm{fwd}})$$ Thus, the query distribution contracts toward the forward-process reference at a rate controlled by reward progress. Furthermore, for any Lipschitz-continuous calibration $p=f(s)$ satisfying $f(0)=0$ and $f(1)=1$, its Lipschitz constant satisfies $L_f\geq1$. The linear mapping $f(s)=s$ attains this minimum. Appendix A.3 additionally establishes second-order suppression of weak reward attraction: $$s_i\leq(1-s)^2a_T+3s^2,\qquad s<0.5$$ Therefore, ***linear calibration provides a controlled transition toward reward supervision while suppressing additional reward attraction quadratically in the low-preference regime.*** To examine the effects of nonlinear calibration and hard switching, we evaluated the following family (see Appendix C, Fig. 9 in the revised paper for the full training curves): $$p_\gamma(s)=\frac{s^\gamma}{s^\gamma+(1-s)^\gamma}$$ | Calibration | Final OCR Score | |---|---:| | $\gamma=0.5$ | 0.8981 | | **$\gamma=1$ (URF)** | **0.9608** | | $\gamma=2$ | 0.8812 | | $\gamma\to\infty$ (Hard Switching) | 0.8641 | The original linear calibration achieves the highest performance, outperforming both nonlinear alternatives and deterministic sample-wise hard switching. ### **[Q1-3] Teacher Early Surpassing** ### ***If the initial student already outperforms the teacher***, the reward calibration interval becomes non-positive, and ***we conservatively set $p=0$ as a fallback to teacher-guided supervision***. This is a stability-oriented safeguard that prevents unreliable reward calibration and affects only 5.59% of all samples. ### **[Q1-4] Stability Under Hard Clipping** ### We thank the reviewer for raising this important concern. We attribute the instability observed in stage-wise switching to the abrupt change in the optimization target distribution from teacher-guided supervision ($P_T$) to the reward-tilted distribution ($P_{\mathrm{ref}}\exp(R)$). ***In contrast, URF enables a continuous transition under a single unified reward-adaptive objective.*** Importantly, not all samples reach $p=1$, even in later training stages. To clarify how reward preferences evolve during training, we additionally analyze the distribution of $p$: | Weight | Initial (1) | 1–85 | 86–170 | 171–255 | 256–340 | 341–425 | 426–509 | Final (509) | |---|---:|---:|---:|---:|---:|---:|---:|---:| | $p=0$ | 49.85% | 19.6% | 11.5% | 9.8% | 8.4% | 7.9% | 6.9% | 8.3% | | $0<p<1$ | 50.15% | 70.3% | 57.3% | 44.2% | 40.9% | 39.9% | 39.7% | 37.5% | | $p=1$ | 0.0% | 10.1% | 31.2% | 46.0% | 50.7% | 52.2% | 53.4% | 54.2% | (See Appendix C, Fig. 8 in the revised paper for detailed training dynamics.) As shown in Fig. 8 (top), the proportion of samples with $p=0$ decreases from 49.85% to 8.3%, while $p=1$ increases from 0% to 54.2%. Thus, ***URF progressively shifts toward reward-driven optimization under a single unified reward-adaptive objective***, while 45.8% of samples still retain $p<1$ at the final iteration.
AuthorsOpenReview ↗

Author Response (3/3)

[Q2] Objective Conflict and Fixed-Point Convergence

We thank the reviewer for raising this important question. As established in [Q1-1], URF combines teacher- and reward-derived supervision into a single unified objective. We further characterize (i) the unified population-level target and (ii) self-consistent fixed-point convergence of the coupled student–EMA dynamics.

(i) Unified population-level target. Let 𝐮i𝐮_i denote the sample-wise URF velocity target derived in [Q1-1]. For a fixed EMA policy θˉ\bar\theta, the expected regression objective admits a single optimal population-level target:

𝐯θˉ†(z,t,c)=E[𝐮i∣ziq=z,ti=t,ci=c;θˉ].𝐯^\dagger_{\bar\theta}(z,t,c)=E[𝐮_i\mid z_i^q=z,t_i=t,c_i=c;\bar\theta].

Indeed, the regression loss decomposes exactly as

L(θ;θˉ)=Ez,t,c[d2(𝐯θ(z,t,c),𝐯θˉ†(z,t,c))]+Cθˉ.L(\theta;\bar\theta)=E_{z,t,c}[d^2(𝐯_\theta(z,t,c),𝐯^\dagger_{\bar\theta}(z,t,c))]+C_{\bar\theta}.

Here, dd denotes Euclidean distance, and CθˉC_{\bar\theta} is independent of θ\theta.

Therefore, sample-wise reward preferences collectively define a single population-level target that minimizes the expected regression loss. This target is uniquely defined on the sampled query support and evolves with the EMA policy.

(ii) Self-consistent fixed-point convergence of the coupled student–EMA dynamics. Since the population target evolves with the EMA policy, we characterize the student–EMA dynamics through a self-consistent equilibrium. Consider the idealized full-gradient updates:

θk+1=θk−α∇θL(θk;θˉk).\theta_{k+1}=\theta_k-\alpha\nabla_\theta L(\theta_k;\bar\theta_k).

θˉk+1=(1−η)θˉk+ηθk+1.\bar\theta_{k+1}=(1-\eta)\bar\theta_k+\eta\theta_{k+1}.

At a fixed point, the student and EMA policies coincide, and the expected gradient vanishes:

θ⋆=θˉ⋆,∇θL(θ⋆;θ⋆)=0.\theta^\star=\bar\theta^\star,\qquad \nabla_\theta L(\theta^\star;\theta^\star)=0.

To establish a sufficient convergence condition, assume that the expected gradient is locally μ\mu-strongly monotone and LgL_g-Lipschitz with respect to the student parameters, and CC-Lipschitz with respect to the EMA parameters.

Define

ρ=1−2αμ+α2Lg2+αC.\rho=\sqrt{1-2\alpha\mu+\alpha^2L_g^2}+\alpha C.

If ρ<1\rho<1 and 0<η≤10<\eta\leq1, then, around an existing fixed point within an invariant neighborhood, the coupled updates satisfy

δk+1≤(1−η+ηρ)δk.\delta_{k+1}\leq(1-\eta+\eta\rho)\delta_k.

Here, δk\delta_k denotes the maximum Euclidean distance of the student and EMA parameters from the fixed point.

Since 1−η+ηρ<11-\eta+\eta\rho<1, the coupled student–EMA dynamics converge linearly to a self-consistent fixed point.

Together, these results establish a unified population-level optimization target and characterize the fixed-point convergence of the coupled student–EMA dynamics.


[Q3] Distributional Quality and Multi-Seed Robustness

[Q3-1] Distributional Quality and Diversity

To directly assess diversity preservation, we additionally measured LPIPS-Div and DreamSim-Div. We randomly selected 30 prompts and generated 12 images per prompt for each method.

MethodLPIPS-Div ↑DreamSim-Div ↑
Base SD3.5-M0.5980.684
DiffusionNFT0.3400.180
DiffusionOPD0.3840.238
URF0.399‾\underline{0.399}0.294‾\underline{0.294}

Although all post-training methods exhibit reduced diversity relative to the pretrained base model, URF achieves the highest LPIPS-Div and DreamSim-Div among the post-training methods, indicating better diversity preservation than OPD and NFT under the evaluated protocol.

Furthermore, URF ranks first or second across five additional Model-Based metrics beyond the three training objectives (Table 1), suggesting that URF mitigates diversity loss and reward over-optimization while preserving broader generative quality.

[Q3-2] Multi-Seed Robustness

We acknowledge that the main results were initially reported using a single training seed. To assess robustness, we conducted additional experiments using two independent seeds (123 and 456) under matched CFG settings and a fixed budget of 50 GPU-hours.

SeedGenEval URFGenEval OPDGenEval NFTOCR URFOCR OPDOCR NFT
1230.9590.8430.6160.9520.9170.923
4560.9520.8280.5240.9480.8780.901
Mean0.9560.8360.5700.9500.8980.912

The corresponding mean and sample standard deviation across the two additional seeds are:

MetricURFOPDNFT
GenEval0.956 ± 0.0050.836 ± 0.0110.570 ± 0.065
OCR0.950 ± 0.0030.898 ± 0.0280.912 ± 0.016

Consequently, URF consistently outperforms both baselines across the additional seeds, with low observed variance under the controlled 50-GPU-hour setting.

Reviewer_7vVNOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes the Unified Reward-adaptive Framework (URF) for training text-to-image diffusion models. It combines teacher guidance from DiffusionOPD with reward-based training from DiffusionNFT. For each generated image, one reward-based weight controls where the student trains—the noisy image state—and what it learns—the target denoising update. Low-reward images rely more on teacher correction, while high-reward images provide their own training signal. This avoids choosing a fixed training step to switch from distillation to RL.

Critical strengths
  • Clear motivation. Figure 2 shows that switching from distillation to RL at different steps can cause a score drop or training collapse on the text-rendering task. This gives a concrete reason to adapt the update for each image.

  • Promising results. In multi-task training, URF scores 0.963 on both GenEval and OCR, compared with 0.955 and 0.938 for DiffusionOPD. The authors report training speedups of 1.7× on GenEval and 7.9× on OCR.

  • Useful ablations. Figure 7 tests the choice of training state and target separately. It also shows that teacher correction for low-reward samples helps on GenEval compared with simply mixing the two training losses.

Critical weaknesses
  • The role of the reward weight is unclear. When the teacher scores no better than the initial student on a prompt, URF sets the weight to zero and follows the teacher. The paper does not show how often this happens or how many samples actually mix both signals.

  • There is a gap between theory and practice. The practical sampler matches the theoretical query distribution when the two distributions are Gaussian with the same covariance. For real student trajectories, it is an approximation. Its advantage over a simpler sampling rule remains unclear.

  • The speed comparison has limits. URF and DiffusionOPD use different learning rates and different GenEval training guidance settings. The main results also use one model family and one training seed, so the size of the speedup is uncertain.

Questions for authors
  • For each task, how often is the reward weight zero, one, or somewhere in between? Would a simple switch between teacher and reward updates work as well?

  • Using the same reward weight, how would URF compare with randomly choosing either a student state or a newly noised final image?

  • Do the speedups hold across training seeds with matched guidance settings? Does the reported time include computing the initial student and teacher reward scores?

3 comments
AuthorsOpenReview ↗

Author Response (1/2)

We sincerely appreciate the reviewer's careful evaluation and constructive feedback.

In response, we conducted additional analyses and experiments on reward-preference dynamics and switching strategies [Q1], theoretical justification of the adaptive query and hard-switching ablations [Q2], and matched-CFG multi-seed evaluation and end-to-end computational efficiency [Q3]. We address each concern below and would be happy to clarify any remaining questions and make every effort to conduct additional experiments if needed.

[Q1] Reward-Preference Weight Distribution and Switching Ablations

We thank the reviewer for constructive comments. To clarify the role of the reward-preference weight pp, we conducted an additional analysis of its distribution throughout training:

WeightInitial (1)1–8586–170171–255256–340341–425426–509Final (509)
p=0p=049.85%19.6%11.5%9.8%8.4%7.9%6.9%8.3%
0<p<10<p<150.15%70.3%57.3%44.2%40.9%39.9%39.7%37.5%
p=1p=10.0%10.1%31.2%46.0%50.7%52.2%53.4%54.2%

(more detailed graph in Appendix C, Fig. 8 in the revised paper).

As shown in Fig. 8 (top), the proportion of samples with p=0p=0 decreases substantially from 49.85% to 8.3%, while the proportion with p=1p=1 increases from 0% to 54.2%. This confirms that URF progressively shifts from teacher-guided supervision toward reward-driven optimization as training proceeds, while maintaining intermediate weights for samples that benefit from both signals.

Importantly, only 5.59% of all samples trigger the fallback condition where the teacher reward does not exceed the initial-student reward. In such cases, we conservatively retain teacher-guided supervision (p=0p=0) to avoid relying on an ill-defined reward calibration interval and potentially unreliable positive reward supervision.

To further investigate whether continuous adaptation is necessary, we conducted additional switching ablations:

  • Sample-wise hard switching (Appendix C, Fig. 9 in the revised paper): URF (γ=1\gamma=1) achieves 0.9608, substantially outperforming sample-wise hard switching (0.8641).
  • Training-stage switching (Fig. 2(a)): Switching from OPD to NFT at steps 30, 60, or 90 produces schedule-dependent degradation or instability, whereas URF maintains stable improvement.

Together, these results demonstrate that continuous sample-wise adaptation outperforms both hard and stage-wise switching.

AuthorsOpenReview ↗

Author Response (2/2)

[Q2] Theory–Implementation Gap in Query Interpolation and Hard-Switching Ablation

We thank the reviewer for raising this important point. We acknowledge that the practical sampler is not generally identical to the theoretical KL barycenter. However, Appendix A.1 provides distribution-free guarantees beyond the shared-covariance Gaussian case.

Specifically, our OU-based sampler preserves both endpoints (p=0p=0 recovers the on-policy state, while p=1p=1 recovers forward noising) and guarantees Wasserstein contraction:

\leq (1-p)W_2(q,q_{\mathrm{fwd}}).$$ Thus, ***even for non-Gaussian student trajectories, the sampler provides a controlled, continuous transition toward the forward-process distribution***. (see Appendix A.1 for a detailed proof) To further examine the benefit of continuous adaptation, we conducted a hard-switching ablation (see Appendix C, Fig. 9 in the revised paper for the full training curves): | Method | Final OCR Score | |---|---:| | **URF ($\gamma=1$)** | **0.9608** | | Hard Switching ($\gamma\to\infty$) | 0.8641 | The results demonstrate that smooth, sample-wise joint adaptation substantially outperforms deterministic hard switching, supporting the practical benefit of continuous interpolation. --- ## **[Q3] Training Efficiency, Comparison Fairness, and Robustness** ## We thank the reviewer for raising these important concerns. We use evaluation CFG 1.0 only for the original DiffusionNFT baseline since DiffusionNFT is designed for CFG-free optimization and achieved higher reward scores under CFG 1.0 than under stronger guidance in our experiments. We have clarified the evaluation CFG settings in the figures and Appendix B.3 in the revised paper. ### **[Q3-1] Hyperparameter Fairness and Multi-Seed Evaluation** ### To ensure a more controlled comparison, we additionally evaluate URF, NFT and OPD under matched student and teacher CFG settings of 4.5, using two additional training seeds (123 and 456). Each run is evaluated under a fixed budget of 50 GPU-hours, with otherwise consistent data and evaluation protocols. The results are summarized below. | Seed | GenEval URF | GenEval OPD | GenEval NFT | OCR URF | OCR OPD | OCR NFT | |---|---|---|---|---|---|---| | 123 | **0.959** | 0.843 | 0.616 | **0.952** | 0.917 | 0.923 | | 456 | **0.952** | 0.828 | 0.524 | **0.948** | 0.878 | 0.901 | | Mean | **0.956** | 0.836 | 0.570 | **0.950** | 0.898 | 0.912 | These results demonstrate that ***URF's advantages persist across multiple seeds under controlled CFG and learning-rate settings***. ### **[Q3-2] End-to-End Computational Cost** ### ***The reported training times in all figures include reward evaluation and the computation of initial-student and teacher reward anchors.*** The measured computational costs in multi-reward training are summarized below: | Method | Images/update | Time/update | Time to OCR 0.90 | |---|---:|---:|---:| | Official OPD | 72 | 183 s | 69.68 GPU-h | | URF (Routed 72) | 72 | 299 s | 25.01 GPU-h | Thus, despite a 1.63x higher per-update cost, URF reaches an OCR score of 0.90 approximately 2.79x faster in total GPU time.

Dear Reviewer 7vVN,

Thank you again for your time and constructive comments. As the discussion period is approaching its close, we wanted to follow up and check whether our responses have addressed your concerns. If any concerns remain, we would be glad to address them within the remaining time. Thank you very much for your effort in reviewing our work.

Comments on the paper

1 comment

Discussion Summary

We sincerely thank all reviewers and the Area Chair for their valuable feedback and constructive discussions.

Below, we summarize our additional theoretical analyses, experiments, and clarifications addressing the reviewers' key concerns.


[ Initial Rebuttal ]

1. Reward-Preference Weight Distribution

Refer: Reviewer 7vVN, Q1.

We showed that p=0p=0 decreases from 49.85% to 8.3%, while p=1p=1 increases from 0% to 54.2%, with only 5.59% triggering teacher-guided fallback (Appendix C, Fig. 8), confirming URF's progressive transition toward reward-driven optimization with sample-wise adaptation.

2. Multi-Seed Robustness

Refer: Reviewer 7vVN, Q3-1.

Under matched settings and a fixed 50-GPU-hour budget, URF consistently outperforms DiffusionOPD and DiffusionNFT across two additional seeds, achieving mean scores of 0.956 vs. 0.836/0.570 on GenEval and 0.950 vs. 0.898/0.912 on OCR, respectively.

3. Theoretical and Experimental Justification

Refer: Reviewer 7Qir, Q1-1 & Q1-2.

We established a unified constrained objective, Wasserstein contraction, optimal Lipschitz sensitivity of linear calibration, and O(p2)O(p^2) suppression of additional reward attraction (Appendix A.1, A.3, A.4). Switching, interpolation, and calibration ablations (Figs. 2, 7, 9) further highlight the uniqueness of URF's joint query–target adaptation over simpler alternatives.

4. Initial Performance Differences and CFG Settings

Refer: Reviewer qz9Y, W2 & Q2.

We clarified that DiffusionNFT achieved stronger rewards under CFG-free settings, motivating our choice of this configuration, and discussed potential sources of DiffusionOPD's initial-score discrepancy, including evaluation stochasticity and the narrower y-axis in Fig. 6(c). We also expanded configuration details (Appendix B.3, Tables 4–5) and emphasized sustained post-convergence improvements beyond initial-score differences.

5. Further Theoretical and Experimental Analyses

Refer: Reviewer 7vVN, Q3-2; Reviewer 7Qir, Q2–Q3; Reviewer nJUe, Q1.

We clarified end-to-end GPU costs, strengthened unified objective, EMA convergence, and diversity analyses, and investigated RAM-based integrations and OPD/NFT compatibility.


[ Follow-up Responses ]

Following our rebuttal, reviewers nJUe and qz9Y raised further concerns, which we addressed in our final responses:

Reviewer nJUe

  • Q1. Generality Beyond OPD/NFT: We emphasized that evaluating all diffusion distillation and RL methods is beyond the scope of a single study. Our focus on OPD/NFT is not arbitrary, as their structural compatibility enables the first-order cancellation yielding O(p2)O(p^2) additional reward attraction, and we explicitly clarified the framework's scope and limitations in the revised manuscript.
  • Q2. Benefits Beyond Reward Supervision: We distinguished URF from existing reward-aware few-step distillation methods and highlighted ablations showing that simply combining reward and distillation objectives can lead to collapse.

Reviewer qz9Y

  • Q1. Training Efficiency: We provided multi-threshold time-to-target and AULC evaluations to demonstrate URF's overall training efficiency beyond single-threshold speedup metrics.
  • Q2. Initial Score Discrepancy: We clarified that the discrepancy's visual prominence is partly due to the narrower y-axis in Fig. 6(c), while emphasizing sustained post-convergence improvements across our experiments.
  • Q3. Switching Strategies: We highlighted our theoretical justification and extensive representative switching/interpolation ablations.

We sincerely hope that these additional analyses and clarifications have addressed the reviewers' remaining concerns, and greatly appreciate their thoughtful feedback, which has helped strengthen our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.