- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper introduces a new reward fine-tuning method for diffusion models. Their method is based on a regression loss for a direction that aims to approximate the gradient of the value function. They mathematically derive a direction that is positively aligned with the gradient of the value function through a minimization problem that does not require reward gradients or backward integration. They experiment with fine-tuning Stable Diffusion 3.5 on various sets of rewards, both differentiable and nondifferentiable, individually and jointly, using different estimators with a similar underlying approach and adapting their method accordingly. They show better performance and efficiency than DiffusionNFT and FlowGRPO, popular fine-tuning methods, and ablate some of the design choices made for each setup.
Critical strengths
The proposed theoretical regression target is novel. The justification for its positive alignment with the gradient of the actual expected reward is neat: even though it relies on an assumption, the alignment holds when the corruption process is Gaussian, as at initialization or under an exponential tilt. It is a nice, elegant way to obtain an approximate reward gradient through a minimization problem without relying on reward gradients or adjoint methods.
Critical weaknesses
- The paper lacks clarity, particularly in its presentation of the background and its contextualization of the proposed method relative to other methods, e.g. the lack of a technical comparison with FDFO or heavy notation used to introduce the value function. It introduces theorems and fragments of theoretical analysis that do not seem connected to or necessary for the narrative. There is no clear explanation of the relationship between the theory and what is actually done in practice, e.g. Proposition 2 assumes posterior samples at a fixed noisy state, which neither implemented estimator provides. Several arbitrary practical choices are introduced without enough motivation in the main paper, although they are evaluated empirically through ablations. The paper also uses nonstandard terminology, “rollout bandwidth,” “trajectory bank,” “query weights”, without sufficiently explaining it, making the method unnecessarily difficult to follow.
- The experiments use DiffusionNFT as the main baseline but do not include subsequent methods that improve upon it, such as Reinforce Adjoint Matching [1]. The differences between the theoretical formulation and the practical implementation make it difficult to determine whether the efficiency gains come from a better estimator of the value-function gradient or from other design choices orthogonal to it, in particular setting the reference penalty coefficient to zero and reward dependent trajectory weights.
Questions for authors
-
Could you add newer baselines that improve upon DiffusionNFT, e.g., RAM and AdvantageFlow [2] ?
-
While Section 6.4 ablates trajectory weighting within the proposed method, does its efficiency advantage over the baselines persist under matched trajectory weighting and reference regularization?
-
Can you clarify the importance of the directional analysis and noisy state regression discussions, as well as their corresponding ablations?
-
Can you clarify the missing conditioning on the state in the practical estimation of the value-function gradient?
-
Can you add a theoretical comparison with FDFO, of which the proposed method appears to be a generalization to multiple samples?
3 comments
Q2. Trajectory weighting and reference regularization.
PickScore and OCR use uniform update weights (), without lowest-reward selection; their performance advantages are reported in W2. GenEval's discrete scores often saturate, motivating updates on the lowest-six trajectories while retaining the other rollouts of the same prompt for regression. With uniform updates, ours reaches GenEval 0.80/0.90 in 17.2/58.7 GPU-hours, versus official DiffusionNFT's 30.3/102.0. With uniform weights and no reference penalty for either method, reaching 0.80 takes 17.2 versus 35.7 GPU-hours for KL-free DiffusionNFT (Table 4). Under these matched settings, ours reaches OCR 0.90 in 3.2 versus 16.9 GPU-hours for DiffusionNFT (App. D.5).
Q3. Importance of directional analysis and noisy-state regression ablation.
Thank you for this question. Under a local linear approximation, the change in expected rollout reward is . The unregularized population coefficient of noisy-state regression therefore approaches as the perturbations shrink, without the clean-image covariance factor (App. A.1). Our clean-image regression yields the preconditioned direction (Proposition 2). Its motivation is the geometry of clean-image variation: the Gaussian subspace example shows how low-dimensional coordinates improve directional estimation with few reward evaluations (Eq. 7), and the ablation tests this using identical perturbations and reward differences (Fig. 3). We have clarified this reasoning in Sections 3.2 and 6.1.
Q4. Conditioning on in practical estimation.
Thank you for highlighting this point. Eq. (6) conditions on a fixed state . In principle, its conditional moments can be estimated by Monte Carlo from several independent stochastic completions of the same , but doing this at every training state multiplies the rollout cost, a key practical constraint in diffusion RL. Our reward-adaptive rollout avoids this cost in two ways, which we have clarified in Section 4 (theory in App. B.3):
- Independent rollouts with the same prompt (discrete rewards, GenEval). At the noise end the conditioning is exact: since , carries no information about the clean image, so the completions of a prompt from independent initial noises are posterior samples at , and their fit is the slope of Eq. (6) at for any output law (Prop. B.3(a)). The approximation is the reuse of this slope at later states. For a Gaussian output law, the reused slope is the average of the state-conditional slopes and still has positive inner product with the state-averaged value gradient (Prop. B.3(b)); its per-state error is zero at and grows toward the data end.
- Local rollouts from the main branch (continuous rewards, PickScore and OCR). Each training state is fitted on the ODE completions of the other states on its own stochastic trajectory, centered at its own completion. These are not posterior samples: the fit estimates the reward slope at the state's completion, with the perturbation spread in place of the posterior spread (App. B.3).
Section 6.3 (Fig. 5) supports this choice. On GenEval, whose scores often remain constant under local perturbations, local fits reach 0.84–0.85 at 28 GPU-hours versus 0.92 for independent rollouts; on PickScore, local rollouts lead independent ones by 0.4–0.7 at matched budgets.
Q5. Theoretical comparison with FDFO.
Thank you for this suggestion. We have expanded the theoretical comparison in Section 2. FDFO uses paired denoised-image differences, while our method jointly regresses reward differences on multiple image differences, accounting for correlations among them. With two nearby trajectories, our row-normalized ridge fit on the single image difference is a scalar multiple of FDFO's secant direction; jointly regressing over the independent rollouts of a prompt, together with update weights, extends the estimator to discrete rewards such as GenEval, which the FDFO paper does not report. Proposition 2 characterizes the population target of this joint fit: under its Gaussian-noising assumptions, is a covariance-preconditioned value-gradient direction.
Our existing PickScore comparison shows ours reaching 23.20 in 14.3 GPU-hours, versus 109.3 for FDFO (Fig. 2; original Fig. 1).
Thank you again for the thorough review. We hope these changes address your concerns, and we are happy to provide more detail.
Thank you for the detailed review and the concrete suggestions.
W1. Clarity, organization, and terminology.
Thank you for these suggestions. We have reorganized and simplified Section 3 to clarify the progression from deriving the policy gradient to estimating ∇Vt for the update. The presentation now follows three steps:
New Figure 1 illustrates one update from sampling through regression to the policy loss, with Algorithm 1 providing the full procedure.
We have also clarified the value-function notation and the relationship between the population assumptions and finite-rollout implementation. Detailed proofs and implementation settings are collected in the appendices. Table 5 maps each theoretical result to the design choice it supports, its scope and its test. The remaining practical choices are motivated where they are introduced and tested in ablations: the rollout type by reward continuity (Sec. 4; Fig. 5), the update weights by GenEval's saturation (Sec. 4; Table 4), and ridge with row normalization by coefficient stability (App. B.5); all other hyperparameters are shared across rewards (Table 11). We also replaced the nonstandard terms: "trajectory bank" → local rollouts from the main branch; "independent-seed group" → independent rollouts; "rollout bandwidth" is replaced by the choice between the two, stated in words; "query weights" → update weights ai. Section 4 follows sampling, joint regression, and policy update without additional names for the sample sets.
W2 / Q1. Additional baselines: RAM and AdvantageFlow.
RAM. We ran its official code on all three rewards using the same 8×A800 hardware and evaluation sets. Ours reaches OCR and GenEval 0.90 in 3.2 and 22.8 GPU-hours, versus RAM's 20.7 and 59.8 (Fig. 2; Tables 3 and 9). Ours also matches or exceeds RAM's published final scores with half or less of the compute (Table 9):
Entries show reward (GPU-hours). Published scores retain RAM's protocol; their schedule costs are measured or estimated on our 8×A800 hardware (≈: estimates from the reported 300/270 steps). Our PickScore and GenEval reproductions cover about half the published schedules and are read at compute close to ours.
AdvantageFlow. Since its code is not publicly available, we evaluate our checkpoints using its evaluation protocol and compare with the curves reported in its paper. Ours reaches PickScore/26 0.89 and OCR 0.90 in 34.8 and 3.2 GPU-hours, versus 47.7 and 20.1 for AdvantageFlow. This comparison uses our A800 runs and its published A100 results (Table 10; App. D.7).