- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper considers multi-objective preference optimization using RL fine-tuning of diffusion models, where multiple reward functions guide updates of model parameters through their gradients. These gradients can differ in magnitude and direction which can lead to a reward dominating others, motivating methods such as MGDA and MARBLE that adaptively combine them. However, the paper illustrates through a toy example that noisy gradient estimates can prevent MGDA from reaching Pareto stationarity. It also identifies highly imbalanced reward weights during diffusion RL training. To address these two issues, the paper introduces two components:
-
Gradient smoothing: Rather than allowing noisy gradient estimates to immediately determine the update direction, the method maintains an exponential moving average of each reward’s past gradients.
-
Anchoring: To ensure that each reward receives a minimum weight, the method incorporates a preference anchor borrowed from SDMGrad.
Critical strengths
-
The paper is well-written and clearly structured with easy-to-follow flow.
-
The paper is well motivated. The toy experiment and gradient analysis show how noisy gradients can affect MGDA which provides a clear motivation for the proposed method.
-
The experiments and ablations support the paper’s main message by showing both smoothing and anchoring contribute to the improvements.
Critical weaknesses
-
Diffusion Blend (Cheng et al., 2026) is not cited in the related work, although it falls in the same broad category as Rewarded Soup.
-
The theory basically says that if STB’s combined direction becomes small and its gradient estimates are accurate, the model is approximately Pareto stationary. But will the direction actually become small when every objective must keep contributing because of the anchor?
-
The weights are recomputed every steps, with cached weights used in between. However, the theory only analyzes the full STB direction. It is unclear how the theoretical guarantee works in between.
-
There is no simple uniform-weight or equally weighted normalized-gradient baseline to show how much difference the proposed weighting makes.
-
It is unclear how the preference can be incorporated in this setting.
Reference: Cheng et al (2026). Diffusion Blend: Inference-Time Multi-Preference Alignment for Diffusion Models. International Conference on Learning Representations (ICLR).
Questions for authors
In addition to the weaknesses raised above, I have the following questions:
-
How much additional computation does STB require compared with the original DiffusionNFT training?
-
Can user preferences be incorporated through a nonuniform preference vector ?
-
STB performs worse than the released DiffusionNFT checkpoint on all three held-out metrics. To better understand this comparison:
- Did STB start from the base SD3.5-Medium model or the released DiffusionNFT checkpoint?
- Were the same training rewards and prompt datasets used for both?
- Were their training compute budgets matched? If so, how?
5 comments
Author Response continued
W5 / Q2: Can non-uniform preferences be incorporated?
Yes. For any positive preference vector summing to one, the effective weights are with , where is the geometry-dependent solution, so the anchor reserves allocation according to while retaining adaptive weighting. We verify this with a preference sweep that compares STB with linear scalarization (LS), which uses directly as fixed weights on the normalized gradients. Both continue training from STB checkpoint 450 for 80 updates under four preference vectors, ordered as GenEval / OCR / PickScore / CLIPScore / HPSv2.1: P1 = (0.450, 0.450, 0.033, 0.033, 0.033), P2 = (0.325, 0.325, 0.117, 0.117, 0.117), P3 = (0.175, 0.175, 0.217, 0.217, 0.217), and P4 = (0.050, 0.050, 0.300, 0.300, 0.300). Each entry reports STB / LS, and Start is checkpoint 450.
| Metric | Start | P1 | P2 | P3 | P4 |
|---|---|---|---|---|---|
| GenEval | 0.926 | 0.929 / 0.916 | 0.931 / 0.943 | 0.932 / 0.936 | 0.935 / 0.899 |
| OCR | 0.864 | 0.915 / 0.935 | 0.913 / 0.924 | 0.902 / 0.837 | 0.871 / 0.879 |
| PickScore | 22.92 | 22.84 / 22.75 | 22.97 / 22.94 | 22.94 / 22.95 | 23.03 / 23.12 |
| CLIPScore | 0.289 | 0.287 / 0.289 | 0.290 / 0.288 | 0.293 / 0.290 | 0.287 / 0.293 |
| HPSv2.1 | 0.329 | 0.316 / 0.308 | 0.331 / 0.326 | 0.327 / 0.329 | 0.332 / 0.335 |
| Aesthetic | 5.81 | 5.84 / 5.73 | 5.81 / 5.78 | 5.71 / 5.76 | 5.85 / 5.83 |
| ImgRwd | 1.44 | 1.38 / 1.29 | 1.45 / 1.42 | 1.41 / 1.42 | 1.44 / 1.47 |
| UniRwd | 3.35 | 3.28 / 3.24 | 3.40 / 3.39 | 3.37 / 3.35 | 3.40 / 3.44 |
From P1 to P4, the preference shifts from GenEval and OCR toward PickScore, CLIPScore, and HPSv2.1, and STB responds with higher PickScore (22.84 → 23.03) and HPSv2.1 (0.316 → 0.332) while OCR decreases (0.915 → 0.871). The floor concerns coefficients, not guaranteed reward gains.
Q1: Additional computation relative to DiffusionNFT.
MGDA and STB recompute the balancing weights once every updates. In the intervening steps, both methods reuse the most recently computed weights and perform a single backward pass on , incurring the same computational cost as a weighted-sum DiffusionNFT update, including rollout generation and reward evaluation. For a fair comparison, we therefore report the average per-step update time of DiffusionNFT, MGDA, and STB over 10 consecutive updates, accounting for the periodic weight recomputation overhead. Average times per update on the same 8-GH200 hardware.
| Method | Average time per update | Relative to DiffusionNFT |
|---|---|---|
| DiffusionNFT (weighted sum) | 500.5 s | 1.00× |
| MGDA | 545.3 s | 1.09× |
| STB | 559.4 s | 1.12× |
MGDA and STB add about 9% and 12% over DiffusionNFT, entirely from the updates that recompute the weights, and STB adds about 2.6% over MGDA. The total training time of STB is nevertheless shorter: the released DiffusionNFT model is trained for 1.7k iterations (about 236 h at the per-update time above), whereas STB uses 1,000 joint updates (about 155 h), roughly one third less.
Q3: Initialization.
STB and the controlled baselines start from base SD3.5-M with fresh LoRA, not the released DiffusionNFT checkpoint.
Q4: Training rewards and prompt datasets.
We use the same five reward types and prompt-source families (Pick-a-Pic, GenEval, OCR), but STB mixes prompts approximately 1:1:1 whereas DiffusionNFT uses a staged curriculum.
Q5: Training compute budgets.
Released DiffusionNFT is trained for 1.7k iterations versus 1,000 updates for STB, so compute is not matched and the comparison does not isolate the effect of the optimization method. We therefore compare STB against MGDA with the same backbone, data, rewards, and training budget, where STB improves all five optimized rewards and all three held-out metrics.
Follow-up on Reviewer Comments
Dear Reviewer nZ3y,
We have provided responses to your questions and comments. We would appreciate it if you could let us know whether you have any additional questions or concerns.
Thank you for your time and consideration.
Author Response
We thank the reviewer for the detailed questions on the theory, baselines, and training cost, which we address below.
W1: Missing reference to Diffusion Blend.
We will add Diffusion Blend alongside Rewarded Soups and distinguish inference-time preference alignment from STB's joint training-time optimization.
W2: Does the anchored direction actually become small?
Proposition 2 is a stationarity certificate, not a convergence proof, since an anchor can exclude a zero-norm combination available to unconstrained MGDA. Our additional analysis uses descent of the preference-weighted objective Fp=∑kpkfk. With controlled estimation, normalization–anchor mismatch, and solver errors, a large direction decreases Fp in proportion to its squared norm and the step size, and because Fp is bounded below, these decreases cannot continue indefinitely. For fixed smooth objectives, positive preferences, nonzero smoothed-gradient norms, and step sizes with divergent sum, the step-size-weighted average expected squared direction norm therefore vanishes if the averaged errors vanish, and Proposition 2 then gives a vanishing average squared Pareto residual. Persistent errors leave an error neighborhood. This analyzes an idealized procedure that recomputes weights every update, not last-iterate or full AdamW convergence. We will add the full assumptions and proof to the appendix.
The mismatch arises because normalization changes the coefficients representing the preference-weighted gradient, which may violate the anchor's lower bounds. For uniform p and λ=0.3, it is zero when the smallest smoothed-gradient norm is at least 23.1% of the mean. Within the first 1,000 steps, 100/101 recorded refreshes satisfy this condition (99.0%), and the remaining ratio is 22.8%. This supports compatibility along the recorded trajectory, not vanishing of all errors.
W3: What is guaranteed between refreshes?
We cache coefficients, not gradients. Each cached update differentiates ∑kαˉkLk using current mini-batch gradients without objective-wise smoothing or normalization, and for fixed p and λ the coefficient EMA preserves the anchored simplex and its coefficient floors (the cache is initialized at the feasible vector p). A pointwise certificate also holds: by simplex feasibility and the triangle inequality, the Pareto residual is at most the norm of the cached reward-gradient combination plus the largest objective-gradient estimation error. It is informative when both terms are small, but does not prove that cached directions become small or remain optimal for the current geometry.
W4: Uniform / equally weighted normalized-gradient baseline.
The Flow-GRPO row in Table 1 is a uniform-weight baseline. It is trained on the same five rewards with fixed equal weights (each reward is normalized to unit variance and the five are summed with weight 1/5 each), and STB outperforms it on all eight metrics. To isolate the effect of the weighting within our own training setup, we also add an equally weighted normalized-gradient baseline that fixes α=p uniformly, the λ→∞ limit of STB, while retaining smoothing and normalization.
The intermediate settings give similar performance with different trade-offs, rather than a unique optimum at 0.3. Fixed uniform weighting gives the highest PickScore, while STB at λ=0.3 improves the other seven metrics. This supports combining an anchor with geometry-dependent allocation.