- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies retraining-free multi-objective alignment of diffusion models. It replaces the standard KL-regularized RL objective with a step-level surrogate whose optimal policy is available in closed form, and shows that under this surrogate the optimal policy for a weighted reward is the normalized product of the single-objective optimal reverse kernels. With Gaussian kernels this yields a precision-weighted fusion of means and variances, requiring no reward access and no training per weight vector.
Critical strengths
-
Well-motivated problem and clear positioning. Multi-objective alignment under user-specific preference weights is practically important, and the paper clearly identifies the gap left by existing methods: per-weight retraining is expensive, guidance methods need reward access at inference, and existing reward-free fusion methods rely on unquantified approximations (Table 1).
-
Simple, closed-form and cheap fusion rule. Theorem 1 gives explicit precision-weighted expressions for the fused mean and variance via a short, transparent derivation. The resulting sampler needs no reward model, gradients or extra sampling, and costs roughly M times a single model, far below CoDe and RGG.
-
Unifying view of existing fusion samplers. The framework recovers DB-MPA as the equal-variance special case and provides an explicit justification for the DeRaDiff sampler. The derived step-level DPO loss with its asymmetric Δ_diff regularizer is a potentially interesting side contribution.
Critical weaknesses
W1 (Novelty). The step-level objective and its closed-form policy are taken from [1]; the geometric-mixture result for weighted rewards is already established for LLMs in [2]; the sampler is identical to DeRaDiff [3] and, under equal variances, to DB-MPA [4]. The only new element is the observation that is linear in the reward,which makes the mixture exact under the surrogate objective. The contribution is thus a re-derivation of an existing algorithm rather than a new method.
W2 (Validity of the theoretical claims). The "no approximation error" claim results from changing the objective (Eq. 3 instead of Eq. 1/2), not from resolving the original intractability; the gap between the surrogate optimum and the true KL-regularized optimum is never analyzed. Moreover, the assumptions of Theorem 1 do not hold in the paper's own experiments. The exact optimum of Eq. 3, , is not Gaussian in general; the trained models are Gaussian approximations, so the fusion carries an unquantified approximation error of the same kind the paper criticizes in [3, 4]. Variance heterogeneity is created by using different DDIM (1.0 vs. 0.8), which changes itself and breaks the cancellation (via ) in the proof of Theorem 1. In addition, is updated during training and CFG is applied per model before fusion; neither is covered by the theory.
W3 (Central claim untested). No comparison against models directly fine-tuned on ; no KL to the pretrained model or regularized objective values are reported, so neither exactness nor optimality is verified. From Table 4, the nearest single-objective model beats MSDDA on the weighted reward at (e.g., : vs. ), and DB-Extended outperforms MSDDA for , contradicting "consistently outperforms".
W4 (Scale and fairness). Only SDv1.5 (an out of data model architecture) and about 25 test prompts in the main setting. CoDe/RGG numbers (including inference time) are copied from [4] with different base models and hardware. DeRaDiff [3] and the original DB-MPA [4] (equal-variance setting) are not compared against, although the proposed sampler reduces to them.
References
[1] Y. Zeng, G. Liu, W. Ma, N. Yang, H. Zhang, and J. Wang. Token-level direct preference optimization. arXiv:2404.11999, 2024.
[2] R. Shi, Y. Chen, Y. Hu, A. Liu, H. Hajishirzi, N. A. Smith, and S. S. Du. Decoding-time language model alignment with multiple objectives. NeurIPS, 2024.
[3] R. D. S. Manujith, T. T. Tzun, K. Kawaguchi, and Y. Zhang. DeRaDiff: Denoising time realignment of diffusion models. arXiv:2601.20198, 2026.
[4] M. Cheng, F. Doudi, D. Kalathil, M. Ghavamzadeh, and P. R. Kumar. Diffusion blend: Inference-time multi-preference alignment for diffusion models. ICLR, 2026.
[5] X. Wu, K. Sun, F. Zhu, R. Zhao, and H. Li. Human preference score: Better aligning text-to-image models with human preference. ICCV, 2023.
[6] D. Ghosh, H. Hajishirzi, and L. Schmidt. GenEval: An object-focused framework for evaluating text-to-image alignment. NeurIPS Datasets and Benchmarks, 2023.
Questions for authors
Q1 (W1). Beyond the linearity of in the reward, what is the technical contribution over [1] and [2]? Given that the sampler coincides with [3], what practical difference does the proposed framework make?
Q2 (W2). How does the optimum of the step-level objective (Eq. 3) relate to that of the original objective (Eq. 2), and can the gap be characterized? The two aligned models use different DDIM eta values, which changes the reference policy itself. How does the proof of Theorem 1 go through in this case? Does Theorem 1 still hold when the reference policy is updated during training and when CFG is applied before fusion?
Q3 (W3). An comparation with models fine-tuned directly on via Eq. 3 would be helpful and explain why single-objective endpoint models achieve higher weighted reward than MSDDA at several in Table 4.
Q4 (W4). Experiements on modern image generation models are needed. Could you provide an user study and broader prompt set?
3 comments
W3/Q3 and W4/Q4
W3/Q3:
Given the rebuttal time constraints, we conducted this comparison in a new controlled toy experiment on SD3.5-Medium, comparing MSDDA with direct weighted-reward fine-tuning (WS) using conflicting black/white patch objectives adapted from PROUD (Yao et al., 2024, Sec. 5.1; arXiv:2407.04493). Images are 512×512 with an empty prompt. Rewards are the negative mean squared errors to white and black on the central 128×128 patch, with pixels in ; .
We train one expert per reward and a separate WS model for each w=0.2/0.5/0.8. Each model uses the same reward scaling, KL to frozen SD3.5-Medium (), LoRA rank 32, learning rate , and 160-update budget. Training and evaluation share a 20-step SDE sampler with the same noise level. MSDDA reuses the two experts across weights; each WS model is trained directly on its weighted reward. Evaluation uses the same 256 seeds for both methods:
| Metric | w=0.2 | w=0.5 | w=0.8 |
|---|---|---|---|
| MSDDA KL | 0.0018 | 0.0034 | 0.0016 |
| WS KL | 0.0011 | 0.0014 | 0.0015 |
| MSDDA − WS: J | +0.0136 ± 0.0006 | +0.0221 ± 0.0022 | +0.0190 ± 0.0008 |
KL is evaluated on each sampler's trajectories, averaged over latent dimensions and summed over 19 trained steps. , with training-matched in raw reward units. Differences use unrounded scores and paired 95% CIs over images. MSDDA achieves higher measured regularized objective values than direct weighted-reward fine-tuning at all three weights, with all paired 95% CIs excluding zero.
Imperfect single-objective optimization and reference mismatch may contribute to Table 4’s weight-dependent performance gaps. We will revise “consistently outperforms” to reflect the observed advantages at specific preference weights.
W4/Q4
Using the same SD3.5-Medium setup described in W3/Q3, we additionally compare MSDDA with RS, CoDe, and RGG. All methods use the same sampler and 256 evaluation seeds; MSDDA and RS use identical experts. Paired scalarized-reward differences (95% CIs over images) are:
| MSDDA minus | w=0.2 | w=0.5 | w=0.8 |
|---|---|---|---|
| RS | +0.0094 ± 0.0006 | +0.0079 ± 0.0017 | +0.0019 ± 0.0004 |
| CoDe | +0.088 ± 0.009 | +0.067 ± 0.005 | +0.109 ± 0.011 |
| RGG | +0.148 ± 0.013 | +0.091 ± 0.004 | +0.165 ± 0.014 |
MSDDA achieves higher scalarized rewards than RS, CoDe, and RGG at all three weights, with all paired 95% CIs excluding zero.
Thank you for the clarifications and additional experiments. My main concerns remain regarding novelty and overclaiming. The sampler can coincide with DeRaDiff, and the response does not sufficiently establish the technical advance beyond existing alignment and composition principles. Moreover, “no approximation error” applies only to the surrogate objective under restrictive assumptions, while the gap to the original objective and practical approximation errors remain uncharacterized. The additional ablations support empirical usefulness but do not establish the broader exactness and optimality claims made in the abstract and main text. Thus I will maintain my decision.
W1/Q1 and W2/Q2
W1/Q1:
We thank the reviewer for the question. Works [1] and [2] establish token-level alignment and multi-objective decoding for LLMs. We build on these principles; our contribution is to show that, under a common fixed reference and exact single-objective optimality, diffusion fusion yields exactly the weighted-reward step-level RL optimum of Eq. (3), without additional fusion approximation.
Compared with DB-MPA [4] and DeRaDiff [3], the distinction is the approximation introduced at fusion. DB-MPA's Lemma 2, Eq. (9), expresses the combined-reward optimal drift as a weighted sum plus Δ(r(w),α)−∑iwiΔ(ri,α). Its fusion rule drops this residual, which can remain nonzero even with exact experts. DeRaDiff replaces full-image geometric composition with reverse-conditional composition through a stepwise approximation (Sec. 4.1, Eqs. (4)–(5)). Our fusion follows directly from Eq. (3), without these additional approximations.
We also derive a diffusion step-level DPO loss with an explicit correction Δdiff, beyond reward linearity. With matched experts and transitions, our sampler can coincide with DeRaDiff [3]. Our contribution is therefore primarily theoretical: the exact correspondence between diffusion training and fusion, its validity conditions, and the associated DPO formulation.
W2/Q2:
Eq. (3) is a fixed-reference policy-improvement surrogate for Eq. (2). Lemma 4.1 of [1] provides a policy-improvement justification under its stated conditions, but does not bound the gap between their respective optima. Our theorem establishes exact fusion for Eq. (3); characterizing its optimality gap relative to Eq. (2) remains outside the current result.
The closed-form Gaussian fusion relies on the Gaussian-optimum assumption stated in Theorem 1; Gaussian reverse-conditional modeling is also used in DeRaDiff [3]. We will clarify this assumption.
DDIM η values. We agree that heterogeneous DDIM η changes reference means and variances, affecting the shared-reference assumption. We used different η values to examine MSDDA beyond the equal-variance setting where it coincides with DB-MPA. To assess this choice, we ran a common-η=1 ablation (original VILA, retrained ImageReward): 25 prompts × 32 seeds, 50 steps, CFG 7.5. Paired scalarized-reward differences:
Reference updates. Appendix E.2 used ambiguous notation: the pretrained KL reference stays fixed, while the rollout policy is refreshed from the trainable model every 1,000 epochs and frozen between refreshes. Refreshing keeps current/collection-policy importance ratios closer to 1 and reduces clipping saturation. Stagewise advantage estimates nevertheless differ from the fixed-reference advantages assumed in Eq. (3). Keeping all other settings unchanged, we trained with rollouts exclusively from frozen SD1.5, estimating returns and the baseline from these trajectories. MSDDA exceeds RS on the same experts at all three weights: paired scalarized-reward differences are +0.035 ± 0.031, +0.104 ± 0.051, and +0.086 ± 0.050 at w=0.2/0.5/0.8 (95% CIs over prompts).
CFG. Training and inference both use CFG 7.5, including trajectory sampling, policy log probabilities and reference predictions, so guidance introduces no additional training–inference mismatch. With a common scale and equal conditional/unconditional covariances within each expert, CFG commutes with Gaussian fusion. Applying CFG before fusion is therefore equivalent to applying it after separately fusing the conditional and unconditional branches.