acceptodds
Under review as a conference paper at ICLR 2027

Step-level Denoising-time Diffusion Alignment with Multiple Objectives

Abstract

Reinforcement learning (RL) has emerged as a powerful tool for aligning diffusion models with human preferences, typically by optimizing a single reward function under a KL regularization constraint. In practice, however, human preferences are inherently pluralistic, and aligned models must balance multiple downstream objectives, such as aesthetic quality and text-image consistency. Existing multi-objective approaches either rely on costly multi-objective RL fine-tuning or on fusing separately aligned models at denoising time, but they generally require access to reward values (or their gradients) and/or introduce approximation error in the resulting denoising objectives. In this paper, we revisit the problem of RL fine-tuning for diffusion models and address the intractability of identifying the optimal policy by introducing a RL formulation. Building on this, we further propose (MSDDA), a retraining-free framework for aligning diffusion models with multiple objectives, obtaining the optimal reverse denoising distribution in closed form, with mean and variance expressed directly in terms of single-objective base models. We prove that this denoising-time objective is exactly equivalent to the step-level RL fine-tuning, introducing no approximation error. Moreover, we provide numerical results, which indicate our method outperforms existing denoising-time approaches.

Primary area:
generative models
Keywords:
Diffusion Models, Multi-Objective Alignment, Denoising-Time Alignment

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_BpiDOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies retraining-free multi-objective alignment of diffusion models. It replaces the standard KL-regularized RL objective with a step-level surrogate whose optimal policy is available in closed form, and shows that under this surrogate the optimal policy for a weighted reward is the normalized product of the single-objective optimal reverse kernels. With Gaussian kernels this yields a precision-weighted fusion of means and variances, requiring no reward access and no training per weight vector.

Critical strengths
  1. Well-motivated problem and clear positioning. Multi-objective alignment under user-specific preference weights is practically important, and the paper clearly identifies the gap left by existing methods: per-weight retraining is expensive, guidance methods need reward access at inference, and existing reward-free fusion methods rely on unquantified approximations (Table 1).

  2. Simple, closed-form and cheap fusion rule. Theorem 1 gives explicit precision-weighted expressions for the fused mean and variance via a short, transparent derivation. The resulting sampler needs no reward model, gradients or extra sampling, and costs roughly M times a single model, far below CoDe and RGG.

  3. Unifying view of existing fusion samplers. The framework recovers DB-MPA as the equal-variance special case and provides an explicit justification for the DeRaDiff sampler. The derived step-level DPO loss with its asymmetric Δ_diff regularizer is a potentially interesting side contribution.

Critical weaknesses

W1 (Novelty). The step-level objective and its closed-form policy are taken from [1]; the geometric-mixture result for weighted rewards is already established for LLMs in [2]; the sampler is identical to DeRaDiff [3] and, under equal variances, to DB-MPA [4]. The only new element is the observation that QπpreQ^{\pi_{\text{pre}}} is linear in the reward,which makes the mixture exact under the surrogate objective. The contribution is thus a re-derivation of an existing algorithm rather than a new method.

W2 (Validity of the theoretical claims). The "no approximation error" claim results from changing the objective (Eq. 3 instead of Eq. 1/2), not from resolving the original intractability; the gap between the surrogate optimum and the true KL-regularized optimum is never analyzed. Moreover, the assumptions of Theorem 1 do not hold in the paper's own experiments. The exact optimum of Eq. 3, πpreexp⁡(Qi/λ)\pi_{\text{pre}}\exp(Q^i/\lambda), is not Gaussian in general; the trained models are Gaussian approximations, so the fusion carries an unquantified approximation error of the same kind the paper criticizes in [3, 4]. Variance heterogeneity is created by using different DDIM η\eta (1.0 vs. 0.8), which changes πpre\pi_{\text{pre}} itself and breaks the cancellation (via ∑iwi=1\sum_i w_i = 1) in the proof of Theorem 1. In addition, πref\pi_{\text{ref}} is updated during training and CFG is applied per model before fusion; neither is covered by the theory.

W3 (Central claim untested). No comparison against models directly fine-tuned on rwr^w; no KL to the pretrained model or regularized objective values are reported, so neither exactness nor optimality is verified. From Table 4, the nearest single-objective model beats MSDDA on the weighted reward at w1∈{0.1,0.2,0.7,0.8,0.9}w_1 \in \{0.1, 0.2, 0.7, 0.8, 0.9\} (e.g., w1=0.8w_1 = 0.8: 0.8⋅0.69+0.2⋅(−0.12)≈0.530.8 \cdot 0.69 + 0.2 \cdot (-0.12) \approx 0.53 vs. 0.480.48), and DB-Extended outperforms MSDDA for w1≥0.7w_1 \ge 0.7, contradicting "consistently outperforms".

W4 (Scale and fairness). Only SDv1.5 (an out of data model architecture) and about 25 test prompts in the main setting. CoDe/RGG numbers (including inference time) are copied from [4] with different base models and hardware. DeRaDiff [3] and the original DB-MPA [4] (equal-variance setting) are not compared against, although the proposed sampler reduces to them.

References

[1] Y. Zeng, G. Liu, W. Ma, N. Yang, H. Zhang, and J. Wang. Token-level direct preference optimization. arXiv:2404.11999, 2024.

[2] R. Shi, Y. Chen, Y. Hu, A. Liu, H. Hajishirzi, N. A. Smith, and S. S. Du. Decoding-time language model alignment with multiple objectives. NeurIPS, 2024.

[3] R. D. S. Manujith, T. T. Tzun, K. Kawaguchi, and Y. Zhang. DeRaDiff: Denoising time realignment of diffusion models. arXiv:2601.20198, 2026.

[4] M. Cheng, F. Doudi, D. Kalathil, M. Ghavamzadeh, and P. R. Kumar. Diffusion blend: Inference-time multi-preference alignment for diffusion models. ICLR, 2026.

[5] X. Wu, K. Sun, F. Zhu, R. Zhao, and H. Li. Human preference score: Better aligning text-to-image models with human preference. ICCV, 2023.

[6] D. Ghosh, H. Hajishirzi, and L. Schmidt. GenEval: An object-focused framework for evaluating text-to-image alignment. NeurIPS Datasets and Benchmarks, 2023.

Questions for authors

Q1 (W1). Beyond the linearity of QπpreQ^{\pi_{\text{pre}}} in the reward, what is the technical contribution over [1] and [2]? Given that the sampler coincides with [3], what practical difference does the proposed framework make?

Q2 (W2). How does the optimum of the step-level objective (Eq. 3) relate to that of the original objective (Eq. 2), and can the gap be characterized? The two aligned models use different DDIM eta values, which changes the reference policy itself. How does the proof of Theorem 1 go through in this case? Does Theorem 1 still hold when the reference policy is updated during training and when CFG is applied before fusion?

Q3 (W3). An comparation with models fine-tuned directly on rwr^w via Eq. 3 would be helpful and explain why single-objective endpoint models achieve higher weighted reward than MSDDA at several w1w_1 in Table 4.

Q4 (W4). Experiements on modern image generation models are needed. Could you provide an user study and broader prompt set?

3 comments
AuthorsOpenReview ↗

W1/Q1 and W2/Q2

W1/Q1:

We thank the reviewer for the question. Works [1] and [2] establish token-level alignment and multi-objective decoding for LLMs. We build on these principles; our contribution is to show that, under a common fixed reference and exact single-objective optimality, diffusion fusion yields exactly the weighted-reward step-level RL optimum of Eq. (3), without additional fusion approximation.

Compared with DB-MPA [4] and DeRaDiff [3], the distinction is the approximation introduced at fusion. DB-MPA's Lemma 2, Eq. (9), expresses the combined-reward optimal drift as a weighted sum plus Δ(r(w),α)−∑iwiΔ(ri,α)\Delta^{(r(w),\alpha)}-\sum_iw_i\Delta^{(r_i,\alpha)}. Its fusion rule drops this residual, which can remain nonzero even with exact experts. DeRaDiff replaces full-image geometric composition with reverse-conditional composition through a stepwise approximation (Sec. 4.1, Eqs. (4)–(5)). Our fusion follows directly from Eq. (3), without these additional approximations.

We also derive a diffusion step-level DPO loss with an explicit correction Δdiff\Delta_{\mathrm{diff}}, beyond reward linearity. With matched experts and transitions, our sampler can coincide with DeRaDiff [3]. Our contribution is therefore primarily theoretical: the exact correspondence between diffusion training and fusion, its validity conditions, and the associated DPO formulation.

W2/Q2:

Eq. (3) is a fixed-reference policy-improvement surrogate for Eq. (2). Lemma 4.1 of [1] provides a policy-improvement justification under its stated conditions, but does not bound the gap between their respective optima. Our theorem establishes exact fusion for Eq. (3); characterizing its optimality gap relative to Eq. (2) remains outside the current result.

The closed-form Gaussian fusion relies on the Gaussian-optimum assumption stated in Theorem 1; Gaussian reverse-conditional modeling is also used in DeRaDiff [3]. We will clarify this assumption.

DDIM η\eta values. We agree that heterogeneous DDIM η\eta changes reference means and variances, affecting the shared-reference assumption. We used different η\eta values to examine MSDDA beyond the equal-variance setting where it coincides with DB-MPA. To assess this choice, we ran a common-η=1\eta=1 ablation (original VILA, retrained ImageReward): 25 prompts × 32 seeds, 50 steps, CFG 7.5. Paired scalarized-reward differences:

MSDDA minusw=0.2w=0.5w=0.8
RS+0.11 ± 0.03+0.21 ± 0.04+0.12 ± 0.04
CoDe+0.32 ± 0.06+0.13 ± 0.08+0.08 ± 0.13
RGG+0.07 ± 0.06+0.17 ± 0.08+0.37 ± 0.14

Reference updates. Appendix E.2 used ambiguous notation: the pretrained KL reference stays fixed, while the rollout policy is refreshed from the trainable model every 1,000 epochs and frozen between refreshes. Refreshing keeps current/collection-policy importance ratios closer to 1 and reduces clipping saturation. Stagewise advantage estimates nevertheless differ from the fixed-reference advantages assumed in Eq. (3). Keeping all other settings unchanged, we trained with rollouts exclusively from frozen SD1.5, estimating returns and the baseline from these trajectories. MSDDA exceeds RS on the same experts at all three weights: paired scalarized-reward differences are +0.035 ± 0.031, +0.104 ± 0.051, and +0.086 ± 0.050 at w=0.2/0.5/0.8 (95% CIs over prompts).

CFG. Training and inference both use CFG 7.5, including trajectory sampling, policy log probabilities and reference predictions, so guidance introduces no additional training–inference mismatch. With a common scale and equal conditional/unconditional covariances within each expert, CFG commutes with Gaussian fusion. Applying CFG before fusion is therefore equivalent to applying it after separately fusing the conditional and unconditional branches.

AuthorsOpenReview ↗

W3/Q3 and W4/Q4

W3/Q3:

Given the rebuttal time constraints, we conducted this comparison in a new controlled toy experiment on SD3.5-Medium, comparing MSDDA with direct weighted-reward fine-tuning (WS) using conflicting black/white patch objectives adapted from PROUD (Yao et al., 2024, Sec. 5.1; arXiv:2407.04493). Images are 512×512 with an empty prompt. Rewards are the negative mean squared errors to white and black on the central 128×128 patch, with pixels in [0,1][0,1]; rw=wrwhite+(1−w)rblackr^w=w r_{\mathrm{white}}+(1-w)r_{\mathrm{black}}.

We train one expert per reward and a separate WS model for each w=0.2/0.5/0.8. Each model uses the same reward scaling, KL to frozen SD3.5-Medium (β=0.01\beta=0.01), LoRA rank 32, learning rate 3×10−43\times10^{-4}, and 160-update budget. Training and evaluation share a 20-step SDE sampler with the same noise level. MSDDA reuses the two experts across weights; each WS model is trained directly on its weighted reward. Evaluation uses the same 256 seeds for both methods:

Metricw=0.2w=0.5w=0.8
MSDDA KL0.00180.00340.0016
WS KL0.00110.00140.0015
MSDDA − WS: J+0.0136 ± 0.0006+0.0221 ± 0.0022+0.0190 ± 0.0008

KL is evaluated on each sampler's trajectories, averaged over latent dimensions and summed over 19 trained steps. J=E[rw]−λKLJ=\mathbb{E}[r^w]-\lambda\mathrm{KL}, with training-matched λ=1.43×10−3\lambda=1.43\times10^{-3} in raw reward units. Differences use unrounded scores and paired 95% CIs over images. MSDDA achieves higher measured regularized objective values than direct weighted-reward fine-tuning at all three weights, with all paired 95% CIs excluding zero.

Imperfect single-objective optimization and reference mismatch may contribute to Table 4’s weight-dependent performance gaps. We will revise “consistently outperforms” to reflect the observed advantages at specific preference weights.

W4/Q4

Using the same SD3.5-Medium setup described in W3/Q3, we additionally compare MSDDA with RS, CoDe, and RGG. All methods use the same sampler and 256 evaluation seeds; MSDDA and RS use identical experts. Paired scalarized-reward differences (95% CIs over images) are:

MSDDA minusw=0.2w=0.5w=0.8
RS+0.0094 ± 0.0006+0.0079 ± 0.0017+0.0019 ± 0.0004
CoDe+0.088 ± 0.009+0.067 ± 0.005+0.109 ± 0.011
RGG+0.148 ± 0.013+0.091 ± 0.004+0.165 ± 0.014

MSDDA achieves higher scalarized rewards than RS, CoDe, and RGG at all three weights, with all paired 95% CIs excluding zero.

Thank you for the clarifications and additional experiments. My main concerns remain regarding novelty and overclaiming. The sampler can coincide with DeRaDiff, and the response does not sufficiently establish the technical advance beyond existing alignment and composition principles. Moreover, “no approximation error” applies only to the surrogate objective under restrictive assumptions, while the gap to the original objective and practical approximation errors remain uncharacterized. The additional ablations support empirical usefulness but do not establish the broader exactness and optimality claims made in the abstract and main text. Thus I will maintain my decision.

Reviewer_gXBfOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes MSDDA, an inference-time fusion method for multi-objective alignment of diffusion models. The method combines the reverse conditional distributions of multiple single-objective models at each denoising step, enabling flexible control over different objectives and their corresponding weights without retraining the generative model for each weight configuration. The authors further formulate a step-level RL objective based on the advantage function of a fixed reference policy and derive a closed-form expression for the optimal multi-objective policy, providing a theoretical justification for denoising-time model fusion. In addition, the paper proposes a corresponding step-level DPO objective. Experiments on Stable Diffusion v1.5 demonstrate that MSDDA achieves competitive aggregate reward scores under multiple objective-weight configurations and incurs lower inference overhead than some inference-time optimization methods that require additional reward computations.

Overall, this work attempts to establish a more rigorous theoretical foundation for existing denoising-time model fusion methods and demonstrates their potential for multi-objective generative alignment. However, several aspects of the current work require further clarification, particularly regarding methodological novelty, the consistency between the theoretical assumptions and the actual training procedure, and the comprehensiveness of the experimental comparisons.

Critical strengths
  1. The paper provides a theoretical analysis of existing denoising-time model fusion methods. Starting from a step-level RL objective, the authors derive a closed-form expression for the optimal multi-objective policy and establish its connection to the weighted geometric fusion of reverse denoising distributions. This analysis provides a relatively clear theoretical perspective on how different single-objective policies can be combined under specific assumptions and offers a useful framework for understanding multi-objective alignment in diffusion models.

  2. The experimental results demonstrate practical advantages over direct parameter fusion. Using the same single-objective models, MSDDA achieves higher aggregate reward scores than Reward Soup (RS) under multiple objective-weight configurations in both the main experiments and the GenEval experiments. These results support the practical value of performing probability distribution fusion at the denoising stage and suggest that the proposed method can balance different generation objectives to some extent.

Critical weaknesses
  1. The methodological novelty requires further clarification. The core operation of the proposed method—fusing the reverse conditional distributions of multiple single-objective models at each diffusion denoising step—is highly similar to the approach adopted in the existing work DeRaDiff. The primary distinction appears to lie in reformulating the step-level RL objective to derive a closed-form optimal solution and provide a theoretical justification for this fusion strategy. However, it remains unclear what specific theoretical advances this derivation offers over prior work and whether it leads to substantive algorithmic improvements or broader applicability. The authors should clarify the theoretical and methodological differences between MSDDA and DeRaDiff and provide experimental evidence demonstrating whether the proposed theoretical framework yields practical benefits beyond those of existing fusion methods.

  2. The assumption of a fixed reference policy in the theoretical derivation may be inconsistent with the actual training procedure. The theoretical analysis relies on a shared and fixed reference policy. However, Appendix E.2 states that the model is updated every 1,000 training epochs in practice. The authors should clarify which model or component is updated and whether this procedure involves updating the reference policy itself. If the reference policy changes throughout training, do the single-objective models obtained at different stages still satisfy the shared-reference-policy assumption required by the theoretical derivation? After training, can the models optimized for different objectives still be regarded as optimal policies within the same theoretical framework? The authors should clarify the relationship between the actual training procedure and the theoretical assumptions, and ideally conduct ablation studies to investigate the impact of reference-policy updates on the final fusion performance.

  3. The fairness of comparisons with existing methods is not sufficiently established. The results for CoDe and RGG in Table 2 are taken from prior publications, but the paper does not sufficiently clarify whether these results were obtained using the same test prompts, sampling parameters, generation settings, and evaluation protocols. Since these factors can substantially affect both generation quality and inference time, directly comparing results reported in different papers may not ensure a fair evaluation. The authors should, where possible, re-evaluate these baselines under a unified experimental setup or provide sufficiently detailed information about the evaluation protocols to substantiate the comparisons.

  4. The advantage of MSDDA over other distribution-fusion baselines is not consistently demonstrated across all weight configurations. The results in Table 4 of the appendix show that MSDDA does not outperform DB-Extended under every objective-weight configuration. In particular, DB-Extended achieves higher aggregate reward scores under some configurations that place greater emphasis on ImageReward. This suggests that the theoretical optimality established by the paper does not consistently translate into superior empirical performance under the evaluated metrics. The authors should further investigate the reasons for these performance differences and discuss the potential effects of imperfect single-objective optimization, the discrepancy between practical policies and their theoretical optima, and variations in performance across different weight configurations.

Questions for authors
  1. Novelty over DeRaDiff: Given that DeRaDiff has already explored denoising-time fusion of multiple diffusion models, what are the specific theoretical contributions of MSDDA beyond providing a mathematical justification for this existing fusion strategy? Does the proposed framework offer any substantive algorithmic improvements or broader applicability?
  2. Consistency between theory and practice: The theoretical derivation assumes a shared, fixed reference policy, whereas Appendix E.2 mentions model updates every 1,000 training epochs. What exactly is updated during training? If the reference policy is also updated, do the resulting single-objective models still satisfy the assumptions required by the theoretical derivation? How does reference-policy updating affect the validity of the theoretical claims and the final fusion performance?
  3. Fairness of baseline comparisons: For CoDe and RGG in Table 2, were the results obtained using the same prompts, sampling configurations, and evaluation protocols as MSDDA? If not, how can the authors ensure that the reported performance and inference-time comparisons are fair and reliable?
  4. Inconsistent empirical superiority: Table 4 shows that MSDDA does not consistently outperform DB-Extended across different objective-weight configurations. How do the authors explain this discrepancy between the theoretical optimality claims and the empirical results? What factors contribute to the performance degradation under certain weight configurations, and under what conditions does MSDDA provide a consistent advantage over existing fusion methods?
1 comment

W1/Q1:

We agree that, when all implementation details are matched, the sampling transitions can coincide with DeRaDiff, so the empirical results alone do not isolate the benefit of our theoretical formulation. The distinction lies in the derivation. DeRaDiff starts from Eq. (1), whose optimal policy is difficult to track because of the circular dependence between the policy and its value estimates, and therefore applies the posterior product stepwise as an approximation. In contrast, we identify a step-level objective in Eq. (3) under which posterior fusion exactly optimizes the corresponding multi-objective problem, under the assumption that the component policies are optimal under a common, fixed reference. Thus, our contribution is not a different sampler, but a theoretical justification that explains when and why this sampler is valid without introducing an additional fusion approximation.

W2/Q2:

Appendix E.2 used ambiguous notation for two policies. The pretrained KL reference πref\pi_{\mathrm{ref}} remains fixed. The rollout policy πb\pi_b is refreshed from πθ\pi_\theta every 1,000 epochs and frozen between refreshes. We will distinguish them and revise the DPOK comparison.

The importance ratio is ρ=πθ(a∣s)/πb,collect(a∣s)\rho=\pi_\theta(a\mid s)/\pi_{b,\mathrm{collect}}(a\mid s), using each transition's collection-time probability. We directly clip it to [1−10−4,1+10−4][1-10^{-4},1+10^{-4}]; outside this interval, the reward term contributes no gradient. Refreshing aims to keep ratios closer to 1 and reduce clipping saturation: the in-range fraction rises from 0.64–0.69 to 0.87–0.92 after refreshes in the ImageReward run. However, its advantage estimates differ from the fixed-reference advantages assumed in Eq. (3).

Rollout-policy ablation. We also trained with rollouts from frozen SD1.5, keeping other settings unchanged. Entries below are ImageReward / VILA; the stagewise scores are from the paper's Table 2, and CoDe/RGG are reruns.

Methodw=0.2w=0.5w=0.8
MSDDA (fixed rollout)0.29 / 0.040.50 / −0.020.63 / −0.12
RS (fixed rollout)0.17 / 0.020.34 / −0.070.53 / −0.17
MSDDA (stagewise)0.22 / 0.480.42 / 0.380.57 / 0.14
RS (stagewise)0.23 / 0.330.34 / 0.050.55 / −0.10

With fixed-rollout experts, paired MSDDA−RS differences in rw=wr1+(1−w)r2r^w=wr_1+(1-w)r_2 are +0.035 ± 0.031, +0.104 ± 0.051, and +0.086 ± 0.050 at the three weights (95% CIs clustered by prompt). All exclude zero.

W3/Q3:

The quoted CoDe/RGG scores use the same SD1.5 backbone and 25 training/25 test prompts as Diffusion Blend (arXiv:2505.18547v1). We reran both with its released code: 50 DDIM steps, CFG 7.5, 32 seeds per test prompt, ImageReward and VILA (rescaled as 4r−24r-2). CoDe uses 20 particles and block size 5; RGG uses γ=0.024\gamma=0.024, α=0.1\alpha=0.1. Every rerun reward is within 0.04 of the quoted value. Entries are ImageReward / VILA.

Method / sourcew=0.2w=0.5w=0.8
CoDe (cited)0.40 / 0.050.60 / 0.010.66 / −0.07
CoDe (rerun)0.40 / 0.070.57 / 0.010.62 / −0.05
RGG (cited)0.21 / 0.420.23 / 0.230.27 / 0.00
RGG (rerun)0.23 / 0.430.24 / 0.250.24 / 0.03

Using the same 800 prompt–seed pairs, paired scalarized-reward differences are below (95% CIs clustered by prompt):

Comparisonw=0.2w=0.5w=0.8
MSDDA − CoDe+0.30 ± 0.06+0.11 ± 0.07−0.00 ± 0.10
MSDDA − RGG+0.04 ± 0.05+0.15 ± 0.04+0.29 ± 0.10

The intervals exclude zero against CoDe at w=0.2/0.5 and RGG at w=0.5/0.8. We also re-timed all methods on one RTX 6000 Ada: SD 3.59, RS 4.97, MSDDA 10.02, CoDe 93.8, RGG 24.2 seconds/image. MSDDA is 9.4× faster than CoDe and 2.4× faster than RGG. We will replace the mixed-hardware timing comparison with these measurements.

W4/Q4:

We will replace “consistently outperforms” with a statement reflecting Table 4's weight-dependent results. Theorem 1 assumes a shared reference and exact single-objective optimizers. Heterogeneous DDIM η\eta, stagewise advantage estimation, and imperfect expert optimization may affect agreement with these conditions. Their individual effects are not isolated in Table 4, so the theorem does not imply superiority over DB-Extended at every weight.

Reviewer_Nqj5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MSDDA, a retraining-free framework that aligns diffusion models with multiple human preferences. It seamlessly merges single-objective base models during the denoising process, achieving exact optimal multi-objective alignment without requiring costly RL fine-tuning or introducing approximation errors.

Critical strengths

Retraining-Free Efficiency: It completely eliminates the need for computationally expensive multi-objective RL fine-tuning by merging existing, single-objective models directly at inference time.

Zero Approximation Error: Unlike previous fusion methods, it mathematically proves that its denoising-time objective is exactly equivalent to step-level RL fine-tuning, introducing no approximation loss.

Closed-Form Solution: It computes the optimal reverse denoising distribution (both mean and variance) in closed form directly from the base models.

Critical weaknesses
  • Experiment is not solid. No experiments on SD3 or other modern neural networks.
Questions for authors

na

1 comment

We thank the reviewer for highlighting the need for experiments on modern architectures. We have added a controlled experiment on SD3.5-Medium at 512×512 resolution, using conflicting black/white patch objectives adapted from PROUD (Yao et al., 2024, Sec. 5.1; arXiv:2407.04493). Let Ω\Omega contain all RGB entries in the central 128×128 patch, with pixel values xp∈[0,1]x_p\in[0,1]. We define rwhite(x)=−1∣Ω∣∑p∈Ω(xp−1)2r_{\mathrm{white}}(x)=-\frac{1}{|\Omega|}\sum_{p\in\Omega}(x_p-1)^2 and rblack(x)=−1∣Ω∣∑p∈Ωxp2r_{\mathrm{black}}(x)=-\frac{1}{|\Omega|}\sum_{p\in\Omega}x_p^2.

The two rewards favor white and black patches, respectively; higher values are better. We evaluate rw=wrwhite+(1−w)rblackr^w=w r_{\mathrm{white}}+(1-w)r_{\mathrm{black}}, where ww is the weight on the white-patch reward.

Both experts use the same training configuration and KL regularization toward frozen SD3.5-Medium. All methods use the same 20-step SDE sampler with the same noise level. MSDDA and RS use identical experts. We also train a separate model directly on each weighted reward (WS), matching each expert’s 160-update budget, reward scaling, and KL weight.

Paired scalarized-reward differences over 256 shared evaluation seeds (95% CIs over images):

MSDDA minusw=0.2w=0.5w=0.8
RS+0.0094 ± 0.0006+0.0079 ± 0.0017+0.0019 ± 0.0004
CoDe+0.088 ± 0.009+0.067 ± 0.005+0.109 ± 0.011
RGG+0.148 ± 0.013+0.091 ± 0.004+0.165 ± 0.014
WS+0.014 ± 0.001+0.022 ± 0.002+0.019 ± 0.001

MSDDA achieves higher scalarized rewards than all compared methods at every evaluated weight, with all paired confidence intervals excluding zero. These results extend our evaluation to a modern backbone under controlled conflicting objectives. We will include the experimental protocol and results in the revised manuscript.

Comments on the paper

1 comment
Area_Chair_qVjXOpenReview ↗

[Urgent] Please check the authors' ICLR rebuttal soon

Dear Reviewers,

The authors have posted their responses to the issues you raised. As the discussion period will end soon, please take a look and have a discussion with the authors at your earliest convenience. Thank you very much!

Best, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.