acceptodds
Withdrawn from ICLR 2027

Multi-Aspect Reward Balance for Diffusion Alignment

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Image generation quality is inherently multidimensional, so aligning diffusion models with human preferences requires jointly optimizing multiple reward objectives. The central challenge is to improve a single model across reward dimensions through joint training with less manual reward balancing. Rollout samples can be informative for only a subset of rewards, making it important to preserve their distinct supervision during joint training. Common approaches train separate reward specialists, combine rewards with fixed weights, or use manually designed training curricula. However, these approaches do not adequately balance the influence of different rewards during joint training. We propose MARBLE (Multi-Aspect Reward BaLancE), which preserves reward-specific credit through independent advantage estimates and balances their influence using interactions among normalized reward gradients. Our analysis shows that a nonzero normalized refresh direction can locally reduce the participating reward losses together. Gradient diagnostics also show fewer conflicts between the refresh update and individual reward gradients. An amortized formulation of the DiffusionNFT loss, coupled with exponential moving average (EMA) coefficient smoothing, makes this adaptive coordination practical for joint training. On SD3.5 Medium with five rewards, MARBLE improves all eight evaluation metrics over both simultaneous weighted-sum training and fixed uniform coefficients. It achieves OCR 0.96 and GenEval 0.94 while retaining 0.97× weighted-sum throughput, combining multi-reward gains in one model with near-baseline training speed.

Primary area:
generative models
Keywords:
Diffusion Model, Multi-reward Training, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_K2bbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes MARBLE, a method for training a single text-to-image diffusion model on several rewards at once, built on the DiffusionNFT objective.

  • Per-reward advantages. Each reward's advantage is standardized separately. Validity masks make the OCR and GenEval rewards affect only their own prompts.
  • Weights from gradients. Every N = 10 steps, per-reward gradients are computed, and a min-norm (MGDA) problem over their normalized versions gives simplex weights. An EMA (ρ = 0.7) smooths them.
  • Cheap updates in between. Between refreshes, the smoothed weights are applied to the current advantages through one combined NFT loss. Prop. 1 justifies this: the loss is affine in the advantage.

On SD3.5-Medium with five rewards (PickScore, HPSv2.1, CLIPScore, OCR, GenEval), MARBLE reports the best average rank over eight metrics, three of them held-out evaluators. It reaches OCR 0.96 and GenEval 0.94 at 0.97× weighted-sum throughput. Ablations vary the weighting, the refresh interval and the EMA decay, and a user study compares three methods.

Critical strengths
  1. A practical recipe for a real problem, with clear specialist gains at near-baseline cost.

    • It trains one model on five rewards simultaneously, with no stage schedule. DiffusionNFT, by contrast, uses hand-set stages of 800/300/200/200/100 iterations (Sec. 1).
    • Validity masks (Eqs. 3–4) cleanly handle rewards that apply only to some prompts.
    • Amortization keeps speed at 0.97× of weighted-sum training, against 0.56× without it (Table 3).
    • OCR 0.96 is the best value in Table 1, reached alongside GenEval 0.94.
    • The GenEval and OCR gains over fixed uniform weights (+0.08 / +0.07) and over DanceGRPO-5r (+0.06 / +0.16) exceed the variation between the paper's own "comparable" settings.

    This matters because practitioners currently rely on hand-designed curricula or separate specialists.

  2. Unusually complete and candid controls.

    • The paper includes fixed uniform and 1:1:1:2:2 weights, reward dropout, refresh-interval and EMA sweeps, alternative solvers, and cached-vs-fresh agreement (Tables 4–6 and S2; App. E).
    • It reports the per-step variant even though that variant is worse.

    This matters because these tables make the claims checkable; my weaknesses below are computed from them.

Critical weaknesses
  1. The evidence supports gains on the two specialist rewards, not the claimed across-the-board superiority.

    • The claims. The abstract says MARBLE "improves all eight evaluation metrics over both simultaneous weighted-sum training and fixed uniform coefficients"; the contributions claim the "best overall performance".
    • Single runs, selected on the test sets. Each configuration was run once. N and ρ were chosen on the same test sets used in Table 1 (Sec. 4.3).
    • The paper's own standard of "comparable". The paper calls N = 5 and N = 10 "comparable" (Sec. 4.3), yet they differ by 0.39 PickScore, 0.11 Aesthetic and 0.06 UniReward. MARBLE's margins over fixed α = 0.2 on those metrics are smaller: +0.07, +0.04 and +0.03 (Table 4).
    • Only the selected setting wins. Only the selected configuration beats fixed α = 0.2 on all eight metrics. The other six reported configurations (Tables 5–6) each lose to DanceGRPO-5r on six of eight metrics, including ImageReward and UniReward.
    • The weighted-sum comparison mostly reflects per-reward standardization. Fixed α = 0.2 already obtains 79–95% of MARBLE's gain over the weighted sum on PickScore, HPSv2.1, Aesthetic, ImageReward and UniReward (Tables 1 and 4). The weighted sum's weights are not given, and it does not raise its own training rewards PickScore and HPSv2.1 above the CFG base (22.38 vs 22.34; 0.274 vs 0.279).
    • Held-out evidence is mixed. UniReward is −0.05 vs DanceGRPO-5r and −0.08 vs DiffusionNFT†. The user study (30 images per method, means only) omits DanceGRPO-5r.
    • Undisclosed baseline provenance. Rows SD-XL through DiffusionNFT† match DiffusionNFT's Table 1 (ICLR 2026) exactly. That table evaluated SD-XL and SD3.5-L at 1024 × 1024, whereas Table S5 states 512 × 512 and guidance 1.0 for all rows.

    This matters because the headline claims are what the paper will be judged on. As reported, they cannot be separated from configuration selection and run-to-run variation. It is also known that tuned scalarization often matches multi-task optimizers (Xin et al., NeurIPS 2022).

  2. The stated mechanism is not what the diagnostics test, nor what most updates apply.

    • The diagnostic holds by construction. The paper notes that the min-norm direction satisfies ĝ_k^T d* ≥ ‖d*‖² (p. 4). For the refresh direction, this implies cos(d*, g_k) ≥ ‖d*‖ > 0 for every reward, with the cosine equal across all rewards that have positive weight. The abstract places the diagnostic on "the refresh update", so Table S1's 0% conflict rate and 0.006 variance follow from the construction.
    • Most updates are different. 9 of 10 updates are cached steps (Eq. 10). By the linearity behind Prop. 1, the cached update appears to equal Σ_k ᾱ_k (n_k/B) g_k, plus a term that vanishes on-policy. That means the weights from the normalized solve are applied to unnormalized gradients. It also means OCR and GenEval, each valid on about a third of the samples, are scaled down accordingly, because Eq. 10 averages over all samples "without renormalizing". The common-descent property is thus not established for these steps. Read from Fig. S12, the cached direction's cosine with a fresh solve is already about 0.83 one step after a refresh.
    • The variant closest to the analysis does worse. Solving α at every step scores below amortized MARBLE on all eight metrics, and its PickScore of 21.80 is below even the CFG base's 22.34 (Table 4).
    • The priority claim is thin. "First to reveal conflicts" (Sec. 5) rests on five mini-batches, 4 of which show a conflict.

    This matters because the paper's insight is advertised as identifying and resolving gradient conflicts. If the evidence covers only the refresh direction, where it holds by construction, why MARBLE works remains open. A simpler account appears to fit the results at least as well: adaptive re-weighting of standardized advantages that leans toward the specialist rewards. The GenEval coefficient often rises to about 0.3 (App. C.3).

Questions for authors
  1. (Suggestion.) Please report at least 3 seeds (mean ± s.d. on all eight metrics) for MARBLE, fixed α = 0.2, 1:1:1:2:2 and DanceGRPO-5r. Please choose N and ρ on a held-out prompt split rather than the test sets.

  2. (Clarification.)

    • Were Table 1's rows SD-XL through DiffusionNFT† taken from DiffusionNFT's Table 1? If so, please say so, correct Table S5, and confirm that MARBLE was evaluated with the identical pipeline (resolution, seeds, UnifiedReward version). This matters because evaluation pipelines differ: for SD3.5-M with CFG, UnifiedReward is 3.33 in Flow-GRPO's paper and 3.03 in DiffusionNFT's.
    • What were the weighted-sum weights and normalization?
    • Was DanceGRPO-5r's summed advantage rescaled before the NFT loss, or were A_max or the learning rate re-tuned for it?
  3. (Clarification / suggestion.)

    • Is the direction d in Table S1 the refresh direction?
    • Please report conflict statistics for the cached updates actually applied, and for fixed α = 0.2, over many batches.
    • Please report per-reward ‖g_k‖ and valid fractions, so the effective cached weights can be seen.
    • What does "Solve α every step" apply: Eq. 7 or Eq. 10, and with or without the EMA?

    A cached variant rescaled by 1/‖g_k‖ and B/n_k would test the rationale directly.

  4. (Suggestion.) Could the user study include DanceGRPO-5r, state how prompts were selected and whether they were shared across methods, and report bootstrap CIs?

What would move me to 3 (weak accept): all three of the following.

  • Multi-seed results, with settings chosen on held-out prompts, keep MARBLE ahead of fixed α = 0.2 and DanceGRPO-5r on GenEval and OCR without losses on the held-out metrics.
  • The baseline provenance and settings are clarified.
  • The claims are recalibrated: specialist gains, and the mechanism framed as adaptive re-weighting unless cached-step diagnostics show reduced conflicts.

Minor (no effect on my rating):

  • Related work. Please soften "first to reveal conflicts". DRaFT (ICLR 2024) already reported that per-reward gradients have "substantially different norms" and weighted the rewards accordingly. Consider citing:
    • FAMO (NeurIPS 2023), on amortized balancing;
    • MoCo (ICLR 2023), on the bias of stochastic MGDA, which bears on the per-step explanation;
    • CaPO (CVPR 2025) and Diffusion-DPO;
    • the tuned-scalarization studies (Xin et al. and Kurin et al., NeurIPS 2022).
  • Throughput. In Table 3, a refresh every 10 steps at the non-amortized 0.56× suggests about 0.93×, not 0.97×. How was throughput measured?
  • Fig. S4. App. C.3 says MARBLE "converges to a higher final value" on HPSv2.1. Fig. S4 does not show this over its plotted range.
  • Missing details.
    • Numbers or images for the "Collapse" row.
    • Total training steps, NFT β, β_KL, mini-batch size and baseline hyperparameters in Table S3.
    • Whether PCGrad, CAGrad and Nash-MTL (App. E.3) were amortized like MARBLE.
  • Citations. DDPO, Flow-GRPO, Parrot, Nash-MTL and DiffusionNFT are cited as arXiv preprints, but appeared at ICLR 2024, NeurIPS 2025, ECCV 2024, ICML 2022 and ICLR 2026 respectively.
Reviewer_yne8OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This work enhances DiffusionNFT to handle multiple reward objectives and argues that the challenge lies in balancing the influence of different rewards during training. The authors propose MARBLE, which balances these influences by estimating reward-coefficient weights in the gradient space. Gradient coordination analysis shows that MARBLE achieves more balanced updates across rewards than reward weighting and is competitive on three representative benchmarks.

Critical strengths
  1. The proposed amortization is useful in practice because it reuses coefficients, and EMA smoothing can reduce coordination costs and stabilize the reward-coefficient weights.
Critical weaknesses
  1. My main concern is the story and presentation of this paper. Specifically, there are two critical questions:
    1. The notations and presentations in Sec. 3 are inconsistent. For example, Eq. (1) only uses R(x)R(x) as the objective, and you do not mention what are Rk(xi)R_k(x_i) and their relationship with R(x)R(x). After reading the section, the objective function is still unclear. The similar case also applies to vθ,mkiv_\theta, m_{ki}, etc. Please define all symbols consistently and explicitly state the multi-objective optimization problem.
    2. MARBLE is only extending the DiffusionNFT to multiple reward objectives, so the scope looks narrow to me. While the authors claim this is the first work to reveal conflicts between reward objectives in the gradient space of diffusion alignment, it is unclear to me why this needs to be addressed in the parameters' gradient space. In addition, similar ideas for resolving conflicting rewards via the score space (gradient of the log-likelihood) have been discussed in Preference Optimization [1] or Test-Time Alignment [2]. The authors could revise their presentation to emphasize the challenges and necessities of balancing in the gradient space.
  2. Figure 1. The authors can clarify that the Model here is the generative model, not the (calibrated) reward model learned from multiple reward models.
Questions for authors
  1. Figure 2. It is unclear what the statement means or why it says that the patterns show why each reward needs its own statistics. Could the authors explain their analysis in more detail?
  2. Sec 3.3., LINE 224--226. The statement Amortizing reward coordination is unclear. This paragraph seems to list two sentences for readers without explanation.
Reviewer_Z2naOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper provide a new method for post training in the image geneation, which is called MARBLE. MARBLE solved an common issue that anyone who fine-tunes one diffusion model on several rewards will meet: the rewards live on different scales, some rewards only apply to some prompts (OCR and GenEval are masked to their own prompt sources), and the per-reward gradients pull against each other. To solve this, MARBLE combines DiffusionNFT with three existing components: masked per-reward advantage normalization, periodic MGDA-based gradient balancing with EMA smoothing, and an exact advantage-aggregation identity that enables applying cached weights in a single backward pass at 0.97× the throughput of weighted-sum training.

Critical strengths
  1. Form this as a MultiTask problem and benefit the gradient methods there is good and useful.

  2. Proposition 1 enables caching and aggregating balancing weights into a single backward pass at 0.97× throughput and 1.14× memory relative to naive weighted-sum training, running 2.1–2.7× faster than PCGrad, CAGrad, or NashMTL.

  3. Per-reward advantage normalization cleanly isolates reward-specific objectives (e.g., OCR, GenEval) without diluting unconditioned prompt rollouts into a shared advantage.

Critical weaknesses

The paper is overstated the novelty: claims to be the "first to reveal conflicts between reward objectives in the gradient space of diffusion alignment." However, gradient-space multi-objective alignment was established in LLMs by GAPO, and gradient scale imbalances in multi-reward diffusion were documented by DRaFT. Neither is cited. The architecture otherwise combines known methods (DiffusionNFT, GRPO normalization, MGDA), leaving Proposition 1 as an efficiency shortcut rather than a new balancing principle.

Questions for authors
  1. I am trying to understand of the reliability of the result. Can you provide mean ± std over at least 3 seeds for MARBLE, weighted-sum, and DanceGRPO-5r on the held-out reward benchmarks?

  2. I think there are some missing baselines: How does MARBLE compare against a weight-averaged specialist model (Rewarded Soups), native Parrot (with prompt expansion), or CaPO (CVPR 2025)? How does the method differs from concurrent multi-reward normalization works (e.g., Multi-GRPO, GDPO, MCDPO, GDRO)?

  3. Across a broad training run (e.g., several hundred batches), what percentage of cached-weight updates (Eq. 10) maintain positive cosine similarity with all active reward gradients?

  4. can this method apply to some quite subjective reward and even to some dimensions with some kind of conflit, e.g., photorealism, beauty on a lady face?

  • Multi-GRPO — "Multi-GRPO: Multi-Group Advantage Estimation for Text-to-Image Generation with Tree-Based Trajectories and Multiple Rewards" — Lyu et al., arXiv, Nov 2025.

  • GDPO — "GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization" — Liu et al, arXiv, ICML 2026.

  • MCDPO — "Multi-dimensional Preference Alignment by Conditioning Reward Itself" — Jang et al., arXiv, Dec 2025; ECCV 2026

  • GDRO: "Group-level Reward Post-training Suitable for Diffusion Models" — Wang et al., arXiv, Jan 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.