- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes MARBLE, a method for training a single text-to-image diffusion model on several rewards at once, built on the DiffusionNFT objective.
- Per-reward advantages. Each reward's advantage is standardized separately. Validity masks make the OCR and GenEval rewards affect only their own prompts.
- Weights from gradients. Every N = 10 steps, per-reward gradients are computed, and a min-norm (MGDA) problem over their normalized versions gives simplex weights. An EMA (ρ = 0.7) smooths them.
- Cheap updates in between. Between refreshes, the smoothed weights are applied to the current advantages through one combined NFT loss. Prop. 1 justifies this: the loss is affine in the advantage.
On SD3.5-Medium with five rewards (PickScore, HPSv2.1, CLIPScore, OCR, GenEval), MARBLE reports the best average rank over eight metrics, three of them held-out evaluators. It reaches OCR 0.96 and GenEval 0.94 at 0.97× weighted-sum throughput. Ablations vary the weighting, the refresh interval and the EMA decay, and a user study compares three methods.
Critical strengths
-
A practical recipe for a real problem, with clear specialist gains at near-baseline cost.
- It trains one model on five rewards simultaneously, with no stage schedule. DiffusionNFT, by contrast, uses hand-set stages of 800/300/200/200/100 iterations (Sec. 1).
- Validity masks (Eqs. 3–4) cleanly handle rewards that apply only to some prompts.
- Amortization keeps speed at 0.97× of weighted-sum training, against 0.56× without it (Table 3).
- OCR 0.96 is the best value in Table 1, reached alongside GenEval 0.94.
- The GenEval and OCR gains over fixed uniform weights (+0.08 / +0.07) and over DanceGRPO-5r (+0.06 / +0.16) exceed the variation between the paper's own "comparable" settings.
This matters because practitioners currently rely on hand-designed curricula or separate specialists.
-
Unusually complete and candid controls.
- The paper includes fixed uniform and 1:1:1:2:2 weights, reward dropout, refresh-interval and EMA sweeps, alternative solvers, and cached-vs-fresh agreement (Tables 4–6 and S2; App. E).
- It reports the per-step variant even though that variant is worse.
This matters because these tables make the claims checkable; my weaknesses below are computed from them.
Critical weaknesses
-
The evidence supports gains on the two specialist rewards, not the claimed across-the-board superiority.
- The claims. The abstract says MARBLE "improves all eight evaluation metrics over both simultaneous weighted-sum training and fixed uniform coefficients"; the contributions claim the "best overall performance".
- Single runs, selected on the test sets. Each configuration was run once. N and ρ were chosen on the same test sets used in Table 1 (Sec. 4.3).
- The paper's own standard of "comparable". The paper calls N = 5 and N = 10 "comparable" (Sec. 4.3), yet they differ by 0.39 PickScore, 0.11 Aesthetic and 0.06 UniReward. MARBLE's margins over fixed α = 0.2 on those metrics are smaller: +0.07, +0.04 and +0.03 (Table 4).
- Only the selected setting wins. Only the selected configuration beats fixed α = 0.2 on all eight metrics. The other six reported configurations (Tables 5–6) each lose to DanceGRPO-5r on six of eight metrics, including ImageReward and UniReward.
- The weighted-sum comparison mostly reflects per-reward standardization. Fixed α = 0.2 already obtains 79–95% of MARBLE's gain over the weighted sum on PickScore, HPSv2.1, Aesthetic, ImageReward and UniReward (Tables 1 and 4). The weighted sum's weights are not given, and it does not raise its own training rewards PickScore and HPSv2.1 above the CFG base (22.38 vs 22.34; 0.274 vs 0.279).
- Held-out evidence is mixed. UniReward is −0.05 vs DanceGRPO-5r and −0.08 vs DiffusionNFT†. The user study (30 images per method, means only) omits DanceGRPO-5r.
- Undisclosed baseline provenance. Rows SD-XL through DiffusionNFT† match DiffusionNFT's Table 1 (ICLR 2026) exactly. That table evaluated SD-XL and SD3.5-L at 1024 × 1024, whereas Table S5 states 512 × 512 and guidance 1.0 for all rows.
This matters because the headline claims are what the paper will be judged on. As reported, they cannot be separated from configuration selection and run-to-run variation. It is also known that tuned scalarization often matches multi-task optimizers (Xin et al., NeurIPS 2022).
-
The stated mechanism is not what the diagnostics test, nor what most updates apply.
- The diagnostic holds by construction. The paper notes that the min-norm direction satisfies ĝ_k^T d* ≥ ‖d*‖² (p. 4). For the refresh direction, this implies cos(d*, g_k) ≥ ‖d*‖ > 0 for every reward, with the cosine equal across all rewards that have positive weight. The abstract places the diagnostic on "the refresh update", so Table S1's 0% conflict rate and 0.006 variance follow from the construction.
- Most updates are different. 9 of 10 updates are cached steps (Eq. 10). By the linearity behind Prop. 1, the cached update appears to equal Σ_k ᾱ_k (n_k/B) g_k, plus a term that vanishes on-policy. That means the weights from the normalized solve are applied to unnormalized gradients. It also means OCR and GenEval, each valid on about a third of the samples, are scaled down accordingly, because Eq. 10 averages over all samples "without renormalizing". The common-descent property is thus not established for these steps. Read from Fig. S12, the cached direction's cosine with a fresh solve is already about 0.83 one step after a refresh.
- The variant closest to the analysis does worse. Solving α at every step scores below amortized MARBLE on all eight metrics, and its PickScore of 21.80 is below even the CFG base's 22.34 (Table 4).
- The priority claim is thin. "First to reveal conflicts" (Sec. 5) rests on five mini-batches, 4 of which show a conflict.
This matters because the paper's insight is advertised as identifying and resolving gradient conflicts. If the evidence covers only the refresh direction, where it holds by construction, why MARBLE works remains open. A simpler account appears to fit the results at least as well: adaptive re-weighting of standardized advantages that leans toward the specialist rewards. The GenEval coefficient often rises to about 0.3 (App. C.3).
Questions for authors
-
(Suggestion.) Please report at least 3 seeds (mean ± s.d. on all eight metrics) for MARBLE, fixed α = 0.2, 1:1:1:2:2 and DanceGRPO-5r. Please choose N and ρ on a held-out prompt split rather than the test sets.
-
(Clarification.)
- Were Table 1's rows SD-XL through DiffusionNFT† taken from DiffusionNFT's Table 1? If so, please say so, correct Table S5, and confirm that MARBLE was evaluated with the identical pipeline (resolution, seeds, UnifiedReward version). This matters because evaluation pipelines differ: for SD3.5-M with CFG, UnifiedReward is 3.33 in Flow-GRPO's paper and 3.03 in DiffusionNFT's.
- What were the weighted-sum weights and normalization?
- Was DanceGRPO-5r's summed advantage rescaled before the NFT loss, or were A_max or the learning rate re-tuned for it?
-
(Clarification / suggestion.)
- Is the direction d in Table S1 the refresh direction?
- Please report conflict statistics for the cached updates actually applied, and for fixed α = 0.2, over many batches.
- Please report per-reward ‖g_k‖ and valid fractions, so the effective cached weights can be seen.
- What does "Solve α every step" apply: Eq. 7 or Eq. 10, and with or without the EMA?
A cached variant rescaled by 1/‖g_k‖ and B/n_k would test the rationale directly.
-
(Suggestion.) Could the user study include DanceGRPO-5r, state how prompts were selected and whether they were shared across methods, and report bootstrap CIs?
What would move me to 3 (weak accept): all three of the following.
- Multi-seed results, with settings chosen on held-out prompts, keep MARBLE ahead of fixed α = 0.2 and DanceGRPO-5r on GenEval and OCR without losses on the held-out metrics.
- The baseline provenance and settings are clarified.
- The claims are recalibrated: specialist gains, and the mechanism framed as adaptive re-weighting unless cached-step diagnostics show reduced conflicts.
Minor (no effect on my rating):
- Related work. Please soften "first to reveal conflicts". DRaFT (ICLR 2024) already reported that per-reward gradients have "substantially different norms" and weighted the rewards accordingly. Consider citing:
- FAMO (NeurIPS 2023), on amortized balancing;
- MoCo (ICLR 2023), on the bias of stochastic MGDA, which bears on the per-step explanation;
- CaPO (CVPR 2025) and Diffusion-DPO;
- the tuned-scalarization studies (Xin et al. and Kurin et al., NeurIPS 2022).
- Throughput. In Table 3, a refresh every 10 steps at the non-amortized 0.56× suggests about 0.93×, not 0.97×. How was throughput measured?
- Fig. S4. App. C.3 says MARBLE "converges to a higher final value" on HPSv2.1. Fig. S4 does not show this over its plotted range.
- Missing details.
- Numbers or images for the "Collapse" row.
- Total training steps, NFT β, β_KL, mini-batch size and baseline hyperparameters in Table S3.
- Whether PCGrad, CAGrad and Nash-MTL (App. E.3) were amortized like MARBLE.
- Citations. DDPO, Flow-GRPO, Parrot, Nash-MTL and DiffusionNFT are cited as arXiv preprints, but appeared at ICLR 2024, NeurIPS 2025, ECCV 2024, ICML 2022 and ICLR 2026 respectively.