- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses the tension between scalarized rewards and single rewards during the post-training of image generation models. To mitigate this, the authors propose Scalar Reward Concordance (SRC), a method that scores each rollout based on the alignment of the weakest active reward with the scalarized advantage. The paper introduces two variants: SRC-Reweight, which reallocates rollout weights without additional sampling, and SRC-Evolve, which selects a fixed-size subset from extra on-policy samples. Evaluated primarily on SD3.5-M (using DiffusionNFT and GRPO-Guard) across general-preference prompts, the method demonstrates consistent aesthetic improvements, while robustly maintaining prompt adherence.
Critical strengths
(1)Clear and intuitive formulation. The paper does an excellent job of articulating the tension between scalarized advantages and single rewards. SRC offers an interpretable, rollout-level metric based purely on observed rewards, elegantly bypassing the requirement for reward gradients.
(2)Well-designed, complementary interventions. Evaluate both reweighting (SRC-Reweight) and additional sampling (SRC-Evolve) under DiffusionNFT and GRPO-Guard.
Critical weaknesses
(1) Limited evidence on conflicts involving task-specific rewards. While the training setup includes GenEval and OCR rewards, SRC is currently only applied to Pick-a-Pic groups (PickScore, HPSv2, and CLIPScore). Although Appendix mentions ceiling effects, it stops short of quantifying these omitted reward-combination scenarios. Consequently, the method's effectiveness in resolving actual conflicts between composition (e.g., number and attribute control), text rendering, and general preference remains largely unproven.
(2) Missing a crucial sequential baseline. The paper lacks a comparison with staged single-reward fine-tuning. Without this control, it is difficult to disentangle whether the performance gains stem from successfully resolving conflicting supervision, or simply from giving the model more capacity to learn undertrained capabilities.
Questions for authors
(1)Can you report the actual reward-combination experiments mentioned in Appendix? Specifically, it would be valuable to see if SRC can resolve the conflict between composition (number/attribute control) and text rendering without degrading overall image quality in the full evaluation suite.
(2) Can you provide a direct comparison between joint SRC-based training and staged single-reward fine-tuning, strictly matched for total training and sampling budgets?
2 comments
Thank you for the response. However, my concerns regarding the effectiveness and stability of the proposed method remain unresolved.
Author Rebuttal
Thank you for recognizing the rollout-level formulation and the complementary reweighting and resampling interventions. We address your concerns below.
W1/Q1. Reward-combination results and task-specific conflicts
Our empirical results show that SRC on 3 rewards of PaP improves aesthetics (Pickscore, HPS) while prompt following (CLIP) is broadly retained. Table 1 assesses this balance through GenEval and OCR correctness alongside the general training rewards and three held-out evaluation rewards.
The DiffusionNFT reward-combination results are reported below and have been added to Appendix C.3. Reward scales and Overall follow Table 1. The extension applies SRC-Evolve to both GenEval and OCR groups in addition to PaP.
The reward-combination experiments extend this assessment to SRC applied to groups containing GenEval or OCR. These configurations also improve aesthetics while broadly retaining task-specific performance, but yield a slightly less favorable balance than PaP-only SRC, with lower corresponding task-specific correctness. This comparison motivated the PaP-only configuration in the main experiments. The results show how the application scope affects the quality–correctness trade-off, but do not mean that task-specific groups must be excluded for SRC to provide benefits.
The qualitative comparisons (Figure 8) also yield similar results by illustrating visual quality and prompt following on the same prompts across methods, including composition and text-rendering requirements.
W2/Q2. Comparison with additional baselines and methods
Here we provide additional results comparing SRC with sequential single-reward fine-tuning below.
The sequential order is set as below:
Each reward is trained for 160 epochs (800 total), with advantages determined solely by the current reward. GenEval and OCR stages use their respective prompt sources, the other stages use PaP prompts. The results are shown below.
Sequential A and B reach Overall scores of 1.0225 and 0.9455, respectively, below DiffusionNFT (1.0676) and both SRC variants. Sequential B achieves the highest AestheticScore but substantially lower GenEval and OCR scores, illustrating the quality–correctness trade-off. The results show that for DiffusionNFT, sequential single-reward fine-tuning does not achieve the same balance of visual quality and task-specific correctness as SRC-based methods. These results have been added to Appendix D.3 (Table 8).