acceptodds
Under review as a conference paper at ICLR 2027

Concordance-Guided Rollout Reweighting and Resampling for Multi-Reward Image Generation Post-Training

Abstract

Multi-reward post-training of text-to-image models commonly reduces several rewards to a single scalar reward. However, the resulting scalarized advantage can disagree in sign with single-reward advantages, concealing conflicting supervision within a rollout group generated for the same prompt. To address this issue, we introduce Scalar Reward Concordance (SRC), a continuous score that quantifies the weakest active-reward support for each rollout's scalarized advantage under a fixed reward scalarization. Derived from within-rollout gradient alignment for the GRPO family, SRC uses only observed reward values. It identifies concordant positive and negative supervision, including rollouts with jointly high or jointly low rewards. Building on this measure, we develop two methods: SRC-Reweight redistributes training weights over existing rollouts without additional sampling, while SRC-Evolve draws additional on-policy rollouts and selects a fixed-size population using SRC. Both methods integrate with GRPO-Guard and DiffusionNFT while preserving the prescribed reward scalarization. We further introduce Joint Success Rate (JSR) to compare models in terms of their ability to satisfy multiple reward requirements simultaneously. Experiments covering five training rewards and three held-out evaluation rewards show improved aesthetics with prompt following broadly retained. SRC-Evolve also improves the joint success rate on the applicable training rewards relative to baselines with matched numbers of generated rollouts.

Primary area:
generative models
Keywords:
image generation, multi-reward post-training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_FFnFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the tension between scalarized rewards and single rewards during the post-training of image generation models. To mitigate this, the authors propose Scalar Reward Concordance (SRC), a method that scores each rollout based on the alignment of the weakest active reward with the scalarized advantage. The paper introduces two variants: SRC-Reweight, which reallocates rollout weights without additional sampling, and SRC-Evolve, which selects a fixed-size subset from extra on-policy samples. Evaluated primarily on SD3.5-M (using DiffusionNFT and GRPO-Guard) across general-preference prompts, the method demonstrates consistent aesthetic improvements, while robustly maintaining prompt adherence.

Critical strengths

(1)Clear and intuitive formulation. The paper does an excellent job of articulating the tension between scalarized advantages and single rewards. SRC offers an interpretable, rollout-level metric based purely on observed rewards, elegantly bypassing the requirement for reward gradients.

(2)Well-designed, complementary interventions. Evaluate both reweighting (SRC-Reweight) and additional sampling (SRC-Evolve) under DiffusionNFT and GRPO-Guard.

Critical weaknesses

(1) Limited evidence on conflicts involving task-specific rewards. While the training setup includes GenEval and OCR rewards, SRC is currently only applied to Pick-a-Pic groups (PickScore, HPSv2, and CLIPScore). Although Appendix mentions ceiling effects, it stops short of quantifying these omitted reward-combination scenarios. Consequently, the method's effectiveness in resolving actual conflicts between composition (e.g., number and attribute control), text rendering, and general preference remains largely unproven.

(2) Missing a crucial sequential baseline. The paper lacks a comparison with staged single-reward fine-tuning. Without this control, it is difficult to disentangle whether the performance gains stem from successfully resolving conflicting supervision, or simply from giving the model more capacity to learn undertrained capabilities.

Questions for authors

(1)Can you report the actual reward-combination experiments mentioned in Appendix? Specifically, it would be valuable to see if SRC can resolve the conflict between composition (number/attribute control) and text rendering without degrading overall image quality in the full evaluation suite.

(2) Can you provide a direct comparison between joint SRC-based training and staged single-reward fine-tuning, strictly matched for total training and sampling budgets?

2 comments
AuthorsOpenReview ↗

Author Rebuttal

Thank you for recognizing the rollout-level formulation and the complementary reweighting and resampling interventions. We address your concerns below.

W1/Q1. Reward-combination results and task-specific conflicts

Our empirical results show that SRC on 3 rewards of PaP improves aesthetics (Pickscore, HPS) while prompt following (CLIP) is broadly retained. Table 1 assesses this balance through GenEval and OCR correctness alongside the general training rewards and three held-out evaluation rewards.

The DiffusionNFT reward-combination results are reported below and have been added to Appendix C.3. Reward scales and Overall follow Table 1. The extension applies SRC-Evolve to both GenEval and OCR groups in addition to PaP.

MethodGenEvalOCRPickScoreCLIPHPSv2Aesth.UniRwdCycleRwdOverall
DiffusionNFT0.86450.92230.89560.30110.32185.72773.34071.23081.0676
SRC-Evolve (PaP only)0.86730.93710.90050.30320.32976.08983.44891.25911.0907
SRC-Evolve w/ GenEval & OCR0.86210.93380.90190.30340.33126.09233.43731.25401.0894
DiffusionNFT x30.87140.93280.89960.30280.32665.79463.35011.25641.0781

The reward-combination experiments extend this assessment to SRC applied to groups containing GenEval or OCR. These configurations also improve aesthetics while broadly retaining task-specific performance, but yield a slightly less favorable balance than PaP-only SRC, with lower corresponding task-specific correctness. This comparison motivated the PaP-only configuration in the main experiments. The results show how the application scope affects the quality–correctness trade-off, but do not mean that task-specific groups must be excluded for SRC to provide benefits.

The qualitative comparisons (Figure 8) also yield similar results by illustrating visual quality and prompt following on the same prompts across methods, including composition and text-rendering requirements.

W2/Q2. Comparison with additional baselines and methods

Here we provide additional results comparing SRC with sequential single-reward fine-tuning below.

The sequential order is set as below:

  • Sequential A (from task-specific to general): GenEval → OCR → PickScore → HPSv2 → CLIPScore
  • Sequential B (from general to task-specific): CLIPScore → HPSv2 → PickScore → OCR → GenEval

Each reward is trained for 160 epochs (800 total), with advantages determined solely by the current reward. GenEval and OCR stages use their respective prompt sources, the other stages use PaP prompts. The results are shown below.

MethodGenEvalOCRPickScoreCLIPHPSv2Aesth.UniRwdCycleRwdOverall
DiffusionNFT0.86450.92230.89560.30110.32185.72773.34071.23081.0676
Sequential A0.82680.77640.88820.29830.31265.78743.30991.13211.0225
Sequential B0.65060.58410.89550.29210.33436.16433.30680.90770.9455
SRC-Reweight0.86330.92570.89620.30020.32305.83173.36311.24311.0728
SRC-Evolve0.86730.93710.90050.30320.32976.08983.44891.25911.0907

Sequential A and B reach Overall scores of 1.0225 and 0.9455, respectively, below DiffusionNFT (1.0676) and both SRC variants. Sequential B achieves the highest AestheticScore but substantially lower GenEval and OCR scores, illustrating the quality–correctness trade-off. The results show that for DiffusionNFT, sequential single-reward fine-tuning does not achieve the same balance of visual quality and task-specific correctness as SRC-based methods. These results have been added to Appendix D.3 (Table 8).

Thank you for the response. However, my concerns regarding the effectiveness and stability of the proposed method remain unresolved.

Reviewer_4mNsOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies multi-reward RL for text-to-image models. A scalarized final advantage from different rewards may conflict with one or more certain reward scores. To alleviate the conflicting supervision, this paper proposes scalar reward concordance(SRC), which measures the weakest support of individual rewards for the scalarized advantage. When all rewards agree, positively or negatively, the SRC will increase. Two methods are proposed based on SRC, reweighting and evolving. Reweighting assign weights based on SRC and evolving needs extra sampling to increase SRC. The experiments demonstrate the effectiveness of SRC-reweighting and SRC-evolving.

Critical strengths
  1. The paper is well written and easy to follow. The motivation and SRC definition, and two methods based on SRC, are articulated clearly.
  2. The proposed methods are simple and effective. They are easy to integrate with existing backbones and have practical value.
  3. The experiments are convincing and the conclusions are solid. The evaluation covers several text-to-image backbones. Also, mixed results and ablation studies are reported, which strengthens credibility.
Critical weaknesses
  1. The problem setting limits the method's ceiling. The reward scalarization for multiple reward RL is a simple strategy and may be the bottleneck. There are many advanced frameworks for multi-reward training. Although this paper addresses the problem of exisiting scalarized pipelines, a general solution for multi-reward post-training is more welcomed.
  2. Using "min" (weakest support) in SRC definition places strong requirements on reward scaling and reliability. This design makes SRC scores rather sensitive to reward noise. And more reward models may lower SRC quickly. The sensitivity of "min" could be explored or further discussed.
Questions for authors

Please see weaknesses above.

3 comments
AuthorsOpenReview ↗

Author Rebuttal

Thank you for the positive assessment of the formulation, practical integration, and transparent reporting of mixed results. We address the scope of fixed scalarization and the reliability of the minimum-contribution criterion below.

W1. The problem setting limits the ceiling

Although SRC is designed based on scalar reward that aggregates multiple rewards, SRC is quite flexible and can also accommodate dynamic reward weights. Its formulation requires a specified reward-weight vector when scoring a rollout group, but does not require that vector to remain fixed throughout training. Methods that dynamically adjust reward weights can therefore supply the current weights, while SRC determines how individual rollouts contribute under those weights. These interventions are complementary: one adapts reward weights, and the other uses rollout-level reward concordance to reweight or select training samples. Our experiments keep reward weights fixed to isolate the contribution of SRC. This experimental choice does not restrict SRC to fixed reward weights.

We agree that a more general solution for multi-reward post-training is a valuable direction for future work.

W2. Sensitivity to reward scaling, noise, and reward count

We use the minimum to characterize the disagreement level of a sample in the group, the intention is to make the training focus more on concordant samples. Adding more rewards may lower SRC, but our methods use SRC within the current rollout group or candidate pool rather than comparing its absolute values across training updates or reward configurations. So, our formulation does not rely on the absolute value of the SRC score. Instead, the relative proportion/ranking is used in reweight/evolve. Therefore, the scores alone do not imply worse selection or reweighting.

Thanks for your clarification. I'll keep my positive rating.

Thank you for your positive feedback. We sincerely appreciate your time and consideration. We would be happy to address any further questions or concerns.

Reviewer_ZCdSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Scalar Reward Concordance (SRC), which measures whether each individual reward supports the direction of the scalarized advantage for a rollout. SRC is used to reweight existing rollouts or select a fixed-size population from additional samples. Experiments on image-generation post-training show modest improvements in reward means and joint success rates.

Critical strengths
  • SRC provides a simple and interpretable rollout-level measure of agreement between individual rewards and the scalarized training signal.

  • Both variants are easy to integrate into existing post-training methods: SRC-Reweight requires no additional sampling, while SRC-Evolve includes a matched-rollout-budget comparison.

Critical weaknesses
  • The motivation overlaps with existing multi-reward methods that preserve reward-specific information before constructing the policy signal. Although SRC introduces rollout-level sign concordance, its benefit beyond methods such as GDPO is not yet fully established, and comparisons with other relevant approaches are limited.
  • Under the matched sampling budget, SRC-Reweight provides only modest gains, while SRC-Evolve relies on additional rollout generation and improves only slightly over the corresponding 3×3\times baselines. Moreover, SRC is applied only to PaP rollout groups, limiting the demonstrated generality.
Questions for authors
  • Could the authors clarify the key distinction and practical advantage of SRC over existing multi-reward advantage methods such as GDPO, Multi-GRPO, CMO, and MARBLE?

  • Why is SRC applied only to PaP rollout groups rather than GenEval and OCR groups, and how does this limitation affect the claimed generality?

  • Do the authors have evidence that the improvements over the matched (3×)(3\times) baselines are robust across training seeds?

2 comments
AuthorsOpenReview ↗

Author Rebuttal (1/2)

Thank you for recognizing SRC's interpretability and its integration into existing post-training methods. We address each of your concerns below.

W1/Q1. Distinction and comparison with other methods

The key distinction of SRC proposed in our paper are the motivation and formulation. GDPO normalizes rewards separately before aggregation, Multi-GRPO develops objective-wise advantage estimation alongside trajectory grouping, CMO dynamically adjusts concept-reward weights using group-level correlations, MARBLE coordinates reward-specific policy gradients. SRC instead measures whether every active single-reward advantage of an individual rollout agrees in sign with its scalarized advantage. SRC-Reweight changes rollout weights, and SRC-Evolve changes the selected training group, while retaining the prescribed scalar reward. Importantly, SRC is not a replacement for other methods, but a complementary intervention that can be applied alongside them.

For GDPO, separate reward normalization does not by itself prevent an aggregated advantage from opposing one reward's advantage for a particular rollout. SRC explicitly detects this disagreement. It also recognizes concordant negative supervision by discouraging jointly low-reward rollouts. SRC therefore captures both positive and negative concordant supervision. From the results in Table 1, we see that GDPO yields more unbalanced between rewards than the baseline.

We additionally implemented CMO and MARBLE on DiffusionNFT, using the same experimental setup as the other comparisons in our paper. The results below are also included in the updated Table 1.

MethodGenEvalOCRPickScoreCLIPHPSv2Aesth.UniRwdCycleRwdOverall
DiffusionNFT0.86450.92230.89560.30110.32185.72773.34071.23081.0676
GDPO0.85310.89290.89830.30210.32245.77943.34511.22541.0634
CMO0.86510.92870.88740.30250.31875.76543.34421.22701.0673
MARBLE0.86740.92350.88910.30140.32335.78253.35061.22751.0693
SRC-Reweight0.86330.92570.89620.30020.32305.83173.36311.24311.0728
SRC-Evolve0.86730.93710.90050.30320.32976.08983.44891.25911.0907

CMO and MARBLE reach Overall scores of 1.0673 and 1.0693, respectively, compared with 1.0728 for SRC-Reweight and 1.0907 for SRC-Evolve. SRC-Evolve improves all eight reported rewards over CMO and six over MARBLE. MARBLE achieves slightly higher GenEval and CLIP scores than SRC-Evolve.

AuthorsOpenReview ↗

Author Rebuttal (2/2)

W2/Q3. Modest gains and robustness of the improvements

First, the reported results are means over three runs, rather than results from a single run. We provide multi-run results of SRC-Evolve below. The tables report each run and the mean ± sample standard deviation across the three runs. Reward definitions and scales follow Table 1.

DiffusionNFT + SRC-Evolve

RunGenEvalOCRPickScoreCLIPHPSv2Aesth.UniRwdCycleRwd
Run 10.86160.93600.89960.30450.33226.10613.45411.2629
Run 20.87090.93190.89710.30100.32976.12603.44771.2656
Run 30.86940.94320.90480.30390.32726.03733.44501.2487
Mean ± SD0.8673 ± 0.00500.9371 ± 0.00570.9005 ± 0.00390.3032 ± 0.00190.3297 ± 0.00256.0898 ± 0.04663.4489 ± 0.00471.2591 ± 0.0091

GRPO-Guard + SRC-Evolve

RunGenEvalOCRPickScoreCLIPHPSv2Aesth.UniRwdCycleRwd
Run 10.78200.91380.90660.30870.33886.09083.36341.3281
Run 20.78740.91200.90430.30340.33656.01993.34881.3351
Run 30.79590.90220.90300.30500.34306.05593.32801.3115
Mean ± SD0.7884 ± 0.00700.9093 ± 0.00630.9046 ± 0.00190.3057 ± 0.00270.3394 ± 0.00336.0555 ± 0.03543.3467 ± 0.01781.3249 ± 0.0121

Second, individual reward means alone cannot establish how often the same image satisfies multiple reward requirements simultaneously. Models with similar reward means can differ in this joint success probability. We therefore introduce and report Joint Success Rate (JSR) in Section 5.4, which measures the probability that one image meets every applicable training-reward threshold, using shared, fixed base-model reference thresholds. In the reported SD3.5-M experiments, SRC-Evolve improves JSR over most interior percentiles relative to the matched-rollout references. Thus, our evaluation includes both individual reward means and direct measurements of joint threshold satisfaction.

Third, we also evaluate our method on the larger FLUX.2 [klein] 9B Base model (Klein9B) with Flow-GRPO in Table 2. From the base initialization, SRC-Reweight and SRC-Evolve improve Overall from the Flow-GRPO baseline's 50.98 to 51.14 and 51.35, respectively. These results support generalization beyond SD3.5-M to a larger model, another post-training backbone, and an additional benchmark.

W2/Q2. Why apply SRC only to PaP groups?

This scope was selected empirically. The following DiffusionNFT experiment compares PaP-only SRC-Evolve with an extension that also applies SRC to GenEval and OCR rollout groups, including the corresponding task-specific reward alongside PickScore, HPSv2, and CLIPScore. Reward scales and Overall follow Table 1. We have added these results to Appendix C.3.

MethodGenEvalOCRPickScoreCLIPHPSv2Aesth.UniRwdCycleRwdOverall
DiffusionNFT0.86450.92230.89560.30110.32185.72773.34071.23081.0676
SRC-Evolve (PaP only)0.86730.93710.90050.30320.32976.08983.44891.25911.0907
SRC-Evolve w/ GenEval & OCR0.86210.93380.90190.30340.33126.09233.43731.25401.0894
DiffusionNFT x30.87140.93280.89960.30280.32665.79463.35011.25641.0781

Extending SRC-Evolve to GenEval and OCR groups slightly improves the three general training rewards and AestheticScore, but reduces task-specific correctness, UniReward, CycleReward, and Overall relative to PaP-only SRC-Evolve. We therefore use PaP-only SRC to better balance visual quality and task-specific correctness.

GenEval and OCR measure task-specific correctness with attainable ceilings at 1, where tied scores can limit within-group discrimination and affect SRC's minimum-contribution criterion.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all three reviewers for their careful reading and constructive feedback. We appreciate their recognition of SRC's simple, interpretable rollout-level formulation, which uses observed rewards without requiring reward gradients (ZCdS, FFnF), and the practical integration of the complementary SRC-Reweight and SRC-Evolve interventions into existing post-training backbones (all three reviewers). We also appreciate the positive assessments of the clear motivation and presentation (4mNs, FFnF), the comparison at matched rollout counts (ZCdS), and the transparent reporting of mixed results and ablations (4mNs).

In response to the reviewers' concerns, we have expanded the empirical evaluation and clarified the scope of our claims. The main additions and revisions relative to the initial submission are:

  1. Additional multi-reward method comparisons. We added DiffusionNFT implementations of CMO and MARBLE to Table 1, reporting all eight individual rewards alongside Overall. We also clarify how SRC's rollout-level reweighting and selection differ from reward normalization, reward-weight adaptation, and reward-specific gradient coordination.
  2. Expanded larger-model evaluation with SFT initialization. We added Qwen-Image-Bench-EN results for Flow-GRPO + SRC-Evolve starting from Klein9B-SFT. We also moved the existing Klein9B base-initialization evaluation from the appendix into a dedicated main-text subsection, presenting the two initializations separately.
  3. Quantitative evidence on SRC's application scope. Appendix C.3 now compares PaP-only SRC-Evolve with an extension to GenEval and OCR rollout groups under DiffusionNFT. We report the individual reward trade-offs and explain that the PaP-only scope was selected empirically.
  4. Sequential single-reward fine-tuning baselines. Appendix D.3 now compares joint training with two orders of sequential fine-tuning across the five training rewards.

We address the remaining questions and discuss the evidence and its limitations in the reviewer-specific responses below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.